CN114598590A - Detection method for stability element and related equipment - Google Patents
Detection method for stability element and related equipment Download PDFInfo
- Publication number
- CN114598590A CN114598590A CN202210500582.4A CN202210500582A CN114598590A CN 114598590 A CN114598590 A CN 114598590A CN 202210500582 A CN202210500582 A CN 202210500582A CN 114598590 A CN114598590 A CN 114598590A
- Authority
- CN
- China
- Prior art keywords
- data stream
- stability
- variance
- time
- stability element
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
Abstract
The invention discloses a detection method and related equipment for a stability element, wherein the method comprises the following steps: constructing a probability data structure, carrying out continuous inspection on the data stream by using the probability data structure, and judging whether the data stream is interrupted within a preset time; if the data stream is not interrupted within the preset time, calculating the variance of the frequency values of the data stream within the preset time; judging whether the variance is smaller than a preset threshold value, and if so, judging that the data stream is a stability element; and monitoring the stabilization time of the stability element in real time, and if the stability element does not have stability any more, judging that the states of the network and the data stream possibly have faults. In the invention, the data stream which is not interrupted within the preset time and the variance of the frequency value of the data stream is smaller than the preset threshold value is called as the stability element, so that the stability element in the data stream is accurately detected, and the diagnosis basis is effectively provided for carrying out abnormity diagnosis on the network, the data stream state and the like.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method and a related device for detecting a stability element.
Background
A bursty element (burst) refers to a stream that suddenly increases or decreases within a period of time in a data stream, and the bursty element has important applications in many fields, such as networks, databases, network security, and the like. On the contrary, the flow with the flow rate maintained stable in a period of time also has an important role, the network and data flow state can be judged by detecting the flow with the flow rate maintained stable, the abnormality diagnosis can be carried out, the hit rate can be improved in an operating system, and the method has an important role in the fields of network, database, network security and the like.
The flow concept similar to the flow maintaining stability is a persistence element, but the persistence element only records the time window number of a flow in a time period, does not perform frequency estimation on the data flow, and even does not have the capability of identifying a stable flow.
Therefore, in the prior art, a data stream algorithm for detecting a stable stream is not available, and it is impossible to utilize a small space to realize accurate detection of a stability element in a data stream, wherein a data stream which is uninterrupted in p time windows and whose frequency variance is smaller than a certain threshold is identified as a stability element.
Disclosure of Invention
The present invention provides a method and a related device for detecting a stability element, and aims to solve the problem that in the prior art, a stability element in a data stream cannot be accurately detected, and therefore, anomaly diagnosis cannot be performed on a network and a data stream state by detecting the stability element.
In order to achieve the purpose, the invention adopts the following technical scheme:
a method of detecting for a stability element, comprising the steps of:
constructing a probability data structure, carrying out continuous inspection on a data stream by using the probability data structure, and judging whether the data stream is interrupted within a preset time;
if the interruption does not occur, calculating the variance of the frequency values of the data stream in the preset time;
judging whether the variance is smaller than a preset threshold value, and if so, determining that the data stream is a stability element;
and monitoring the stabilization time of the stability element in real time, and if the stability element is judged to have no stability any more according to the stabilization time, the network and the state of the data stream are in fault.
In the method for detecting a stability element, if the interruption does not occur, calculating a variance of a frequency value of the data stream occurring within the preset time includes:
if the data stream is not interrupted within the preset time, calculating the variance of the frequency value of the data stream recorded in the counter within the preset time to obtain a first variance;
if the frequency value is larger than the maximum count value of the counter, adding (maximum count value + 1)/2' to each frequency value, and calculating the second variance after performing modulus operation on the (maximum count value + 1) of the calculation result.
In the method for detecting a stability element, the constructing a probability data structure, performing persistence check on a data stream by using the probability data structure, and determining whether the data stream is interrupted within a preset time specifically includes:
after the probability data structure is built, mapping the data stream into a corresponding bucket in a stability filter by utilizing a hash function;
performing modulus operation on the total number of the bits in the bucket at the current moment when the data stream appears to obtain the bit corresponding to the current time window, and setting the bit corresponding to the current time window to be 1;
and judging whether '0' exists in the time window in the preset time, if so, interrupting the data stream in the preset time, otherwise, not interrupting the data stream in the preset time.
In the method for detecting a stability element, the determining whether the data stream is interrupted within a preset time further includes:
and if the interruption occurs, the data stream is an unstable element, and the data stream is not processed.
In the method for detecting a stability element, the determining whether the variance is smaller than a preset threshold, and if so, determining that the data stream is a stability element specifically includes:
comparing the magnitudes of the first variance and the second variance with the preset threshold respectively, and if any one of the first variance and the second variance is smaller than the preset threshold, determining that the data stream is a stability element.
In the detection method for the stability element, the preset time is the time of the first p time windows of the current time window.
In the detection method for the stability element, the maximum count value = (2 ^ number of bits of counter) -1.
In the detection method for the stability element, the probability data structure includes: the stability filter, the multi-dimensional sketch structure and the stability element screener.
In the method for detecting the stability element, the stability filter is composed of d queues, wherein the d queues correspond to d hash functions, each queue contains g buckets, each bucket contains m bits, each bit corresponds to one time window, and each time window corresponds to one counter.
In the detection method for the stability elements, the multidimensional sketch structure consists of q queues, wherein the q queues correspond to q hash functions, each queue comprises j buckets, and each bucket comprises w counters.
In the method for detecting the stability element, the stability element filter is composed of k buckets, each bucket comprises x units, each unit comprises 3 areas, and the id of the data stream, the starting stability time of the data stream and the ending stability time of the data stream are recorded respectively.
A detection system for a stability element, the detection system for a stability element comprising:
the interruption detection module is used for constructing a probability data structure, carrying out continuous inspection on a data stream by using the probability data structure and judging whether the data stream is interrupted within preset time;
the variance calculation module is used for calculating the variance of the frequency value of the data stream in the preset time if the interruption does not occur;
the comparison module is used for judging whether the variance is smaller than a preset threshold value or not, and if so, determining that the data stream is a stability element;
and the fault judgment module is used for monitoring the stabilization time of the stability element in real time, and if the stability element is judged to have no stability any more according to the stabilization time, the network and the state of the data stream have faults.
A computer readable storage medium storing a detection program for a stability element, which when executed by a processor implements the steps of the detection method for a stability element as described above.
Compared with the prior art, the method for detecting the stability element and the related equipment provided by the invention comprise the following steps: constructing a probability data structure, carrying out continuity check on the data stream by using the probability data structure, and judging whether the data stream is interrupted within preset time; if the data stream is not interrupted within the preset time, calculating the variance of the frequency value of the data stream within the preset time; judging whether the variance is smaller than a preset threshold value, and if so, judging that the data stream is a stability element; and monitoring the stabilization time of the stability element in real time, and if the stability element does not have stability any more, judging that the states of the network and the data stream possibly have faults. In the invention, the data stream which is not interrupted within the preset time and the variance of the frequency value of the data stream is smaller than the preset threshold value is called as the stability element, so that the stability element in the data stream is accurately detected, and the diagnosis basis is effectively provided for carrying out abnormity diagnosis on the network, the data stream state and the like.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of a method for detecting a stability element according to the present invention;
FIG. 2 is a flowchart of step S100 according to a preferred embodiment of the method for detecting a stability element of the present invention;
FIG. 3 is a schematic diagram of a stability filter data structure and data flow insertion provided by the present invention;
FIG. 4 is a flowchart of step S200 according to a preferred embodiment of the method for detecting a stability element of the present invention;
FIG. 5 is a schematic diagram of the multi-dimensional sketch structure and operation provided in the present invention;
FIG. 6 is a diagram illustrating the data structure and operation of a persistent stability element filter according to the present invention;
FIG. 7 is a schematic block diagram of a detection system for stability elements provided by the present invention;
FIG. 8 is a schematic diagram of an operating environment of a data flow state diagnostic system according to a preferred embodiment of the present invention.
Reference numerals: 100: an interrupt detection module; 200: a variance calculation module; 300: a comparison module; 400: and a fault judgment module.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The invention provides a detection method and related equipment for a stability element; according to the method and the device, whether the interruption of the data stream occurs within the preset time is judged, the variance of the frequency value of the data stream occurring within the preset time is calculated, whether the variance is smaller than the preset threshold value is judged, the data stream which has not occurred within the preset time and has the frequency value variance smaller than the preset threshold value is called as the stability element, so that the stability element in the data stream is accurately detected, and a judgment standard is effectively provided for whether the network and the state of the data stream have faults.
The following describes a design scheme of a detection method for a stability element by using specific exemplary embodiments, and it should be noted that the following embodiments are only used for explaining the technical scheme of the invention, and are not specifically limited:
referring to fig. 1, a method for detecting a stability element according to the present invention includes the following steps:
s100, constructing a probability data structure, carrying out continuity check on a data stream by using the probability data structure, and judging whether the data stream is interrupted within preset time;
wherein the probabilistic data structure comprises: the stability filter, the multidimensional sketch structure and the continuous stability element screener.
Specifically, in order to detect whether the data stream is the stability element, a novel and efficient probabilistic data structure called SteadySketch is provided, which can be in a smaller space, thereby realizing accurate detection of the stability element; unlike the detection of the stability element by using the probability data structure, in the detection and query of the persistence element, only the number of time windows in which the persistence element appears after a time period can be obtained, but it is not possible to report whether there is an interruption of the persistence element in the time period, and it is not possible to report the frequency count and frequency count stability of the persistence element; however, steadyssketch overcomes both of the above drawbacks well, and can probe the stability element accurately at high speed and in limited memory.
The SteadySketch basic structure consists of three parts, wherein the first part is the stability filter, the basic data structure of the stability filter is expanded by a bloom filter and consists of d queues, the d queues correspond to d hash functions, each queue contains g buckets, but the difference of the stability filter and the conventional bloom filter is that the buckets in each queue are not only bits of one bit, but also contain m bits, each bit corresponds to one time window, and each time window corresponds to one counter; d is a positive integer greater than 1, typically set to a number less than 5; g is a positive integer greater than 1, and can be thousands or even tens of thousands; the stability filter is used for carrying out continuity check on the stability element and can be used for judging whether the data stream is interrupted within a period of time (preset time);
in the method, a brand-new probability data structure is established, then, the probability data structure is utilized to carry out continuity check on the data stream, and whether the data stream is interrupted within the preset time is judged so as to carry out the next operation, thereby effectively judging whether the data stream is interrupted within the preset time.
Further, referring to fig. 2, step S100 is to construct a probability data structure, perform persistence check on a data stream by using the probability data structure, and determine whether the data stream is interrupted within a preset time, which specifically includes:
s110, after the probability data structure is built, mapping the data stream into a corresponding bucket in a stability filter by utilizing a hash function;
s120, performing modulus operation on the total number of the bits in the bucket at the current moment when the data stream appears to obtain the bit corresponding to the current time window, and setting the bit corresponding to the current time window to be 1;
s130, judging whether a time window in the preset time has '0', if so, interrupting the data stream in the preset time, otherwise, not interrupting the data stream in the preset time. The preset time is the time of p time windows before the current time window, p and m are positive integers greater than 1, and m is greater than or equal to (p + 2).
Specifically, after the probabilistic data structure is constructed, first, the operation of the first stage is performed: for example, 192.168.1.110000 TCP 121.14.88.7680 forms a quintuple, the source IP address is 192.168.1.1, the source port is 10000, the protocol is TCP, the destination IP address is 121.14.88.76, and the destination port is 80), and the data flow is calculated into one of g buckets in d queues, namely the data flow is mapped into the corresponding bucket in the stability filter by using a hash function;
then, performing modulus operation on m according to the current time when the data stream appears (modulus operation is to solve the remainder of division of two numbers), correspondingly selecting the bit of a current time window according to a modulus result, and judging whether the current window is already 1, if so, judging that the data stream does not appear for the first time, otherwise, judging that the data stream appears for the first time;
setting the bit of the current time window to be 1, reading the bit values in the first p time windows of the current time window (the time of the first p time windows is the preset time), judging whether the data stream is interrupted within the preset time as long as one of the bit values in the first p time windows is 0, otherwise, judging whether the data stream is not interrupted within the preset time, and finally marking whether the data stream appears for the first time and whether the data stream is interrupted.
Wherein the stability filter data structure and data stream insertion diagram are shown in FIG. 3, AiRepresenting a queue, AiThe next square is a barrel, each barrel contains m bits (the initial value of each bit is '0' in a small positive square in the figure), and if the number of bits before the current time window is less than p bits, the required number of bits is selected in the following cycle. For example, stream e in the figure2At t2When time occurs, first pass through the hash function (g)2（e2) E) the data stream2Mapping into corresponding bucket, then, calculating t2% m (modulo) to select the bit representing the current time window (the big and small squares in the figure), if the bit of the current time window is found to be marked as "1", it indicates that this element has appeared in the current time window, i.e. the data stream e2It does not occur for the first time in the previous p time windows, and if it is found that the bit of the current time window is not marked as "1", it indicates that this element does not occur in the current time window, i.e. the data stream e2First occurrence within the first p time windows; moreover, since the bits in the previous p time windows are all "1" (since the current time window is the 3 rd bit, and "p" is "3" for example), that is, the first 3 time windows of the current time window need to be read, the last one bit is read circularly and correspondingly, and similarly, if the first n time windows of the current time window (3 at this time) need to be read circularly, the last one bit is read circularly and correspondingly(n-2) bit number), so the data stream e is transmitted2The label is: "uninterrupted, and is the first occurrence in the current time window". For the same reason, for data stream e1And e3Respectively marked, wherein only the data stream e is shown in the figure2The following conditions are satisfied: the previous p time windows are uninterrupted and occur for the first time in the current time window.
Further, the determining whether the data stream is interrupted within a preset time further includes:
and S140, if the interruption occurs, the data stream is an unstable element, and the data stream is not processed.
Specifically, if the data stream is interrupted within the first p time windows, the data stream may be determined to be an unstable element, and the frequency variance calculation may not be performed.
Further, please continue to refer to fig. 1, S200, if no interruption occurs, calculating a variance of the frequency value of the data stream occurring within the preset time;
specifically, if the data stream has not been interrupted in the previous p time windows, the second stage of operation is entered: and calculating the frequency variance of the data stream. At this time, a second partial structure in the SteadySketch, namely, a multidimensional sketch structure (RollingSketch), is required to be used, and the multidimensional sketch structure is composed of q queues, and the q queues correspond to q hash functions, wherein each queue contains j buckets, each bucket contains w counters, one counter corresponds to a time window, the size of each counter is 1 byte (8 bits), frequency statistics is performed on data streams coming in the time window respectively, q, j and w are positive integers greater than 1, and w is greater than or equal to (p + 2); the counter is used for recording the frequency of the occurrence of the data stream in a plurality of time windows in a preset time; i.e. the variance of the frequency count of the data stream is calculated in the multi-dimensional sketch structure.
It should be noted that in order to reduce the amount of computation in the present application, the second stage operation may be entered when the data stream satisfies the condition that no interruption occurs in the previous p time windows and occurs for the first time in the current time window.
Further, referring to fig. 4, if the interruption does not occur, the step S200 of calculating the variance of the frequency value appearing in the data stream within the preset time includes:
s210, if the data stream is not interrupted within a preset time, calculating the variance of the frequency value of the data stream recorded in a counter within the preset time to obtain a first variance;
s220, if the frequency value is larger than the maximum count value of the counter, adding (maximum count value + 1)/2) to each frequency value, performing modulus operation on the calculation result (maximum count value + 1), and then performing variance calculation to obtain a second variance. Wherein the maximum count value = (number of bits of 2^ counter) -1.
Specifically, the step of calculating the frequency variance of the data stream in the multidimensional sketch structure is as follows:
firstly, similarly, a flow quintuple is calculated to one of g buckets in d queues through d hash functions, namely, the data flow is mapped into a corresponding bucket in a stability filter by using the hash function, then, the current time of the data flow is subjected to modulus extraction on w, a counter of a current time window is correspondingly selected according to a modulus extraction result, and a frequency value in the counter of the current time window is increased by one.
If the data stream is not interrupted in the first p time windows and appears on the current time window for the first time, in the second stage, performing variance calculation on the frequency values recorded in the first p counters in the data stream to obtain the first variance, but if any frequency value in the frequency values is greater than the maximum count value of the counter (the number-1 of bits of the 2^ counter), adding "(maximum count value + 1)/2" to the frequency values, performing modulo calculation on the calculation result "(maximum count value + 1)", and finally performing variance calculation on the modulo calculation result, wherein the process of calculating the variance after modulo calculation is also called offset variance calculation, and the final calculation result is the second variance, and the offset variance calculation process is called a regeneration mechanism, after the frequency value in the counter exceeds the maximum count value, the frequency value is calculated from 0 again and is regarded as overflow once, each overflow is regarded as regeneration once, and the dispersion among the frequency values is stored through offset variance calculation, so that variance estimation errors are effectively reduced, and the accuracy of variance calculation is not obviously reduced while the space occupation is reduced.
For example: the true frequency count value of the data stream in the first three time windows should be: 254. 256 and 258, the true variance should be 2.67; however, the frequency count value stored in the counter (for example, the maximum count value that can only be stored is 255 if an 8-bit counter is taken as an example, and the counter is recalculated after exceeding the maximum count value) is: 254. 0 and 2, the variance (first variance) thus calculated is: 21337, adding 128 to the stored frequency values and performing modulus extraction on 256 to obtain the frequency values after offset as follows: 126. 128 and 130, the variance after migration (second variance) is: 2.67, and the variance after offset is the same as the true value.
The multidimensional sketch structure and the operation schematic diagram are shown in fig. 5, in the data structure in the drawing, for example, 1 bucket (or slot) in one queue (a bucket is a tangent plane in the drawing) is used, each bucket has 6 counters (small squares in the drawing), each counter has 8 bytes, and in practical application, multiple queues corresponding to multiple hash functions can be set; in the figure, the sequence numbers (r), (r) and (e) respectively represent the operation sequence during the insertion, and when the element e is inserted into RollingScut at t time, the element e firstly passes through a hash function (f)i(e) Selects the Counter (Counter) mapped to the corresponding bucket, and then increments the frequency value in the Counter of the current time window by one by t% m, i.e. rotates the 8-bit Counter (Rolling Counter) clockwise in the figure by one.
If the marking of the stability filter in the first stage shows that the data stream is uninterrupted for the first p time windows (or the current time window appears for the first time), the variance of this element is calculated, the first time the bucket representing the first p time windows is fetchedThe frequency value of the counter of the port, and the variance (offset variance) V0Then, 128 is added to each frequency value to calculate the offset variance (offset frequency variance) V1As long as the variance V0And offset variance V1And if one of the variances is smaller than a preset threshold value, the data flow is determined as an instantaneous stability element.
Further, please continue to refer to fig. 1, S300, determining whether the variance is smaller than a preset threshold, and if so, determining that the data stream is a stability element;
specifically, after the variances (the first variance and the second variance) are calculated, the preset threshold is compared with the variance, and if the variances are both smaller than the preset threshold, and generally the preset threshold is 5, it may be determined that the data stream is a stability element.
Further, the step S300 of determining whether the variance is smaller than a preset threshold, and if so, determining that the data stream is a stability element, specifically including:
s310, comparing the first variance and the second variance with the preset threshold respectively, and if any one of the first variance and the second variance is smaller than the preset threshold, determining that the data stream is a stability element.
Specifically, after the first variance and the second variance are calculated, the preset threshold is compared with the first variance and the second variance, and if any one of the first variance and the second variance is smaller than the preset threshold, and generally the preset threshold is 5, the data stream may be determined to be a stability element and reported to a data center. The method for judging whether the data stream is the stability element is not limited to detecting the data stream, and can also be applied to searching ocean currents with stable directions and flow velocities in the ocean so as to provide help for navigation, monitoring the stability of the flow signals in the wireless sensor network and the like.
Further, please continue to refer to fig. 1, S400, the stability time of the stability element is monitored in real time, and if it is determined that the stability element no longer has stability according to the stability time, the states of the network and the data stream fail.
Specifically, after the second-stage reporting of the stability element is finished, the third-stage operation is performed, that is, the stability element is further detected in the third partial structure of the probabilistic data structure, whether the stability element is a persistent stability element (which is called a persistent stability element if the stability element can still be maintained for a period of time in the third stage, or is called a transient stability element), and the stability time of the stability element is monitored in real time, and if it is determined according to the stability time that the stability element does not have stability any more, it may be determined that one or both of the states of the network and the data stream may have a fault, or the data stream may have an interruption, and in the wireless sensor network, if a stream is not stable, the node that sends the stream is likely to have a problem, such as a lack of power or a shift in position, etc., so that some relevant sensing operations may be performed.
A third part of the structure of the probability data structure is the persistent stability element filter, which uses an unbiased spacing data structure to store and count stability elements, a basic data structure of the persistent stability element filter is composed of k buckets, each bucket contains x units (cells), each unit contains 3 regions, and the id of the data stream, the start stability time of the data stream and the end stability time of the data stream are respectively recorded, where k and x are positive integers greater than 1; the continuous stability element screener is mainly used for detecting and screening continuous stability elements, storing the elements judged to be the continuous stability elements and recording the stability duration of the elements;
the specific real-time monitoring process is as follows: similarly, mapping the data stream to a corresponding bucket through a hash function, preferentially judging whether the data stream with the same id exists in all the cells in the bucket, then judging whether vacant cells exist, if the data stream with the same id does not exist and the vacant cells exist, directly inserting the id of the data stream, starting the stabilization time as the current time window, and ending the stabilization time as the next time window; if the data streams with the same id exist and the ending stable time of the data streams is the current time window, updating the ending stable time of the data streams to be the next time window; if the data streams with the same id exist but the ending stable time of the data streams is not the current time window, updating the starting stable time and the ending stable time of the data streams to be the current time again, and reporting the existing data streams to the data center; if the data streams with the same id do not exist and no vacant cell exists, removing operation is carried out, namely, an element with the shortest stability duration (difference value between ending stable time and starting stable time) is selected, the id of the element is updated to be the id of the current data stream, and the ending stable time is updated to be the current time window.
The data structure of the persistent stability element filter and the operation diagram thereof are shown in fig. 6, in which the parameter p is set to 5 (for convenience of calculation, this time, the parameter p is set to 5, and in practice, the p values in the three parts should be the same, so the settling time of the data stream entering the third stage should be 5). The specific real-time monitoring process is as follows: firstly, data flow is carried out through a hash function<e1，21>And<e2，30>mapping to the same bucket (small square in the figure, unit is small cylinder in the figure) when the data stream e1Upon insertion, the data flow e is found to be present in the bucket1And the end settling time (Recent) is the last time window (20), this indicates the data stream e1If the stabilization process is still continuing, the ending stabilization time is updated to be current 21; when the data stream e2When inserted, the barrel is inserted directly because it has no identical elements but empty positions.
Similarly, the data stream<e4，26>And<e5，17>mapping to the same barrel through a hash function, wherein the data flow e4When inserted, the bucket is found to contain the same elements, but already existing dataStream e4Has been interrupted (existing data stream e)4With the data stream e being inserted4Has no overlapping part of the stability duration), the data stream e to be inserted is obtained4Reporting is done considering a persistent stability element that has ended, a new Start stability time (Start, 21 in the figure) and an end stability time (26 in the figure) are inserted. When the data stream e5When there is no empty place in the bucket at the time of insertion (the lower 4 cells in the figure are all full), a data stream with the shortest settling duration is selected (e in the figure)3Only lasting 5 time windows) and replacing the ID of the data stream with the shortest stabilization duration and the stabilization time (16) which is ended with the current data stream e5ID and current time (17), so only e is shown in the figure1Belonging to the elements of continuous stability. And when the final report is reported, all the elements in all the continuous stability element filters are output.
Further, in the present application, during the first and second stages of operations performed after the probabilistic data structure is constructed, that is, during the process of detecting whether the data stream is interrupted in the stability filter, and during the process of calculating the variance of the stability element in the multi-dimensional sketch structure, a data structure clearing process is also performed in synchronization with the data stream, the data structure clearing process is mainly performed in the first and second stages, and the main operation is to periodically clear the bit and counter in the counters and filters in the RollingSketch.
Since only the flow information (information of data flow) of m time windows is maintained in the data structure, in order to ensure the accuracy of the flow information, the values of the previous time windows need to be cleared. In order to reduce the repeated clearing operation, the clearing operation is performed only when the time window is switched, and the bit values of the counter and the filter of the next time window are selected to be cleared in order to prevent the influence on the insertion of the current time window. Therefore, when the time window is switched, the stability filter and the bit and counter corresponding to the next time window of the current time window in RollingSketch are cleared. And as the flushing operation does not influence the operation of the current time window, the parallel operation can be carried out by using multiple threads and the like, thereby not influencing the performance such as throughput rate and the like.
Further, referring to fig. 7, the present invention provides a detection system for a stability element, where the detection system for a stability element includes: the system comprises an interruption detection module 100, a variance calculation module 200, a comparison module 300 and a fault judgment module 400;
the interruption detection module 100 is configured to construct a probability data structure, perform continuity check on a data stream by using the probability data structure, and determine whether an interruption occurs to the data stream within a preset time; the variance calculating module 200 is configured to calculate a variance of a frequency value appearing in the data stream within the preset time if the interruption does not occur; the comparing module 300 is configured to determine whether the variance is smaller than a preset threshold, and if the variance is smaller than the preset threshold, determine that the data stream is a stability element; the failure determination module 400 is configured to monitor the stability time of the stability element in real time, and if it is determined that the stability element does not have stability any more according to the stability time, the states of the network and the data stream fail.
Specifically, in the present application, after the probabilistic data structure is constructed, the interruption detection module 100 performs a continuity check on a data stream, and determines whether the data stream is interrupted within a preset time, then, the variance calculation module 200 calculates a variance of a frequency value occurring within the preset time for the data stream that is not interrupted and appears for the first time on a current time window, and finally, the comparison module 300 compares the variance with the preset threshold, determines the data stream having the variance smaller than the preset threshold as the stability element, and finally, the failure determination module 400 determines whether the statuses of the network and the data stream are likely to fail according to the stable status of the stability element after monitoring the stable time of the stability element in real time, thereby implementing accurate detection of the stability element in a limited space and a high-speed data stream, and diagnosing whether the network or data flow state is possible to be abnormal or not according to the state of the stability element.
Furthermore, the present invention also provides a data flow state diagnostic system, which comprises a processor 10, a memory 20 and a display 30. FIG. 8 shows only some of the components of the data flow status diagnostic system, but it should be understood that not all of the shown components are required and that more or fewer components may be implemented instead.
The memory 20 may be an internal storage unit of the data stream state diagnosis system in some embodiments, such as a hard disk or a memory of the data stream state diagnosis system. The memory 20 may also be an external storage device of the data stream status diagnosing system in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the data stream status diagnosing system. Further, the memory 20 may also include both an internal storage unit and an external storage device of the data stream state diagnostic system. The memory 20 is used for storing application software installed in the data stream state diagnosis system and various types of data. The memory 20 may also be used to temporarily store data that has been output or is to be output. In an embodiment, the memory 20 stores a detection program 40 for the stability element, and the detection program 40 for the stability element can be executed by the processor 10, so as to implement the detection method for the stability element in the present invention.
The processor 10 may be a Central Processing Unit (CPU), a microprocessor or other data Processing chip in some embodiments, and is used for running the program codes stored in the memory 20 or Processing data, such as executing the detection method for the stability element.
The display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch panel, or the like in some embodiments. The display 30 is used for displaying information at the device and for displaying a visual user interface. The components 10-30 of the device communicate with each other via a system bus.
In an embodiment, the following steps are implemented when the processor 10 executes the detection program 40 for the stability element in the memory 20:
constructing a probability data structure, carrying out continuous inspection on a data stream by using the probability data structure, and judging whether the data stream is interrupted within a preset time;
if the interruption does not occur, calculating the variance of the frequency values of the data stream in the preset time;
judging whether the variance is smaller than a preset threshold value, and if so, determining that the data stream is a stability element;
and monitoring the stabilization time of the stability element in real time, and if the stability element is judged to have no stability any more according to the stabilization time, the states of the network and the data stream are failed.
If the interruption does not occur, calculating a variance of a frequency value appearing in the data stream within the preset time, specifically including:
if the data stream is not interrupted within the preset time, calculating the variance of the frequency value of the data stream recorded in the counter within the preset time to obtain a first variance;
if the frequency value is larger than the maximum count value of the counter, adding (maximum count value + 1)/2' to each frequency value, and calculating the second variance after performing modulus operation on the (maximum count value + 1) of the calculation result.
The constructing a probability data structure, performing persistence check on a data stream by using the probability data structure, and determining whether the data stream is interrupted within a preset time specifically includes:
after the probability data structure is built, mapping the data stream into a corresponding bucket in a stability filter by utilizing a hash function;
performing modulus operation on the total number of the bits in the bucket at the current moment when the data stream appears to obtain the bit corresponding to the current time window, and setting the bit corresponding to the current time window to be 1;
and judging whether a time window in the preset time has '0', if so, interrupting the data stream in the preset time, otherwise, not interrupting the data stream in the preset time.
Wherein, the judging whether the data stream is interrupted within the preset time further comprises:
and if the interruption occurs, the data stream is an unstable element, and the data stream is not processed.
Wherein, the determining whether the variance is smaller than a preset threshold, and if so, determining that the data stream is a stability element specifically includes:
comparing the magnitudes of the first variance and the second variance with the preset threshold respectively, and if any one of the first variance and the second variance is smaller than the preset threshold, determining that the data stream is a stability element.
And the preset time is the time of p time windows before the current time window.
Wherein the maximum count value = (number of bits of 2^ counter) -1.
Wherein the probabilistic data structure comprises: the stability filter, the multi-dimensional sketch structure and the stability element screener.
The stability filter is composed of d queues, wherein the d queues correspond to d hash functions, each queue contains g buckets, each bucket contains m bits, each bit corresponds to one time window, and each time window corresponds to one counter.
The multidimensional sketch structure comprises q queues, wherein the q queues correspond to q hash functions, each queue comprises j buckets, and each bucket comprises w counters.
The stability element filter is composed of k buckets, each bucket comprises x units, each unit comprises 3 areas, and id of the data stream, starting stabilization time of the data stream and ending stabilization time of the data stream are recorded respectively.
Further, the present invention provides a computer readable storage medium storing a probing program for a stability element, the probing program for a stability element implementing the steps of the probing method for a stability element as described above when executed by a processor; since the steps of the detection method for the stability element are described in detail above, no further description is given here.
In summary, the present invention provides a detection method and related device for a stability element, where the method includes the following steps: constructing a probability data structure, carrying out continuous inspection on the data stream by using the probability data structure, and judging whether the data stream is interrupted within a preset time; if the data stream is not interrupted within the preset time, calculating the variance of the frequency value of the data stream within the preset time; judging whether the variance is smaller than a preset threshold value, and if so, judging that the data stream is a stability element; and monitoring the stabilization time of the stability element in real time, and if the stability element does not have stability any more, judging that the states of the network and the data stream possibly have faults. In the invention, the data stream which is not interrupted within the preset time and the variance of the frequency value of the data stream is smaller than the preset threshold value is called as the stability element, so that the stability element in the data stream is accurately detected, and a diagnosis basis can be provided for carrying out abnormity diagnosis on the network, the data stream state and the like.
It should be understood that equivalents and modifications of the technical solution and inventive concept thereof may occur to those skilled in the art, and all such modifications and alterations should fall within the scope of the appended claims.
Claims (14)
1. A method for detecting a stability element, comprising the steps of:
constructing a probability data structure, carrying out continuous inspection on a data stream by using the probability data structure, and judging whether the data stream is interrupted within a preset time;
if the interruption does not occur, calculating the variance of the frequency values of the data stream in the preset time;
judging whether the variance is smaller than a preset threshold value, and if so, determining that the data stream is a stability element;
and monitoring the stabilization time of the stability element in real time, and if the stability element is judged to have no stability any more according to the stabilization time, the states of the network and the data stream are failed.
2. The method according to claim 1, wherein if the interruption does not occur, calculating a variance of frequency values of the data stream occurring within the preset time includes:
if the data stream is not interrupted within the preset time, calculating the variance of the frequency value of the data stream recorded in the counter within the preset time to obtain a first variance;
if the frequency value is larger than the maximum count value of the counter, adding (maximum count value + 1)/2' to each frequency value, and calculating the second variance after performing modulus operation on the (maximum count value + 1) of the calculation result.
3. The method for detecting a stability element according to claim 1, wherein the constructing a probability data structure, performing persistence check on a data stream by using the probability data structure, and determining whether an interruption of the data stream occurs within a preset time specifically includes:
after the probability data structure is built, mapping the data stream into a corresponding bucket in a stability filter by utilizing a hash function;
performing modulus operation on the total number of the bits in the bucket at the current moment when the data stream appears to obtain the bit corresponding to the current time window, and setting the bit corresponding to the current time window to be 1;
and judging whether a time window in the preset time has '0', if so, interrupting the data stream in the preset time, otherwise, not interrupting the data stream in the preset time.
4. The method for detecting the stability element according to claim 1, wherein the determining whether the data stream is interrupted within a preset time further comprises:
and if the interruption occurs, the data stream is an unstable element, and the data stream is not processed.
5. The method according to claim 2, wherein the determining whether the variance is smaller than a preset threshold, and if so, determining that the data stream is a stability element specifically includes:
comparing the magnitudes of the first variance and the second variance with the preset threshold respectively, and if any one of the first variance and the second variance is smaller than the preset threshold, determining that the data stream is a stability element.
6. The detection method for the stability element according to claim 3, wherein the preset time is the time of the first p time windows of the current time window.
7. The detection method for a stability element according to claim 2, characterized in that the maximum count value = (number of bits of 2^ counter) -1.
8. A detection method for a stability element according to claim 3, wherein the probabilistic data structure comprises: a stability filter, a multi-dimensional sketch structure and a stability element screener.
9. The method of claim 8, wherein the stability filter comprises d queues, wherein the d queues correspond to d hash functions, each queue comprises g buckets, each bucket comprises m bits, each bit corresponds to one of the time windows, and each time window corresponds to one of the counters.
10. The method for probing stability elements according to claim 8, wherein the multidimensional sketch structure comprises q queues, wherein the q queues correspond to q hash functions, each queue comprises j buckets, and each bucket comprises w counters.
11. The method of claim 8, wherein the stability element filter comprises k buckets, each bucket comprises x units, each unit comprises 3 regions, and the id of the data stream, the start stability time of the data stream, and the end stability time of the data stream are recorded.
12. A detection system for a stability element, the detection system for a stability element comprising:
the interruption detection module is used for constructing a probability data structure, carrying out continuous inspection on a data stream by using the probability data structure and judging whether the data stream is interrupted within preset time;
the variance calculation module is used for calculating the variance of the frequency value of the data stream in the preset time if the interruption does not occur;
the comparison module is used for judging whether the variance is smaller than a preset threshold value or not, and if so, determining that the data stream is a stability element;
and the fault judgment module is used for monitoring the stabilization time of the stability element in real time, and if the stability element is judged to have no stability any more according to the stabilization time, the network and the state of the data stream have faults.
13. A data flow state diagnostic system, characterized in that the data flow state diagnostic system comprises: a memory, a processor and a detection program for a stability element stored on the memory and executable on the processor, the detection program for a stability element implementing the steps of the detection method for a stability element according to any one of claims 1 to 11 when executed by the processor.
14. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a probing program for a stability element, which when executed by a processor implements the steps of the probing method for a stability element according to any of claims 1-11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210500582.4A CN114598590B (en) | 2022-05-10 | 2022-05-10 | Detection method for stability element and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210500582.4A CN114598590B (en) | 2022-05-10 | 2022-05-10 | Detection method for stability element and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114598590A true CN114598590A (en) | 2022-06-07 |
| CN114598590B CN114598590B (en) | 2022-07-12 |
Family
ID=81811678
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210500582.4A Active CN114598590B (en) | 2022-05-10 | 2022-05-10 | Detection method for stability element and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114598590B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108989133A (en) * | 2018-08-27 | 2018-12-11 | 山东大学 | Network detection optimization method based on ant group algorithm |
| CN110535825A (en) * | 2019-07-16 | 2019-12-03 | 北京大学 | A kind of data identification method of character network stream |
| WO2021229361A1 (en) * | 2020-05-14 | 2021-11-18 | Telefonaktiebolaget Lm Ericsson (Publ) | Heavy hitter flow classification based on inter-packet gap analysis |
-
2022
- 2022-05-10 CN CN202210500582.4A patent/CN114598590B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108989133A (en) * | 2018-08-27 | 2018-12-11 | 山东大学 | Network detection optimization method based on ant group algorithm |
| CN110535825A (en) * | 2019-07-16 | 2019-12-03 | 北京大学 | A kind of data identification method of character network stream |
| WO2021229361A1 (en) * | 2020-05-14 | 2021-11-18 | Telefonaktiebolaget Lm Ericsson (Publ) | Heavy hitter flow classification based on inter-packet gap analysis |
Non-Patent Citations (1)
| Title |
|---|
| 闫义博等: "大数据系统Benchmark测试综述", 《网络新媒体技术》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114598590B (en) | 2022-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7506314B2 (en) | Method for automatically collecting trace detail and history data | |
| US7603458B1 (en) | System and methods for processing and displaying aggregate status events for remote nodes | |
| US7693876B2 (en) | Monitoring usage rate patterns in storage resources | |
| CN110995482B (en) | Alarm analysis method and device, computer equipment and computer readable storage medium | |
| US7936260B2 (en) | Identifying redundant alarms by determining coefficients of correlation between alarm categories | |
| US8819220B2 (en) | Management method of computer system and management system | |
| EP2337266A2 (en) | Detecting and classifying anomalies in communication networks | |
| US6647517B1 (en) | Apparatus and method for providing error ordering information and error logging information | |
| EP4075749A1 (en) | Detection method and detection device for heavy flow data stream | |
| GB2418799A (en) | Managing network slowdown by identifying a hop along a path which exceeds a hop limit within the path. | |
| CN114124655B (en) | Network monitoring method, system, device, computer equipment and storage medium | |
| CN112486914B (en) | Data packet storage and quick-checking method and system | |
| CN113590429A (en) | Server fault diagnosis method and device and electronic equipment | |
| CN114598590B (en) | Detection method for stability element and related equipment | |
| US20140298148A1 (en) | Trend-analysis scheme for reliably reading data values from memory | |
| US8719626B2 (en) | Proactively removing channel paths in error from a variable scope of I/O devices | |
| JP2012080181A (en) | Method and program for fault information management | |
| CN117271222A (en) | Cloud disaster recovery backup method and system for big data | |
| CN111654405A (en) | Method, device, equipment and storage medium for fault node of communication link | |
| JPH10301815A (en) | Fault information logging device | |
| CN115690681A (en) | Processing method of abnormity judgment basis, abnormity judgment method and device | |
| CN112732517B (en) | Disk fault alarm method, device, equipment and readable storage medium | |
| JP2022106040A (en) | Collector device, network monitoring method, program, and network monitoring system | |
| CN112799911A (en) | Node health state detection method, device, equipment and storage medium | |
| CN112751822B (en) | Communication apparatus, operation method, abnormality determination apparatus, abnormality determination method, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |