CN111314121A - Link abnormity detection method and device - Google Patents
Link abnormity detection method and device Download PDFInfo
- Publication number
- CN111314121A CN111314121A CN202010078851.3A CN202010078851A CN111314121A CN 111314121 A CN111314121 A CN 111314121A CN 202010078851 A CN202010078851 A CN 202010078851A CN 111314121 A CN111314121 A CN 111314121A
- Authority
- CN
- China
- Prior art keywords
- screening
- flow
- service
- abnormal
- link
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 68
- 238000012216 screening Methods 0.000 claims abstract description 205
- 238000012545 processing Methods 0.000 claims abstract description 129
- 238000012544 monitoring process Methods 0.000 claims abstract description 42
- 239000011159 matrix material Substances 0.000 claims abstract description 33
- 238000004458 analytical method Methods 0.000 claims abstract description 26
- 230000002159 abnormal effect Effects 0.000 claims description 157
- 238000000034 method Methods 0.000 claims description 33
- 238000012896 Statistical algorithm Methods 0.000 claims description 31
- 238000013145 classification model Methods 0.000 claims description 24
- 238000001914 filtration Methods 0.000 claims description 21
- 238000011144 upstream manufacturing Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000003062 neural network model Methods 0.000 claims description 15
- 238000012549 training Methods 0.000 claims description 13
- 230000005856 abnormality Effects 0.000 claims description 12
- 125000004122 cyclic group Chemical group 0.000 claims description 9
- 238000007619 statistical method Methods 0.000 claims description 8
- 238000009499 grossing Methods 0.000 claims description 4
- 230000002547 anomalous effect Effects 0.000 claims 3
- 238000004422 calculation algorithm Methods 0.000 description 16
- 230000008569 process Effects 0.000 description 15
- 238000012706 support-vector machine Methods 0.000 description 11
- 230000000306 recurrent effect Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004064 recycling Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 238000000528 statistical test Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/147—Network analysis or design for predicting network behaviour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/142—Network analysis or design using statistical or mathematical methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
Abstract
The embodiment of the specification provides a link abnormity detection method and a device, wherein the link abnormity detection method comprises the following steps: acquiring flow monitoring data of each service node in a service link and link parameter information of the service link; respectively inputting a feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtaining the output predicted traffic of each service node at a target time; acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow; and carrying out exception screening processing on the elements in the flow residual error sequence by utilizing at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
Description
Technical Field
The embodiment of the specification relates to the technical field of internet, in particular to a link abnormity detection method. One or more embodiments of the present specification also relate to a link abnormality detection apparatus, a computing device, and a computer-readable storage medium.
Background
With the continuous development of internet technology, more and more services in life are converted from offline to online processing, generally, the whole processing process of different services involves multiple processing links, and multiple processing links jointly form a complete service link, while potential link abnormality often causes certain influence on normal processing of the services, once any one processing link in the service link is abnormal, other processing links in the service link may not normally perform service processing, and if more service nodes are integrated in the service link, abnormality is also likely to occur, because lost messages may not be tracked due to link abnormality, the reliability of information transmission in the link is low.
Therefore, it is desirable to provide a method for detecting link anomaly to detect an abnormal node in a link, so as to avoid affecting normal operation of a service.
Disclosure of Invention
In view of this, the embodiments of the present disclosure provide a link anomaly detection method. One or more embodiments of the present disclosure also relate to a link anomaly detection apparatus, a computing device, and a computer-readable storage medium to solve the technical problems in the prior art.
According to a first aspect of embodiments herein, there is provided a link anomaly detection method, including:
acquiring flow monitoring data of each service node in a service link and link parameter information of the service link;
respectively inputting a feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtaining the output predicted traffic of each service node at a target time;
acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow;
and carrying out exception screening processing on the elements in the flow residual error sequence by utilizing at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
Optionally, the performing, by using at least one anomaly analysis strategy, an anomaly filtering process on an element in the traffic residual sequence, and determining a service node corresponding to the element in a filtering result as an abnormal node includes:
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combining the elements in a screening result to generate a first flow residual error subsequence;
and performing exception screening processing on the elements in the first flow residual subsequence by using an exception classification model, and determining service nodes corresponding to the elements in a screening result as exception nodes.
Optionally, the performing, by using at least one anomaly analysis strategy, an anomaly filtering process on an element in the traffic residual sequence, and determining a service node corresponding to the element in a filtering result as an abnormal node includes:
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combining the elements in a screening result to generate a second flow residual error subsequence;
performing abnormal screening processing on elements in the second flow residual subsequence by using an abnormal classification model, and combining the elements in a screening result to generate a third flow residual subsequence;
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal screening rule, and combining the elements in a screening result to generate a fourth flow residual error subsequence;
determining service nodes corresponding to elements in the third flow residual subsequence and the fourth flow residual subsequence as abnormal nodes;
wherein the exception screening rule is determined according to the type of service related to the service link.
Optionally, the performing, by using an abnormal statistical algorithm, abnormal screening processing on the elements in the flow residual error sequence, and combining the elements in the screening result to generate a second flow residual error subsequence includes:
performing abnormal screening processing on elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results;
and determining an intersection of the at least two screening results, and combining elements in the intersection to generate the second flow residual subsequence.
Optionally, the inputting the feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model to perform traffic prediction, and obtaining the output predicted traffic of each service node at the target time includes:
performing characteristic processing on the traffic monitoring data of each service node and the link parameter information to obtain a characteristic matrix;
and respectively inputting the characteristic matrix into a gated cyclic neural network model for flow prediction, and acquiring the predicted flow of each service node output by the gated cyclic neural network model at the target time.
Optionally, after the step of performing an exception screening process on the elements in the traffic residual sequence by using at least one exception analysis policy and determining the service node corresponding to the element in the screening result as an exception node is performed, the method further includes:
determining the number of abnormal nodes, and calculating the proportion of the number of the abnormal nodes to the total number of the service nodes in the service link;
and under the condition that the ratio is larger than a preset threshold value, adding the real flow corresponding to the abnormal node into a training sample to form a new training sample, and optimizing the gated cyclic neural network model based on the new training sample.
Optionally, after the step of performing an exception screening process on the elements in the traffic residual sequence by using at least one exception analysis policy and determining the service node corresponding to the element in the screening result as an exception node is performed, the method further includes:
updating the real flow of the abnormal node according to the predicted flow corresponding to the abnormal node;
and carrying out deviation smoothing processing on residual error data corresponding to the abnormal node based on a sliding window, and updating the element value of an element corresponding to the abnormal node in the residual error sequence according to a processing result.
Optionally, the link anomaly detection method further includes:
and under the condition that the service node is a newly added service node, acquiring an upstream service node which is closest to the newly added service node in the service link, and taking a flow residual sequence of the upstream service node as the flow residual sequence of the newly added service node.
Optionally, the link anomaly detection method further includes:
acquiring index data of the newly added service node and other service nodes under the condition that the upstream service node of the newly added service node does not exist in the service link;
calculating the correlation degree of the index data of the newly added service node and the index data of other service nodes;
and taking the service node to which the index data with the maximum correlation degree in the calculation result belongs as a reference node, and setting the flow residual sequence of the newly added service node according to the residual sequence of the reference node.
According to a second aspect of embodiments herein, there is provided a link abnormality detection apparatus including:
the data acquisition module is configured to acquire flow monitoring data of each service node in a service link and link parameter information of the service link;
the traffic prediction module is configured to input a feature matrix obtained by performing feature processing on traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and acquire predicted traffic of each output service node at a target time;
the data calculation module is configured to acquire real flow of each service node at the target time and calculate and generate a corresponding flow residual error sequence according to the real flow and the predicted flow;
and the screening module is configured to perform exception screening processing on the elements in the traffic residual error sequence by using at least one exception analysis strategy, and determine the service nodes corresponding to the elements in the screening result as exception nodes.
Optionally, the screening module includes:
the first abnormal statistical algorithm screening submodule is configured to perform abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combine the elements in a screening result to generate a first flow residual error subsequence;
and the first abnormal classification model screening submodule is configured to perform abnormal screening processing on the elements in the first flow residual subsequence by using an abnormal classification model, and determine service nodes corresponding to the elements in the screening result as abnormal nodes.
Optionally, the screened module comprises:
the second abnormal statistical algorithm screening submodule is configured to perform abnormal screening processing on the elements in the flow residual error sequence by using an abnormal statistical algorithm, and combine the elements in the screening result to generate a second flow residual error subsequence;
the second abnormal classification model screening sub-module is configured to perform abnormal screening processing on elements in the second flow residual sub-sequence by using an abnormal classification model, and combine the elements in a screening result to generate a third flow residual sub-sequence;
the abnormal screening rule screening submodule is configured to perform abnormal screening processing on the elements in the flow residual error sequence by using an abnormal screening rule and combine the elements in the screening result to generate a fourth flow residual error subsequence;
an abnormal node determining sub-module configured to determine, as an abnormal node, a service node corresponding to an element in the third traffic residual sub-sequence and the fourth traffic residual sub-sequence;
wherein the exception screening rule is determined according to the type of service related to the service link.
Optionally, the second anomaly statistical algorithm filtering sub-module is further configured to:
performing abnormal screening processing on elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results;
and determining an intersection of the at least two screening results, and combining elements in the intersection to generate the second flow residual subsequence.
According to a third aspect of embodiments herein, there is provided a computing device comprising:
a memory and a processor;
the memory is to store computer-executable instructions, and the processor is to execute the computer-executable instructions to:
acquiring flow monitoring data of each service node in a service link and link parameter information of the service link;
respectively inputting a feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtaining the output predicted traffic of each service node at a target time;
acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow;
and carrying out exception screening processing on the elements in the flow residual error sequence by utilizing at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of any one of the link anomaly detection methods.
One embodiment of the present specification implements feature processing on traffic data of each service node in a service link, and performs feature processing on link parameter information of the service link and the monitored traffic data, inputs a feature matrix obtained by the feature processing into a traffic prediction model to predict traffic of each service node at a target time to obtain predicted traffic, and screens a traffic residual sequence generated by calculation according to the predicted traffic and actual traffic of each service node by using an anomaly analysis strategy to determine a service node that may have an anomaly in the service link, thereby ensuring timeliness of detection of an anomaly node, and effectively improving accuracy and coverage of detection of an anomaly node in the service link.
Drawings
Fig. 1 is a processing flow diagram of a link anomaly detection method according to an embodiment of the present specification;
fig. 2 is a flowchart illustrating a processing procedure of a link anomaly detection method according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of a link anomaly detection device according to an embodiment of the present specification;
fig. 4 is a block diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.
The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms to which one or more embodiments of the present specification relate are explained.
Rnn (current Neural network): is a type of neural network used to process sequence data.
Long-short term memory model (LSTM): the method is a special RNN model and is proposed for solving the problem of gradient diffusion of the RNN model.
Gated Recycling Unit (GRU): belongs to the variant structure of RNN recurrent neural network.
GESD (generated extreme determined device) algorithm: a statistical test algorithm for detecting sequence outliers.
Support Vector Machines (SVMs): the method is a generalized linear classifier which performs binary classification on data according to a supervised learning mode.
In the present specification, a link abnormality detection method is provided, and the present specification relates to a link abnormality detection apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
Fig. 1 shows a flowchart of a link anomaly detection method provided according to an embodiment of the present specification, which includes steps 102 to 108.
Step 102, obtaining flow monitoring data of each service node in a service link and link parameter information of the service link.
Due to the continuous development of internet technology, more and more services in life are converted from offline to online processing, generally, the whole processing process of different services involves multiple processing links, and multiple processing links jointly form a complete service link, while potential link abnormality often causes certain influence on normal processing of the services, and once any one processing link in the service link is abnormal, other processing links in the service link may not normally perform service processing.
Based on this, the link anomaly detection method provided in the embodiments of the present specification includes monitoring traffic data of each service node in a service link, performing feature processing on link parameter information of the service link and the monitored traffic data, inputting a feature matrix obtained by the feature processing into a traffic prediction model to predict traffic of each service node at a target time to obtain predicted traffic, and screening a traffic residual sequence generated by calculation according to the predicted traffic and a real traffic of each service node by using an anomaly analysis strategy to determine a service node in the service link that may have an anomaly, thereby ensuring timeliness of anomaly node detection and effectively improving accuracy and coverage of anomaly node detection in the service link.
Specifically, the service link refers to a link formed by a plurality of processing links of the online service; the flow monitoring data refers to historical flow data of each processing link in a service link; the link parameter information of the service link includes service information, node parameter information and the like.
Taking the service as an online movie ticket purchasing service as an example, if the movies available for users to view are movie M1 and movie M2, the service link preset in the server for the movie ticket service of online movie M1 is to select a movie (movie M1) - > select a target movie viewing location- > select a field- > click ticket purchase, and the service link preset for the movie ticket service of online movie M2 purchase is to select a movie (movie M2) - > select a target movie viewing location- > select a field- > click ticket purchase, each service link includes 4 service nodes, wherein the traffic monitoring data of the service node selecting a movie refers to the click volume of the movie M1 or the movie M2 at different times; similarly, the flow monitoring data of other service nodes refers to the click rate of each service node at different time; the service link parameter information in the service includes: service information and node parameter information, etc.
After the traffic monitoring data and the link parameter information of each service node in the service link are obtained, the traffic of each service node at the target time is predicted by using a traffic prediction model and the traffic monitoring data and the link parameter information.
And 104, inputting the characteristic matrix obtained by performing characteristic processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model to perform traffic prediction, and acquiring the output predicted traffic of each service node at a target time.
Specifically, performing feature processing on the traffic monitoring data and the link parameter information refers to extracting information more useful for traffic prediction from the traffic monitoring data and the link parameter information, and matrixing the information to obtain a corresponding feature matrix.
In addition, the traffic prediction model in the embodiment of the present specification includes a gated recurrent neural network model, and the traffic of the target time of each service node is predicted by using the gated recurrent neural network model, which may be specifically implemented by the following means:
performing characteristic processing on the traffic monitoring data of each service node and the link parameter information to obtain a characteristic matrix;
and respectively inputting the characteristic matrix into a gated cyclic neural network model for flow prediction, and acquiring the predicted flow of each service node output by the gated cyclic neural network model at the target time.
In specific implementation, the gated recurrent neural network model is specifically a Gated Recurrent Unit (GRU), the GRU belongs to a variant structure of the RNN recurrent neural network, and only has one reset gate rtAnd an update gate ztWherein the door z is updatedtDetermines the information to be discarded and the new information to be added, i.e. the degree to which the state information for controlling the previous moment is brought into the current state, updates the gate ztThe larger the value of (b), the more state information is brought in at the previous time. Reset door rtControlling how much information was written to h in the previous statet(predicted value of output) on, reset gate rtThe smaller the less information of the previous state is written.
After the traffic monitoring data of each service node and the link parameter information are obtained, the data and the information are subjected to feature processing to obtain a feature matrix, and after the feature matrix is input into the GRU model, the GRU model processes the data in a forward propagation mode and outputs a traffic prediction result.
Taking the service as an example for purchasing a movie ticket service on line, acquiring flow monitoring data of 4 service nodes in the service at 30 times (t-29 to t respectively) in a day and link parameter information of a service link, performing feature processing on the flow monitoring data and the link parameter information to generate a 30 x 1 order feature matrix, and inputting the feature matrix into a GRU model to obtain predicted flow h of each service node output by the model at the time ttWherein x of the GRU model is inputt-29To xtCorresponding to the data in rows 1 to 30 in the 30 x 1 th order feature matrix, respectively.
After the predicted flow of each service node at the target time is obtained, the abnormal nodes can be screened by combining the real flow of each service node at the moment.
The GRU model reserves important characteristics through various gate functions, can effectively reduce the probability of data loss in the long-term forward propagation process, and improves the reliability of information transmission in a link. And the convergence time of GRU model training is short, so that the model can be updated in a real-time computing system in an iterative manner, the model can be updated in time to adapt to the change of flow distribution, and the accuracy of a prediction result is kept at a high level.
And 106, acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow.
Specifically, the target time may be a certain time t, or may be a plurality of times (t, t +1,. eta.. eta., respectively); since the residual is the difference between the true value and the predicted value (fitting value), the flow residual sequence is composed of a plurality of residuals, and each residual is an element in the flow residual sequence.
After the predicted flow of each service node at the target time is obtained, the residual error of each service node can be calculated by combining the real flow and the predicted flow of each service node at the moment, a flow residual error sequence is generated, and abnormal nodes are screened in a mode of screening abnormal elements in the residual error sequence.
Still taking the service as an example of online purchasing movie ticket service, and the target time is a certain time t, the movies available for the user to view are movie M1 and movie M2, then the preset service links in the server are selecting movie (movie M1) - > selecting target viewing location- > selecting field- > clicking ticket purchasing, and selecting movie (movie M2) - > selecting target viewing location- > selecting field- > clicking ticket purchasing, for the service link: selecting a film (film M1) - > selecting a target film viewing place- > selecting a field- > clicking to buy tickets, acquiring flow monitoring data of 4 service nodes in a service link and link parameter information of the service link, respectively carrying out feature processing on the flow monitoring data of the 4 service nodes and the link parameter information of the service link to generate a feature matrix, inputting the characteristic matrix into GRU model to obtain predicted flow rates of 4 service nodes at time t of A1, A2, A3 and A4, then obtaining the real flow rates of the 4 service nodes at the time t as B1, B2, B3 and B4 respectively, taking the absolute value of the difference value between the real value and the predicted value corresponding to the 4 service nodes as respective residual error, and generating flow residual error sequences as { | A1-B1|, | A2-B2|, | A3-B3|, | A4-B4| }; and (3) service link: the process of selecting a movie (movie M2) — > selecting a target viewing location- > selecting a field- > clicking for ticket purchase is similar to the process described above, and is not described in detail herein.
And after the flow residual error sequence is obtained, an abnormal service node in the service link is determined by screening abnormal elements in the flow residual error sequence.
And 108, performing exception screening processing on the elements in the flow residual error sequence by using at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
Specifically, the anomaly filtering strategy is a filtering strategy for filtering an element with an anomaly in the traffic residual sequence, and the anomaly analysis strategy used in the embodiment of the present specification includes at least one of three strategies, namely an anomaly statistical algorithm, an anomaly classification model, and an anomaly filtering rule.
In an implementation manner provided by the embodiment of the present specification, an anomaly statistical algorithm and an anomaly classification model are used to perform anomaly screening processing on elements in a flow residual sequence, which can be specifically implemented in the following manner:
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combining the elements in a screening result to generate a first flow residual error subsequence;
and performing exception screening processing on the elements in the first flow residual subsequence by using an exception classification model, and determining service nodes corresponding to the elements in a screening result as exception nodes.
Specifically, the anomaly statistical algorithm includes a GESD algorithm, a boxcar graph algorithm, a standard deviation algorithm, and the like, the anomaly classification model includes a Support Vector Machine (SVM) model, the screening is performed by using the anomaly statistical algorithm to obtain a first screening result, the SVM model is used to perform secondary screening on the first screening result to obtain a second screening result, an element in the second screening result is determined to be an abnormal element, and a service node corresponding to the abnormal element is determined to be a service node.
In addition, the abnormal statistical algorithm includes a GESD algorithm, a boxplot algorithm, a standard deviation algorithm, and the like, and the embodiment of the present specification performs abnormal screening processing on elements in the flow residual sequence by using the three algorithms to obtain three screening results, and generates the first screening result by taking an intersection of the three screening results.
The SVM is used for carrying out secondary screening on the primary screening result, so that the accuracy of the detection result is further improved, the SVM model can be used for detecting the trainable model without large sample size, and better performance can be improved.
In another implementation manner provided in the embodiment of the present specification, performing exception screening processing on elements in a flow residual sequence by using an exception statistical algorithm, an exception classification model, and an exception screening rule may specifically be implemented by:
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combining the elements in a screening result to generate a second flow residual error subsequence;
performing abnormal screening processing on elements in the second flow residual subsequence by using an abnormal classification model, and combining the elements in a screening result to generate a third flow residual subsequence;
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal screening rule, and combining the elements in a screening result to generate a fourth flow residual error subsequence;
determining service nodes corresponding to elements in the third flow residual subsequence and the fourth flow residual subsequence as abnormal nodes;
wherein the exception screening rule is determined according to the type of service related to the service link.
Similar to the foregoing embodiment, in this embodiment, the anomaly statistical algorithm also includes a GESD algorithm, a boxplot algorithm, a standard deviation algorithm, and the like, the anomaly classification model includes a support vector machine model, and the anomaly screening rule refers to a screening rule set according to different service conditions to perform auxiliary screening;
screening elements in the flow residual sequence by using an abnormal statistical algorithm to obtain a first screening result, carrying out secondary screening on the first screening result by using an SVM model to obtain a second screening result, screening the elements in the flow residual sequence by using an abnormal screening rule again to obtain a third screening result, determining the elements in the second screening result and the third screening result as abnormal elements, and determining a service node corresponding to the abnormal element as a service node.
In specific implementation, the exception screening rule is determined according to different service types, for example, if service data in a service periodically changes, the exception screening rule may be determined as: determining the elements with the ring ratio or the same ratio threshold value larger than the threshold value as abnormal elements; if a server node for processing service data in a service has a preset resource load range, the exception screening rule may be determined as: and determining the elements with the element values larger than the threshold value as abnormal elements.
Different screening rules are set according to different service types for auxiliary screening, which is beneficial to further improving the accuracy of the detection result.
The SVM is used for carrying out secondary screening on the primary screening result, so that the accuracy of the detection result is further improved, the SVM model can be used for detecting the trainable model without large sample size, and better performance can be improved.
In the embodiment of the present description, at least two algorithms, namely a GESD algorithm, a boxplot algorithm, and a standard deviation algorithm, are used for exception screening, which can be specifically implemented in the following manner:
performing abnormal screening processing on elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results;
and determining an intersection of the at least two screening results, and combining elements in the intersection to generate the second flow residual subsequence.
Still taking the service as an example of online purchasing of movie tickets, the flow residual sequence generated by calculation according to the real flow and the predictive flow is { | a1-B1|, | a2-B2|, | A3-B3|, | a4-B4| }, if two statistical methods are used for carrying out abnormal screening processing to obtain two corresponding screening results { | a1-B1|, | a2-B2|, | A3-B3| } and { | a1-B1|, | a2-B2|, | a4-B4| }, the determined second flow residual sub-sequence for the two screening results is | a1-B1|, | a2-B2| }.
Further, after the element in the traffic residual sequence is subjected to exception screening processing by using at least one exception analysis strategy, and the service node corresponding to the element in the screening result is determined as an exception node, in order to avoid that the exception node affects the screening of other service nodes, the related data of the exception node needs to be updated, which can be specifically realized by the following method:
updating the real flow of the abnormal node according to the predicted flow corresponding to the abnormal node;
and carrying out deviation smoothing processing on residual error data corresponding to the abnormal node based on a sliding window, and updating the element value of an element corresponding to the abnormal node in the residual error sequence according to a processing result.
In addition, when a new service node exists in the service link, a traffic residual sequence is also required to be used in the process of screening the new service node, and the traffic residual sequence of the new service node is determined in the following manner:
and under the condition that the service node is a newly added service node, acquiring an upstream service node which is closest to the newly added service node in the service link, and taking a flow residual sequence of the upstream service node as the flow residual sequence of the newly added service node.
Following the above example, for a traffic link: selecting a film (film M1) - > selecting a target film watching place- > selecting a field- > clicking to purchase a ticket, and if a new service node ' confirmation payment ' exists after the service node ' clicking to purchase the ticket, forming a new service link for the service link: select movie (movie M1) - > select target viewing location- > select field- > click buy ticket- > confirm payment, the process of performing anomaly detection on the first 4 service nodes in the new service link is as described above, and is not described herein again, because the traffic monitoring data (historical traffic data) of the newly added service node cannot be acquired, a plurality of non-abnormal residual data of the service node, which is a click ticket purchase (an upstream service node closest to the confirmed payment service node) in the new service link, are used as the residual sequence of the confirmed payment service node, when the traffic residual error detection method is used for carrying out abnormality detection on the traffic residual error detection nodes, a corresponding number of residual error data can be selected according to the length of target time to be used as elements to be added to the traffic residual error sequence, and abnormal service nodes are screened in a mode of screening abnormal elements in the traffic residual error sequence.
Under the condition that the newly-added service node has no historical flow data, the non-abnormal residual data of the upstream service node closest to the newly-added service node is used as the residual sequence of the newly-added service node, so that the newly-added service node is subjected to abnormal screening processing through the residual sequence, and the accuracy and the coverage rate of abnormal node detection in a service link are improved.
Corresponding to the foregoing embodiment, in the case that there is no upstream node of the newly added service node in the service link, the flow residual sequence of the newly added service node is determined in the following manner:
acquiring index data of the newly added service node and other service nodes under the condition that the upstream service node of the newly added service node does not exist in the service link;
calculating the correlation degree of the index data of the newly added service node and the index data of other service nodes;
and taking the service node to which the index data with the maximum correlation degree in the calculation result belongs as a reference node, and setting the flow residual sequence of the newly added service node according to the residual sequence of the reference node.
Along with the above example, if a movie M3 is newly added to a movie available for users to watch, in addition to the movie M1 and the movie M2, a service link set in the server for online movie ticket service of purchasing the movie M3 is to select a movie (movie M3) - > select a target movie watching location- > select a field- > click to purchase tickets, and abnormality detection needs to be performed on 4 service nodes in the service link, because the service node "select movie (movie M3)" is a newly added service node, and there is no upstream service node closest to the service node in the service link, index data of each service node in "select movie (movie M3)" and other service links need to be acquired; and calculating the correlation between the selected film (film M3) and each service node according to the index data, if the calculation result is that the correlation between the selected film (film M3) and the selected film (film M1) is the largest, and the correlation value is 95% and is greater than a preset correlation threshold value of 90%, taking the service node selected film (film M1) as a reference node, and setting a residual sequence of the selected film (film M3) according to non-abnormal residual data of the selected film (film M1).
Under the condition that the newly added service node has no historical flow data, the service node with the maximum correlation with the newly added service node is used as a reference node, a residual sequence is set for the newly added service node, and the newly added service node is subjected to exception screening processing through the residual sequence, so that the accuracy and the coverage rate of exception node detection in a service link are improved.
In addition, after the step of performing an exception screening process on the elements in the traffic residual sequence by using at least one exception analysis strategy and determining the service node corresponding to the element in the screening result as an exception node is performed, the method further includes:
determining the number of abnormal nodes, and calculating the proportion of the number of the abnormal nodes to the total number of the service nodes in the service link;
and under the condition that the ratio is larger than a preset threshold value, adding the real flow corresponding to the abnormal node into a training sample to form a new training sample, and optimizing the gated cyclic neural network model based on the new training sample.
Specifically, under the condition that the ratio of the abnormal node to the total number of the nodes in the service link is determined to be greater than the preset threshold according to the screening result, the real traffic corresponding to the abnormal node is used for optimizing the GRU model so as to improve the accuracy of the prediction result output by the model.
One embodiment of the present specification implements feature processing on traffic data of each service node in a service link, and performs feature processing on link parameter information of the service link and the monitored traffic data, inputs a feature matrix obtained by the feature processing into a traffic prediction model to predict traffic of each service node at a target time to obtain predicted traffic, and screens a traffic residual sequence generated by calculation according to the predicted traffic and actual traffic of each service node by using an anomaly analysis strategy to determine a service node that may have an anomaly in the service link, thereby ensuring timeliness of detection of an anomaly node, and effectively improving accuracy and coverage of detection of an anomaly node in the service link.
The following describes the link anomaly detection method further by taking a specific application of the link anomaly detection method provided in this specification as an example, with reference to fig. 2. Fig. 2 shows a flowchart of a processing procedure of a link anomaly detection method according to an embodiment of the present specification, and specific steps include step 202 to step 216.
And step 206, acquiring the real flow of each service node at the target moment, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow.
And 208, performing abnormal screening processing on the elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results.
Step 210, determining an intersection of the at least two screening results, and combining elements in the intersection to generate the flow residual subsequence.
And 212, performing abnormal screening processing on the elements in the flow residual subsequence by using an abnormal classification model, and combining the elements in the screening result to generate a first screening processing result.
And 214, performing abnormal screening processing on the elements in the flow residual sequence by using an abnormal screening rule, and combining the elements in the screening result to generate a second screening processing result.
One embodiment of the present specification implements feature processing on traffic data of each service node in a service link, and performs feature processing on link parameter information of the service link and the monitored traffic data, inputs a feature matrix obtained by the feature processing into a traffic prediction model to predict traffic of each service node at a target time to obtain predicted traffic, and screens a traffic residual sequence generated by calculation according to the predicted traffic and actual traffic of each service node by using an anomaly analysis strategy to determine a service node that may have an anomaly in the service link, thereby ensuring timeliness of detection of an anomaly node, and effectively improving accuracy and coverage of detection of an anomaly node in the service link.
Corresponding to the above method embodiment, the present specification further provides an embodiment of a link anomaly detection apparatus, and fig. 3 shows a schematic structural diagram of a link anomaly detection apparatus provided in an embodiment of the present specification. As shown in fig. 3, the apparatus includes:
a data obtaining module 302, configured to obtain traffic monitoring data of each service node in a service link and link parameter information of the service link;
a traffic prediction module 304, configured to input a feature matrix obtained by performing feature processing on traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtain predicted traffic of each output service node at a target time;
a data calculation module 306, configured to obtain real traffic of each service node at the target time, and calculate and generate a corresponding traffic residual sequence according to the real traffic and the predicted traffic;
and the screening module 308 is configured to perform exception screening processing on the elements in the traffic residual sequence by using at least one exception analysis strategy, and determine service nodes corresponding to the elements in the screening result as exception nodes.
Optionally, the screening module 308 includes:
the first abnormal statistical algorithm screening submodule is configured to perform abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combine the elements in a screening result to generate a first flow residual error subsequence;
and the first abnormal classification model screening submodule is configured to perform abnormal screening processing on the elements in the first flow residual subsequence by using an abnormal classification model, and determine service nodes corresponding to the elements in the screening result as abnormal nodes.
Optionally, the filtered module 308 includes:
the second abnormal statistical algorithm screening submodule is configured to perform abnormal screening processing on the elements in the flow residual error sequence by using an abnormal statistical algorithm, and combine the elements in the screening result to generate a second flow residual error subsequence;
the second abnormal classification model screening sub-module is configured to perform abnormal screening processing on elements in the second flow residual sub-sequence by using an abnormal classification model, and combine the elements in a screening result to generate a third flow residual sub-sequence;
the abnormal screening rule screening submodule is configured to perform abnormal screening processing on the elements in the flow residual error sequence by using an abnormal screening rule and combine the elements in the screening result to generate a fourth flow residual error subsequence;
an abnormal node determining sub-module configured to determine, as an abnormal node, a service node corresponding to an element in the third traffic residual sub-sequence and the fourth traffic residual sub-sequence;
wherein the exception screening rule is determined according to the type of service related to the service link.
Optionally, the second anomaly statistical algorithm filtering sub-module is further configured to:
performing abnormal screening processing on elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results;
and determining an intersection of the at least two screening results, and combining elements in the intersection to generate the second flow residual subsequence.
Optionally, the flow prediction module 304 includes:
the characteristic processing submodule is configured to perform characteristic processing on the traffic monitoring data of each service node and the link parameter information to obtain a characteristic matrix;
and the flow prediction sub-module is configured to input the characteristic matrix into a gated recurrent neural network model for flow prediction, and acquire the predicted flow of each service node output by the gated recurrent neural network model at the target time.
Optionally, the link anomaly detection apparatus further includes:
the proportion calculation module is configured to determine the number of abnormal nodes and calculate the proportion of the number of the abnormal nodes to the total number of the service nodes in the service link;
and the model optimization module is configured to add the real traffic corresponding to the abnormal node to a training sample to form a new training sample and optimize the gated recurrent neural network model based on the new training sample under the condition that the ratio is larger than a preset threshold.
Optionally, the link anomaly detection apparatus further includes:
the first data updating module is configured to update the real traffic of the abnormal node according to the predicted traffic corresponding to the abnormal node;
and the second data updating module is configured to perform deviation smoothing processing on residual error data corresponding to the abnormal node based on a sliding window, and update the element value of an element corresponding to the abnormal node in the residual error sequence according to a processing result.
Optionally, the link anomaly detection apparatus further includes:
and the flow residual sequence determining module is configured to acquire an upstream service node closest to the newly added service node in the service link and use the flow residual sequence of the upstream service node as the flow residual sequence of the newly added service node under the condition that the service node is the newly added service node.
Optionally, the link anomaly detection apparatus further includes:
the index data acquisition module is configured to acquire the index data of the newly added service node and other service nodes under the condition that the upstream service node of the newly added service node does not exist in the service link;
the relevancy calculation module is configured to calculate the relevancy between the index data of the newly added service node and the index data of other service nodes;
and the flow residual sequence setting module is configured to take the service node to which the index data with the maximum correlation degree in the calculation result belongs as a reference node, and set the flow residual sequence of the newly added service node according to the residual sequence of the reference node.
In an embodiment of the present description, different anomaly analysis strategies are used to screen abnormal elements in a traffic residual sequence to determine a service node in a service link that may have an anomaly, so that timeliness of abnormal node detection is ensured, and accuracy and coverage rate of abnormal node detection in the service link are effectively improved.
The above is a schematic scheme of a link abnormality detection apparatus of the present embodiment. It should be noted that the technical solution of the link anomaly detection apparatus and the technical solution of the link anomaly detection method belong to the same concept, and details of the technical solution of the link anomaly detection apparatus, which are not described in detail, can be referred to the description of the technical solution of the link anomaly detection method.
FIG. 4 illustrates a block diagram of a computing device 400 provided in accordance with one embodiment of the present description. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. Processor 420 is coupled to memory 410 via bus 430 and database 450 is used to store data.
Computing device 400 also includes access device 440, access device 440 enabling computing device 400 to communicate via one or more networks 460. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 440 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present description, the above-described components of computing device 400, as well as other components not shown in FIG. 4, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 4 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.
Computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 400 may also be a mobile or stationary server.
Wherein the memory 410 is configured to store computer-executable instructions and the processor 420 is configured to execute the following computer-executable instructions:
acquiring flow monitoring data of each service node in a service link and link parameter information of the service link;
respectively inputting a feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtaining the output predicted traffic of each service node at a target time;
acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow;
and carrying out exception screening processing on the elements in the flow residual error sequence by utilizing at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned link anomaly detection method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above-mentioned link anomaly detection method.
An embodiment of the present specification also provides a computer readable storage medium storing computer instructions, which when executed by a processor, are used for implementing the steps of the link anomaly detection method.
The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above-mentioned link anomaly detection method belong to the same concept, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the above-mentioned link anomaly detection method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for an embodiment of the specification.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.
Claims (15)
1. A link anomaly detection method includes:
acquiring flow monitoring data of each service node in a service link and link parameter information of the service link;
respectively inputting a feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtaining the output predicted traffic of each service node at a target time;
acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow;
and carrying out exception screening processing on the elements in the flow residual error sequence by utilizing at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
2. The method for detecting link anomalies according to claim 1, wherein the performing anomaly filtering processing on the elements in the traffic residual error sequence by using at least one anomaly analysis strategy and determining the service node corresponding to the element in the filtering result as an anomalous node includes:
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combining the elements in a screening result to generate a first flow residual error subsequence;
and performing exception screening processing on the elements in the first flow residual subsequence by using an exception classification model, and determining service nodes corresponding to the elements in a screening result as exception nodes.
3. The method for detecting link anomalies according to claim 1, wherein the performing anomaly filtering processing on the elements in the traffic residual error sequence by using at least one anomaly analysis strategy and determining the service node corresponding to the element in the filtering result as an anomalous node includes:
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combining the elements in a screening result to generate a second flow residual error subsequence;
performing abnormal screening processing on elements in the second flow residual subsequence by using an abnormal classification model, and combining the elements in a screening result to generate a third flow residual subsequence;
performing abnormal screening processing on elements in the flow residual error sequence by using an abnormal screening rule, and combining the elements in a screening result to generate a fourth flow residual error subsequence;
determining service nodes corresponding to elements in the third flow residual subsequence and the fourth flow residual subsequence as abnormal nodes;
wherein the exception screening rule is determined according to the type of service related to the service link.
4. The link anomaly detection method according to claim 3, wherein the performing anomaly filtering processing on the elements in the traffic residual error sequence by using an anomaly statistical algorithm and combining the elements in the filtering result to generate a second traffic residual error subsequence includes:
performing abnormal screening processing on elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results;
and determining an intersection of the at least two screening results, and combining elements in the intersection to generate the second flow residual subsequence.
5. The method for detecting link anomaly according to claim 1, wherein the inputting of a feature matrix obtained by performing feature processing on traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction and obtaining of predicted traffic of each output service node at a target time includes:
performing characteristic processing on the traffic monitoring data of each service node and the link parameter information to obtain a characteristic matrix;
and respectively inputting the characteristic matrix into a gated cyclic neural network model for flow prediction, and acquiring the predicted flow of each service node output by the gated cyclic neural network model at the target time.
6. The method for detecting link anomalies according to claim 5, wherein after the step of performing anomaly filtering processing on the elements in the traffic residual error sequence by using at least one anomaly analysis strategy and determining the service node corresponding to the element in the filtering result as an anomalous node is performed, the method further includes:
determining the number of abnormal nodes, and calculating the proportion of the number of the abnormal nodes to the total number of the service nodes in the service link;
and under the condition that the ratio is larger than a preset threshold value, adding the real flow corresponding to the abnormal node into a training sample to form a new training sample, and optimizing the gated cyclic neural network model based on the new training sample.
7. The method according to claim 1, wherein after the step of performing anomaly filtering on the elements in the traffic residual error sequence by using at least one anomaly analysis strategy and determining the service node corresponding to the element in the filtering result as the abnormal node is performed, the method further comprises:
updating the real flow of the abnormal node according to the predicted flow corresponding to the abnormal node;
and carrying out deviation smoothing processing on residual error data corresponding to the abnormal node based on a sliding window, and updating the element value of an element corresponding to the abnormal node in the residual error sequence according to a processing result.
8. The link anomaly detection method according to claim 1, further comprising:
and under the condition that the service node is a newly added service node, acquiring an upstream service node which is closest to the newly added service node in the service link, and taking a flow residual sequence of the upstream service node as the flow residual sequence of the newly added service node.
9. The link anomaly detection method according to claim 8, further comprising:
acquiring index data of the newly added service node and other service nodes under the condition that the upstream service node of the newly added service node does not exist in the service link;
calculating the correlation degree of the index data of the newly added service node and the index data of other service nodes;
and taking the service node to which the index data with the maximum correlation degree in the calculation result belongs as a reference node, and setting the flow residual sequence of the newly added service node according to the residual sequence of the reference node.
10. A link abnormality detection apparatus comprising:
the data acquisition module is configured to acquire flow monitoring data of each service node in a service link and link parameter information of the service link;
the traffic prediction module is configured to input a feature matrix obtained by performing feature processing on traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and acquire predicted traffic of each output service node at a target time;
the data calculation module is configured to acquire real flow of each service node at the target time and calculate and generate a corresponding flow residual error sequence according to the real flow and the predicted flow;
and the screening module is configured to perform exception screening processing on the elements in the traffic residual error sequence by using at least one exception analysis strategy, and determine the service nodes corresponding to the elements in the screening result as exception nodes.
11. The link anomaly detection device according to claim 10, said screening module comprising:
the first abnormal statistical algorithm screening submodule is configured to perform abnormal screening processing on elements in the flow residual error sequence by using an abnormal statistical algorithm, and combine the elements in a screening result to generate a first flow residual error subsequence;
and the first abnormal classification model screening submodule is configured to perform abnormal screening processing on the elements in the first flow residual subsequence by using an abnormal classification model, and determine service nodes corresponding to the elements in the screening result as abnormal nodes.
12. The link anomaly detection device according to claim 10, said filtering module comprising:
the second abnormal statistical algorithm screening submodule is configured to perform abnormal screening processing on the elements in the flow residual error sequence by using an abnormal statistical algorithm, and combine the elements in the screening result to generate a second flow residual error subsequence;
the second abnormal classification model screening sub-module is configured to perform abnormal screening processing on elements in the second flow residual sub-sequence by using an abnormal classification model, and combine the elements in a screening result to generate a third flow residual sub-sequence;
the abnormal screening rule screening submodule is configured to perform abnormal screening processing on the elements in the flow residual error sequence by using an abnormal screening rule and combine the elements in the screening result to generate a fourth flow residual error subsequence;
an abnormal node determining sub-module configured to determine, as an abnormal node, a service node corresponding to an element in the third traffic residual sub-sequence and the fourth traffic residual sub-sequence;
wherein the exception screening rule is determined according to the type of service related to the service link.
13. The link anomaly detection device of claim 12, said second anomaly statistical algorithm screening submodule further configured to:
performing abnormal screening processing on elements in the flow residual error sequence by using at least two statistical methods to obtain at least two corresponding screening results;
and determining an intersection of the at least two screening results, and combining elements in the intersection to generate the second flow residual subsequence.
14. A computing device, comprising:
a memory and a processor;
the memory is to store computer-executable instructions, and the processor is to execute the computer-executable instructions to:
acquiring flow monitoring data of each service node in a service link and link parameter information of the service link;
respectively inputting a feature matrix obtained by performing feature processing on the traffic monitoring data of each service node and the link parameter information into a traffic prediction model for traffic prediction, and obtaining the output predicted traffic of each service node at a target time;
acquiring the real flow of each service node at the target time, and calculating and generating a corresponding flow residual error sequence according to the real flow and the predicted flow;
and carrying out exception screening processing on the elements in the flow residual error sequence by utilizing at least one exception analysis strategy, and determining the service nodes corresponding to the elements in the screening result as exception nodes.
15. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the link anomaly detection method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010078851.3A CN111314121A (en) | 2020-02-03 | 2020-02-03 | Link abnormity detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010078851.3A CN111314121A (en) | 2020-02-03 | 2020-02-03 | Link abnormity detection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111314121A true CN111314121A (en) | 2020-06-19 |
Family
ID=71158184
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010078851.3A Pending CN111314121A (en) | 2020-02-03 | 2020-02-03 | Link abnormity detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111314121A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112291302A (en) * | 2020-09-28 | 2021-01-29 | 北京京东尚科信息技术有限公司 | Internet of things equipment behavior data analysis method and processing system |
| CN112333045A (en) * | 2020-11-03 | 2021-02-05 | 国家工业信息安全发展研究中心 | Intelligent flow baseline learning method, equipment and computer readable storage medium |
| CN112532439A (en) * | 2020-11-24 | 2021-03-19 | 山东科技大学 | Network flow prediction method based on attention multi-component space-time cross-domain neural network model |
| CN112822046A (en) * | 2021-01-04 | 2021-05-18 | 新华三大数据技术有限公司 | Flow prediction method and device |
| CN113296991A (en) * | 2020-11-16 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Abnormality detection method and apparatus |
| CN113868111A (en) * | 2021-12-06 | 2021-12-31 | 金锐同创(北京)科技股份有限公司 | Mutual access relation adjusting method and device based on service node and electronic equipment |
| CN114584391A (en) * | 2022-03-22 | 2022-06-03 | 恒安嘉新（北京）科技股份公司 | Method, device, equipment and storage medium for generating abnormal flow processing strategy |
| CN115118631A (en) * | 2022-06-27 | 2022-09-27 | 平安银行股份有限公司 | Link exception handling method and device, electronic equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101686235A (en) * | 2008-09-26 | 2010-03-31 | 中联绿盟信息技术(北京)有限公司 | Device and method for analyzing abnormal network flow |
| CN103532776A (en) * | 2013-09-30 | 2014-01-22 | 广东电网公司电力调度控制中心 | Service flow detection method and system |
| CN105471854A (en) * | 2015-11-18 | 2016-04-06 | 国网智能电网研究院 | Adaptive boundary abnormity detection method based on multistage strategies |
| US20160323163A1 (en) * | 2013-12-19 | 2016-11-03 | Bae Systems Plc | Method and apparatus for detecting fault conditions in a network |
| CN107483251A (en) * | 2017-08-22 | 2017-12-15 | 国网辽宁省电力有限公司辽阳供电公司 | A kind of Network exception detecting method based on the monitoring of distributed probe |
| CN108322347A (en) * | 2018-02-09 | 2018-07-24 | 腾讯科技（深圳）有限公司 | Data detection method, device, detection service device and storage medium |
| CN108833184A (en) * | 2018-06-29 | 2018-11-16 | 腾讯科技（深圳）有限公司 | Service fault localization method, device, computer equipment and storage medium |
| CN108880838A (en) * | 2017-05-10 | 2018-11-23 | 阿里巴巴集团控股有限公司 | Monitoring method and device, the computer equipment and readable medium of traffic failure |
| CN110247910A (en) * | 2019-06-13 | 2019-09-17 | 深信服科技股份有限公司 | A kind of detection method of abnormal flow, system and associated component |
| CN110647447A (en) * | 2019-08-01 | 2020-01-03 | 百度时代网络技术（北京）有限公司 | Abnormal instance detection method, apparatus, device and medium for distributed system |
-
2020
- 2020-02-03 CN CN202010078851.3A patent/CN111314121A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101686235A (en) * | 2008-09-26 | 2010-03-31 | 中联绿盟信息技术(北京)有限公司 | Device and method for analyzing abnormal network flow |
| CN103532776A (en) * | 2013-09-30 | 2014-01-22 | 广东电网公司电力调度控制中心 | Service flow detection method and system |
| US20160323163A1 (en) * | 2013-12-19 | 2016-11-03 | Bae Systems Plc | Method and apparatus for detecting fault conditions in a network |
| CN105471854A (en) * | 2015-11-18 | 2016-04-06 | 国网智能电网研究院 | Adaptive boundary abnormity detection method based on multistage strategies |
| CN108880838A (en) * | 2017-05-10 | 2018-11-23 | 阿里巴巴集团控股有限公司 | Monitoring method and device, the computer equipment and readable medium of traffic failure |
| CN107483251A (en) * | 2017-08-22 | 2017-12-15 | 国网辽宁省电力有限公司辽阳供电公司 | A kind of Network exception detecting method based on the monitoring of distributed probe |
| CN108322347A (en) * | 2018-02-09 | 2018-07-24 | 腾讯科技（深圳）有限公司 | Data detection method, device, detection service device and storage medium |
| CN108833184A (en) * | 2018-06-29 | 2018-11-16 | 腾讯科技（深圳）有限公司 | Service fault localization method, device, computer equipment and storage medium |
| CN110247910A (en) * | 2019-06-13 | 2019-09-17 | 深信服科技股份有限公司 | A kind of detection method of abnormal flow, system and associated component |
| CN110647447A (en) * | 2019-08-01 | 2020-01-03 | 百度时代网络技术（北京）有限公司 | Abnormal instance detection method, apparatus, device and medium for distributed system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112291302A (en) * | 2020-09-28 | 2021-01-29 | 北京京东尚科信息技术有限公司 | Internet of things equipment behavior data analysis method and processing system |
| CN112333045A (en) * | 2020-11-03 | 2021-02-05 | 国家工业信息安全发展研究中心 | Intelligent flow baseline learning method, equipment and computer readable storage medium |
| CN113296991A (en) * | 2020-11-16 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Abnormality detection method and apparatus |
| CN113296991B (en) * | 2020-11-16 | 2024-02-13 | 阿里巴巴集团控股有限公司 | Abnormality detection method and device |
| CN112532439A (en) * | 2020-11-24 | 2021-03-19 | 山东科技大学 | Network flow prediction method based on attention multi-component space-time cross-domain neural network model |
| CN112822046A (en) * | 2021-01-04 | 2021-05-18 | 新华三大数据技术有限公司 | Flow prediction method and device |
| CN113868111A (en) * | 2021-12-06 | 2021-12-31 | 金锐同创(北京)科技股份有限公司 | Mutual access relation adjusting method and device based on service node and electronic equipment |
| CN114584391A (en) * | 2022-03-22 | 2022-06-03 | 恒安嘉新（北京）科技股份公司 | Method, device, equipment and storage medium for generating abnormal flow processing strategy |
| CN114584391B (en) * | 2022-03-22 | 2024-02-09 | 恒安嘉新（北京）科技股份公司 | Method, device, equipment and storage medium for generating abnormal flow processing strategy |
| CN115118631A (en) * | 2022-06-27 | 2022-09-27 | 平安银行股份有限公司 | Link exception handling method and device, electronic equipment and storage medium |
| CN115118631B (en) * | 2022-06-27 | 2023-09-01 | 平安银行股份有限公司 | Link exception handling method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111314121A (en) | Link abnormity detection method and device | |
| Yoon et al. | Data valuation using reinforcement learning | |
| WO2019114344A1 (en) | Graphical structure model-based method for prevention and control of abnormal accounts, and device and equipment | |
| CN113905391B (en) | Integrated learning network traffic prediction method, system, equipment, terminal and medium | |
| CN114285728B (en) | Predictive model training method, traffic prediction device and storage medium | |
| US11790369B2 (en) | Systems and method for enhanced active machine learning through processing of partitioned uncertainty | |
| WO2023279674A1 (en) | Memory-augmented graph convolutional neural networks | |
| CN110929785A (en) | Data classification method and device, terminal equipment and readable storage medium | |
| Fang et al. | SDGNet: A handover-aware spatiotemporal graph neural network for mobile traffic forecasting | |
| Xia et al. | Deciphering spatio-temporal graph forecasting: A causal lens and treatment | |
| Murugesan et al. | Active learning from peers | |
| EP4343616A1 (en) | Image classification method, model training method, device, storage medium, and computer program | |
| Ji et al. | Multi-range gated graph neural network for telecommunication fraud detection | |
| Ngo et al. | Adaptive anomaly detection for internet of things in hierarchical edge computing: A contextual-bandit approach | |
| Ahsan et al. | Network intrusion detection using machine learning approaches: Addressing data imbalance | |
| CN112486784A (en) | Method, apparatus and medium for diagnosing and optimizing data analysis system | |
| Cacciarelli et al. | Robust online active learning | |
| US20230353476A1 (en) | Unified recommendation engine | |
| Kong et al. | A novel ConvLSTM with multifeature fusion for financial intelligent trading | |
| CN115146844A (en) | Multi-mode traffic short-time passenger flow collaborative prediction method based on multi-task learning | |
| US20230052255A1 (en) | System and method for optimizing a machine learning model | |
| US20230063489A1 (en) | Malware Detection with Multi-Level, Ensemble Artificial Intelligence Using Bidirectional Long Short-Term Memory Recurrent Neural Networks and Natural Language Processing | |
| Mortazavi et al. | Efficient Mobile Cellular Traffic Forecasting using Spatial-Temporal Graph Attention Networks | |
| Chen et al. | Efficient and interpretable SRU combined with TabNet for network intrusion detection in the big data environment | |
| Zheng et al. | Flow-by-flow traffic matrix prediction methods: Achieving accurate, adaptable, low cost results |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200619 |