CN108923975A - A kind of traffic behavior analysis method of Based on Distributed network - Google Patents
A kind of traffic behavior analysis method of Based on Distributed network Download PDFInfo
- Publication number
- CN108923975A CN108923975A CN201810728186.0A CN201810728186A CN108923975A CN 108923975 A CN108923975 A CN 108923975A CN 201810728186 A CN201810728186 A CN 201810728186A CN 108923975 A CN108923975 A CN 108923975A
- Authority
- CN
- China
- Prior art keywords
- network
- state
- node
- behavior
- traffic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 claims abstract description 90
- 238000012549 training Methods 0.000 claims abstract description 19
- 238000012544 monitoring process Methods 0.000 claims abstract description 10
- 230000006399 behavior Effects 0.000 claims description 146
- 239000000523 sample Substances 0.000 claims description 40
- 230000008569 process Effects 0.000 claims description 28
- 230000007704 transition Effects 0.000 claims description 26
- 239000011159 matrix material Substances 0.000 claims description 21
- 235000019580 granularity Nutrition 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000003993 interaction Effects 0.000 claims description 4
- 238000012512 characterization method Methods 0.000 claims description 3
- 239000000758 substrate Substances 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 claims description 2
- 239000002184 metal Substances 0.000 claims 1
- 238000007726 management method Methods 0.000 abstract description 16
- 238000001514 detection method Methods 0.000 abstract description 11
- 230000005856 abnormality Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004141 dimensional analysis Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention provides a kind of traffic behavior analysis method of Based on Distributed network, the method includes：On-premise network flow collection scheme；Acquire historical traffic data；Training pattern；Obtain traffic behavior model；Acquire real-time traffic data；Estimate the behavior of network global traffic.Entire distributed network is considered as an entirety by the present invention, by acquiring network node flow information, using network node traffic behavior sky when context relation, analyze network flow inner behavior state, it realizes the monitoring to network global traffic behavior, can assist carrying out the network managements such as scheduling of resource, abnormality detection work.
Description
Technical Field
The invention relates to the field of network management application, in particular to a flow behavior analysis method for a distributed network.
Background
With the rapid development of networks and information technologies, the unprecedented expansion of network scales and the wide use of various network applications, the networks are deeply integrated into the fields of politics, economy, culture and the like, the diversity of the networks brings great convenience to the working and life of people, meanwhile, the complexity of the networks increases the difficulty of network management and maintenance, and network administrators face a lot of difficulties. The advance of the IPv6 technology prompts the transition of a network protocol from IPv4 to IPv6, so that the situation of IPv4/IPv6 double-stack parallel frequently occurs in the network, and the difficulty of network exception checking is increased. Various networks including wireless local area networks, wireless metropolitan area networks, public mobile communication networks and the like are accessed to the internet, so that the scale of the network is enlarged, and the operation and maintenance of the network are difficult due to the heterogeneity of the network. The appearance of cloud computing, the rise of social networks and the development of multimedia technologies promote that network application flow is complex and changeable, and the bandwidth of normal services in the network can be occupied in serious cases, which brings challenges to the optimization configuration of network bandwidth. On the other hand, the infinite network security problems affect the normal operation of the network of the operators, enterprises and the like, which often causes economic loss, and the security management of the network becomes a significant task of the network administrator.
In order to solve the above-mentioned problems in a complex network environment, enhance network management capability, and establish a stable and secure network environment, a number of methods for analyzing network behavior have been proposed in academia and industry. The method comprises the following steps: single-point and multipoint oriented traffic analysis methods. In the traffic analysis method facing single point, the paper "Zhao D, score I, Sayed B, et al, botnet detection based on traffic analysis and flow intervals [ J ]. Computers & Security,2013,39(4): 2-16" 0 proposes a network traffic analysis method, which mainly analyzes network flow characteristics in a communication network, including information of a source and destination IP address, a source and destination port, a protocol, a packet length, and the like, and further detects a zombie host in the network. The method provides a relatively complete network flow initial characteristic set through deep analysis of an IP data packet, dynamically selects a characteristic subset for anomaly detection according to different types of network anomalies, and finally carries out class prediction on unknown samples according to the characteristic subset by using a Bayesian classifier. The technology of the green alliance provides a Network traffic analysis product (http:// www.nsfocus.com.cn/products/details _22_2.html), which can collect traffic information of routing equipment in the Network through a Simple Network Management Protocol (SNMP), a Netflow Protocol and the like, and perform multi-dimensional analysis including traffic conditions, traffic composition, traffic variation trend and the like in the Network. In the multipoint-oriented traffic analysis method, the article "Jiang D, Xu Z, Zhang P, et al.a transform domain-based traffic detection from network to network-wide traffic [ J ]. Journal of network & Computer Applications,2014,40(C): 292-. The article "Li Y, Luo X, Qian Y, et al, network-Wide Traffic analysis Detection and localization Based on Robust multivariable predictive signalling Model [ J ]. physical systems in Engineering,2015 (1), (1) - (26)" proposes a method for detecting and locating abnormal Traffic at network side, which comprises measuring the Traffic of OD pairs in the network, such as number of data packets, number of bytes, and number of Traffic, constructing a Traffic matrix, constructing a Traffic normal behavior Model by applying a Multivariate t distribution implicit variable probability theory method, and realizing abnormal Detection and location by evaluating a sample Mahalanobis distance. The leersingpeng patent discloses a system and a method for analyzing network flow, the system firstly collects original flow information of each node in a network through a flow collection module, then extracts application layer flow information in the original flow information, and analyzes whether abnormal flow exists in an application system through statistical comparison of the application layer flow information, thereby realizing application layer analysis based on network flow. The Guo ancestral patent discloses a distributed network traffic analysis system and method, the system firstly collects traffic information in the network through a traffic collection module, then extracts information of a network layer, a transmission layer and an application layer in the original traffic information, and analyzes and processes information of the network layer, the transmission layer and the application layer, and analyzes total traffic conditions, IP to IP traffic data, IP layer network data information and application layer protocol information. The Keyuan network provides a full-flow security analysis product (Keyuan network full-flow security analysis system https:// app. huaweifull. com/product/00301-.
The above method can solve different network problems to some extent, but has some limitations:
(1) in the single-point-oriented traffic analysis method, only traffic passing through the node can be acquired, only local network traffic information can be analyzed, understanding of overall network traffic behavior is difficult to obtain, and network management cannot only depend on local information or only aims at local problems, and an overall optimal scheme and strategy need to be formulated from a global perspective.
(2) In the multipoint-oriented traffic analysis method, multi-node traffic data is generally associated, but the time and space context association relationship brought by network topology structure information and network interconnection is not utilized, the internal behavior state of traffic among network nodes and the overall traffic behavior state of the whole network are difficult to depict, and for a network with complex system characteristics, the overall behavior characteristics of the whole network are difficult to reflect through local superposition.
Disclosure of Invention
In order to overcome the limitation of the prior art, the invention provides a traffic behavior analysis method facing a distributed network. The method considers the network as a whole, analyzes the network global flow behavior by utilizing the space-time context information of the network node flow behavior, can reveal the flow internal behavior state between the network nodes and the whole flow behavior state of the whole network, and enables a network administrator to have global knowledge about the network under jurisdiction.
In order to realize the purpose of the invention, the technical scheme is as follows:
a traffic behavior analysis method for a distributed network can realize the analysis of the global traffic behavior of the network, and specifically comprises the following steps:
a model training stage: collecting network historical traffic data as training data of a training model to obtain a network traffic behavior model;
a learning stage: and inputting the acquired real-time traffic data into a trained network traffic behavior model, and obtaining the network global traffic behavior through iterative computation by utilizing a maximum posterior estimation criterion.
Preferably, before acquiring the traffic data, a network probe is deployed in the administration network to acquire the traffic data, specifically, the probe is deployed at a network node, the traffic data of different granularities and different protocol levels are acquired at the network node, and are transmitted to a traffic analysis center for data analysis.
Preferably, the implementation process of the model training phase specifically includes: determining a network flow behavior model structure and estimation model parameters;
determining a network flow behavior model structure:
the distributed network traffic behavior information is divided into two layers: the system comprises a hidden state layer and an observation data layer, wherein the observation data layer is formed by network node flow data obtained by measuring through a network probe, and the hidden state layer is formed by a behavior mode of a network node, represents a driving factor in the network and directly drives the external appearance of the network node flow; random variables are used for representing the hidden state and the observed value, so that the hidden state layer and the observed data layer form two random fields, namely a hidden state field and an observed field;
defining mathematical symbols: in a network having N nodes,a collection of network nodes is represented as a collection of network nodes,indicating the nth node at the t-th time slot,whereinRepresents all the space-time position node sets, haveT is the number of time slots; using St,nRepresenting a node xt,nThe hidden state variable of (a) is,denotes a random variable StnAn example of whereinRepresents a set of hidden states, thenThe representation is defined inA family of hidden state random variables; thus, can be represented as S at [1, T]A hidden-state field on the surface of the substrate,represents a configuration of S in whichRepresenting a hidden-state place possible configuration set; using a similar expression, Ot,nRepresenting a node xt,nIs compared with the observed value of (a) variable,denotes a random variable Ot,nAn example of whereinRepresenting a set of observations, thenThe representation is defined inA family of observed values random variables; thus, can be represented by O in [1, T ]]The field of view of (a) is,represents a configuration of O, whereinIndicating a set of possible configurations of observation sites;
the method comprises the steps of using an HMRF model to depict a space-time evolution relation between a hidden state field and an observation field;
for hidden state fields, one assumption is introduced: a node is only related to the state of a one-hop neighbor node in space and only related to the state of the previous time in time; based on the statistical learning method, the probability of the hidden state field can be obtained through the following formula:
wherein,indicates that node x is not includedt,nA set of network space-time location nodes of (a),andrespectively represent nodes xt,nThe space neighbor state and the time neighbor state of (1), and lambda represents a hidden state field parameter;
the local probability in formula (1) is obtained by the following formula:
where m represents the node state, time transition probabilityCalculating according to a time hidden state transition probability matrix A, wherein A represents a state transition probability matrix of a hidden state from the time t to the time t +1, and the time hidden state transition forms a first-order Markov chain; the A matrix is represented by:
wherein P isijSubscripts i and j of (a) indicate that the node is at t and at j, respectivelythe hidden state of the t +1 moment; the spatial transition probability is obtained by the following formula:
wherein, Ut,n(m) represents an edge energy function, andrepresenting a node xt,nThe spatial neighbor nodes of (a) are,representing a node xt,nThe number of spatial neighbor nodes of (2), wherein the potential function is defined as: vt,n(m) num · α, where the parameter α is used to delineate the strength of the relationship between the current node and its spatial neighboring nodes, and num represents the number of different states of the spatial neighboring nodes from the current node state;
for an observation field, network node observations are obtained by network probes, i.e. the observation field of the networkIs known data; the observation value of a node is only related to the state of the node, and the output probability of an observation field under the drive of a hidden state is obtained by the following formula:
wherein the continuous multiplication symbol subscript (t, n) representsPr[Ot,n＝k|St,n＝m,θm]Representing the probability that the observed value is k when the node n is in the state of m at the time t, and calculating the observed value O convenientlyt,nDiscretizing, using frequency to approximate alternative probabilitiesApproximating the conditional probability, parameter θ, using the frequency distribution of the observed values at state mmThe distribution parameter representing the observed value in a particular state, here represented by the output probability matrix B, referred to as the observed field parameter, is represented by the following equation:
wherein P ismkRepresenting the probability that the node outputs an observation value k in the state m;
thus, a network traffic behavior model structure is determined, the network traffic behavior model is characterized by an HMRF model, so that the model parameters are omega ═ { A, α, B },
estimating model parameters;
in order to facilitate practical engineering application, frequency approximation probability is used in the calculation process, and therefore an observed value O needs to be calculated before calculationt,nDiscretizing;
the training process inputs historical flow data o, namely network node observed values, and outputs model parameters omega as { A, α, B }, and the flow of estimating the model parameters comprises the following steps:
(3-1) initializing iteration polling initial value i, iteration stop condition Iter and initial hidden state field s(1)；
Wherein an iterative polling initial value i is initialized to 1; the iteration stop condition is set as iteration stop times Iter, and is preferably set to 5-8 times according to experience; in addition, the iteration stopping condition can also be set as a parameter change range threshold value in the two iteration processes, and when the change range is smaller than a given threshold value, the iteration is stopped; initial hidden state field s(1)Initializing by using a clustering algorithm according to the observed value of historical flow data, determining the number of clustering categories according to the monitoring requirement of an actual network, wherein the number of the categories corresponds to the behavior of network nodesThe number of states, therefore, the number of categories reflects the granularity of network traffic behavior characterization, and the more the number of behavior states, the finer the granularity of traffic behavior that can be characterized;
(3-2) updating model parameters according to the configuration condition of the hidden state field, updating a time hidden state transition probability matrix A according to the frequency of time state jump, and setting the frequency of the transition from the time t to the state j when the time t is in the state i and the time t +1 as AijThen in A the probability of a state transition PijIs obtained by the following formula:
determining α value according to empirical formula, preferably between 0.5-10, wherein the more α, the larger the interaction between nodes is, the more the influence of the state of the neighbor node on the state of the current node is, and vice versa, updating output probability matrix B according to frequency distribution of corresponding state output observation value, and setting frequency of state m in sample and observation value k as BmkThen B outputs the probability PmkIs obtained by the following formula:
meanwhile, the iterative polling times are added with 1, namely i is i + 1;
(3-3) judging whether a stopping condition is met, namely judging that i is greater than Iter;
3-3-1) if the judgment is negative, updating the hidden state field s according to the estimated behavior state process(i)The input data is historical flow data o and current model parametersThe output data is an updated hidden state field s(i)Wherein the initial hidden state field of the process of estimating the behavior state uses the current hidden state field s(i-1)；
Returning to the step (3-2);
3-3-2), if the judgment result is yes, outputting a final model parameter omega, { a, α, B };
according to the above steps, the model parameters Ω ═ { a, α, B } trained on the historical flow data can be used as the flow behavior model.
Preferably, the learning phase comprises the following steps:
according to the acquired traffic behavior model and the acquired real-time traffic data of the network node, the internal behavior state of the network node traffic can be estimated,
the above process is equivalent to estimating a configuration of the hidden state field given the model parameters Ω and the observation field oAccording to the MAP estimation criterion, the optimal hidden state field estimation value is foundEquivalent to solving the following equation:
according to Bayes' theorem, there areDue to Pr [ o]Is a constant, so Pr [ s | o, Ω)]∝Pr[o|s,Ω]·Pr[s|Ω](ii) a Wherein the prior probability Pr [ s | omega [ ]]And likelihood probability Pr [ o | s, Ω)]Respectively calculated by formulas (1) and (2), namely obtained by the following formulas:
using an iterative meterObtaining a network global optimal hidden state field in a calculation mode, wherein the behavior state estimation process is to input data which are real-time flow data o and model parameters omega which are { A, α, B }, and output data which are network hidden state field estimation valuesThe process for estimating the behavior state comprises the following steps:
(4-1) initializing iterative polling initial value i and initial hidden state field s(0)Iteration stop condition Iter;
an iteration polling initial value i is initialized to 1; initializing the state field according to the prior knowledge of the relation between the state field and the observation field, or initializing the state field by using a clustering algorithm according to the observation field; the iteration stop condition is set as iteration stop times Iter, and is preferably set to 3-5 times according to experience;
(4-2) for each node in the network, traversing all possible state values, selecting the state value with the maximum probability as the estimation result of the current iteration turn of the node according to the MAP estimation criterion, and equivalently solving the following formula:
meanwhile, the iterative polling times are added with 1, namely i is i + 1;
(4-3) judging whether a stopping condition is met, namely judging that i is greater than Iter;
4-3-1), if the judgment is negative, returning to the step (4-2) and updating the state of each node again;
4-3-2) if the judgment is yes, outputting the final state field estimation valueTherefore, the network global flow behavior state is obtained, and a network administrator can monitor the whole network behavior accordingly.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that: the invention discloses a flow behavior analysis method facing a distributed network. In a jurisdiction network region, a network probe is deployed to acquire network node flow data, and a network flow behavior model is established by utilizing space-time context information of network node flow behaviors.
Drawings
FIG. 1 is a schematic flow diagram of the overall process;
FIG. 2 is a schematic diagram of an actual deployment framework of the method;
FIG. 3 is a schematic diagram of network node traffic behavior information in the method;
FIG. 4 is a flow chart of the method for estimating model parameters;
FIG. 5 is a flow chart of the method for estimating a behavior state;
FIG. 6 is a state diagram illustrating the arrival pattern of packets at network nodes at different times in the embodiment;
FIG. 7 is a fitting probability density function of normalized observations corresponding to a packet arrival pattern in an embodiment.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the patent; for the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;
it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted. The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
The invention overcomes the limitation of the prior art, the network is regarded as a whole, the space-time context correlation information of the network node flow behaviors is utilized, the correlation is derived from the correlation of the interconnection of the network and the network flow of the time before and after, namely the mutual action of the adjacent network nodes, the similarity of the flow behaviors of the adjacent nodes and the similarity of the flow behaviors of the time before and after the nodes, the method can reveal the internal flow behavior state between the network nodes and the whole flow behavior state of the whole network, so that a network administrator can have a global understanding on the network under jurisdiction and guide the development of network management work such as resource scheduling, abnormal detection and the like.
General frame
A flow behavior analysis method facing to a distributed network belongs to network management application and realizes the analysis of the global flow behavior of the network, and the general flow diagram of the method is shown in figure 1 and comprises six steps which are respectively: step S1, deploying a network traffic collection scheme; step S2, collecting historical flow data; step S3, training a model; step S4, acquiring a flow behavior model; step S5, collecting real-time flow data; and step S6, estimating the network global traffic behavior.
The step S1 is realized by deploying a network probe at a network node, and the captured network node traffic data is transmitted to a traffic analysis center for further analysis;
step S2, the network probe collects traffic data for the network nodes, and the traffic data is used as training data of a traffic behavior model;
step S3 is to train a model that can characterize the space-time behavior of network traffic according to the collected historical traffic data, and the method uses a Hidden Markov Random Field (HMRF) mathematical model to model the dynamic variation process of distributed network traffic;
step S4, the flow analysis center obtains a flow behavior model;
step S5 is to collect the network traffic data to be analyzed by the network probe in real time in practical application;
the step S6 is to obtain a network global traffic behavior state from the real-time traffic data through traffic behavior model estimation, so that a network administrator can monitor the overall network behavior and guide the network management work such as resource scheduling and anomaly detection.
The execution method of the invention is as follows: in practical application, collected real-time flow data is input into the trained flow behavior model, and network global flow behavior is obtained through iterative computation by utilizing a Maximum A Posteriori (MAP) estimation criterion, so that the monitoring of the whole network behavior is realized, and further network management work is assisted.
The contents of the steps of the method are described in detail below with reference to fig. 1.
Step S1, deploying a network traffic collection scheme
In order to analyze the network traffic behavior, a network traffic collection scheme needs to be deployed first. As shown in fig. 2, in the method, a network probe is deployed on a node of a dominating network to acquire network traffic, and meanwhile, the acquired traffic data is transmitted to a traffic analysis center for subsequent network traffic behavior analysis. The network traffic collection deployment scheme mainly comprises the following substeps of S1-1, deploying a probe by a network node, S1-2, collecting network traffic data by the network probe, and S1-3, wherein the network probe is communicated with a traffic analysis center.
Step S1-1, the network node deploys the probe. The scheme can be suitable for different network scenes, including the internet formed by traditional routers and switches, SDN-based networks, NFV-based networks and the hybrid networks. The deployed network probe is a functional entity, and the probe can be a physical device, such as a dedicated server or a hardware probe, or a software functional entity integrated on a network device, such as a NetFlow or sFlow function on a router or a switch, an SNMP-agent service, or a virtual probe implemented by NFV.
And step S1-2, the network probe collects network traffic data. The method can collect flow data with different granularities at network nodes, and the flow data comprises information such as data packet records, flow level records, flow statistic records and the like. And aiming at the flow data with different granularities, different flow acquisition schemes are adopted.
The method includes the steps that recorded information of a flow data packet is captured by a network node, a server needs to be deployed at the position of network equipment, the complete flow data packet information including information such as a captured data packet timestamp, an IP address, a port, a protocol and a packet length can be captured in a port mirroring mode, and the server serves as a network probe and undertakes tasks such as storage of the flow data, preliminary processing of the flow data, communication with a flow analysis center and the like. In order to collect network flow of a network node flow level, a NetFlow or sFlow function is started on a network device supporting NetFlow or sFlow, IP flow information on all ports of the device is collected, the flow information contained in a NetFlow or sFlow message mainly comprises the size of a data packet, flow per second, total flow and the like, and the NetFlow or sFlow function is integrated on the network device as a network probe and is output and transmitted to a flow analysis center in a flow message mode. In order to collect the traffic statistic Information of the network node, the SNMP-agent service is started on the network equipment by using a traffic collection method based on an SNMP protocol and used as a network probe of the network node, the traffic statistic data on the equipment is stored in a local Management Information Base (MIB) in a specific form, and the traffic analysis center requests the MIB data in the network equipment to realize the acquisition of various effective traffic data of the network node.
And step S1-3, the network probe communicates with the traffic analysis center. For the network probes which capture the information of the traffic data packets, each network probe transmits local traffic data to a traffic analysis center based on the communication mode of the client server; in a flow acquisition scheme based on NetFlow or sFlow, a network probe actively sends acquired flow data to a flow analysis center serving as an acquisition device; in the SNMP-based traffic collection scheme, a traffic analysis center actively requests traffic statistical information from a network probe serving as an SNMP-agent.
According to the scheme, the deployment of the network traffic collection scheme is realized, the traffic information of different granularities and different protocol layers of the network nodes can be collected, and the traffic can be processed according to the actual network management requirements.
Step S2, collecting historical flow data
According to the network traffic collection scheme actually deployed in step S1, the traffic analysis center collects historical traffic data at the network node through the network probe, and uses the historical traffic data as training data of the model training process. The historical traffic data includes traffic data of different granularities and different protocol levels, such as information of byte number, data packet arrival rate, IP address, application protocol, etc., and may also be further processed traffic information, such as fourier transform, wavelet transform, information entropy for calculating traffic statistical variables, etc., on the basic time domain traffic data.
In order to reduce the traffic from the network probe to the traffic analysis center, for some preliminary processing such as calculating frequency domain signals or information entropy, a calculation task may be deployed on the network probe, and only the processing result of the traffic is sent to the traffic analysis center.
According to the mode, the traffic analysis center can obtain historical traffic data of the network nodes, and accordingly a network traffic behavior model can be trained.
Step S3, training the model
In step S3, 2 sub-steps need to be completed: step S3-1, determining the flow behavior model structure, step S3-2, estimating the model parameters.
In step S3-1, first, the idea of modeling the network traffic behavior by the method is introduced. As shown in fig. 3, for a node (e.g., a switch or a router) in a network, the method divides traffic behavior information of the node into two parts: observable and unobservable portions. The observable part refers to traffic data obtained by directly measuring through a network probe, such as information of byte number, packet arrival rate, IP address, application protocol, and the like, and these measured values reflect the appearance of the traffic behavior of the node, which is hereinafter referred to as "observed value". The non-observable part refers to intrinsic factors driving the external appearance of the node traffic, such as behavior patterns, intrinsic mechanisms and the like, which cannot be measured by a network probe and can only be estimated according to an observable value of the node, and is referred to as a "hidden state" hereinafter.
Extending to a distributed network, as shown in fig. 2, the modeling method divides the traffic behavior information of the distributed network into two layers: a hidden state layer and an observation data layer. The observation data layer is composed of network node flow data measured by a network probe, and the hidden state layer is composed of a behavior mode of the network node, represents the driving factors in the network and directly drives the external expression of the network node flow. Random variables are used herein to represent hidden states and observations, so the hidden state layer and the observed data layer constitute two random fields, a hidden state field and an observed field (hereinafter "state" means equivalent to "hidden state" and "state field" means equivalent to "hidden state field").
In step S3-1, next, the mathematical notation used in the method is defined. In a network having N nodes,a collection of network nodes is represented as a collection of network nodes,indicates at the t-th time slotA node, whereinRepresents all the space-time position node sets, haveT is the number of slots. Using St,nRepresenting a node xt,nThe hidden state variable of (a) is,denotes a random variable St,nAn example of whereinRepresents a set of hidden states, thenThe representation is defined inA family of hidden state random variables. Thus, S can be used to indicate that [1, T ]]A hidden-state field on the surface of the substrate,represents a configuration of S in whichIndicating a potential configuration set of hidden-state places. Using a similar expression, Ot,nRepresenting a node xt,nIs compared with the observed value of (a) variable,denotes a random variable Ot,nAn example of whereinRepresenting a set of observations, thenThe representation is defined inRandom variable family of observations. Thus, O can be used to indicate the value in [1, T]The field of view of (a) is,represents a configuration of O, whereinIndicating a set of possible configurations of observation sites.
In step S3-1, finally, a HMRF model is used to characterize the spatio-temporal evolution relationship between the hidden state field and the observed field.
For a hidden state field, the modeling method introduces an important hypothesis: a node is only related to the state of its one-hop neighbor node in space and only to the state of its previous time instant in time. Based on the statistical learning method, the probability of the state field can be obtained by the following formula:
wherein,indicates that node x is not includedt,nA set of network space-time location nodes of (a),andrespectively represent nodes xt,nThe spatial neighbor state and the temporal neighbor state of (a), and λ represents a state field parameter.
The local probability in formula (1) is obtained by the following formula:
where m represents the node state, time transition probabilityAnd calculating according to a time hidden state transition probability matrix A, wherein A represents a state transition probability matrix of the hidden state from the time t to the time t +1, and the time hidden state transition forms a first-order Markov chain. The A matrix is represented by:
wherein P isijThe subscripts i and j of (a) indicate the hidden states that the node is in at times t and t +1, respectively. The spatial transition probability is obtained by the following formula:
wherein, Ut,n(m) represents an edge energy function, andrepresenting a node xt,nThe spatial neighbor nodes of (a) are,representing a node xt,nThe number of spatial neighbor nodes of (2), wherein the potential function is defined as: vt,nAnd (m) num · α, where the parameter α is used to describe the strength of the mutual influence relationship between the current node and its spatial neighboring nodes, and num represents the number of different states of the spatial neighboring nodes from the current node state.
For the observation field, the network node observation value is directly obtained through the traffic collection scheme deployed in step S1, that is, the observation field of the networkIs known data. The method considers that the observed value of a node is only related to the state of the node, and the output probability of an observed field under the drive of a hidden state is obtained through the following formula:
wherein the continuous multiplication symbol subscript (t, n) representsPr[Ot,n＝k|St,n＝m,θm]Representing the probability that the observed value is k when the node n is in the state of m at the time t, and calculating the observed value O convenientlyt,nDiscretizing, using frequency to approximate the replacement probability, i.e. using the frequency distribution of the observed values in state m to approximate the conditional probability, parameter θmThe distribution parameter representing the observed value in a particular state, here represented by the output probability matrix B, referred to as the observed field parameter, is represented by the following equation:
wherein P ismkRepresenting the probability that a node outputs an observation of k at state m.
Thus, step S3-1 is completed, the flow behavior model structure is determined, and the flow behavior model is characterized by the HMRF model so that the model parameters are Ω ═ { a, α, B }, and step S3-2 is described below to estimate the model parameters.
In step S3-2, after the historical traffic data is collected and the traffic behavior model structure is determined, the historical traffic data is used to train the model parameter Ω ═ { a, α, B }. to facilitate practical engineering application, the method uses frequency approximation probability in the calculation process, so the observed value O needs to be calculated before calculationt,nDiscretization is performed. The flow of estimating model parameters is shown in FIG. 4, and the historical flow is input in the training processData o, i.e., network node observed values, output model parameters Ω ═ a, α, B.
(1) Initializing iteration polling initial value i, iteration stop condition Iter and initial state field s(1)。
Where the iterative polling initial value i is initialized to 1. The iteration stop condition is set as iteration stop times Iter, and is preferably set to 5-8 times according to experience; in addition, the iteration stop condition may also be set as a threshold value of a parameter variation range of two iteration processes, and when the variation range is smaller than a given threshold value, the iteration is stopped. Initial state field s(1)The method comprises the steps of initializing by using a clustering algorithm according to a historical traffic data observation value, such as a Kmeans algorithm, determining the number of clustering categories according to actual network monitoring requirements, wherein the number of the categories corresponds to the number of network node behavior states, so that the number of the categories reflects the granularity of network traffic behavior characterization, and the more the number of the behavior states, the finer the granularity of traffic behavior which can be characterized.
(2) Updating model parameters according to the configuration condition of the state field, updating a time hidden state transition probability matrix A according to the frequency of time state jump, and recording the frequency of the transition from the time t to the state j when the time t is in the state i and the time t +1 as AijThen in A the probability of a state transition PijIs obtained by the following formula:
determining α value according to empirical formula, preferably between 0.5-10, wherein the more α, the larger the interaction between nodes is, the more the influence of the state of the neighbor node on the state of the current node is, and vice versa, updating output probability matrix B according to frequency distribution of corresponding state output observation value, and setting frequency of state m in sample and observation value k as BmkThen B outputs the probability PmkIs obtained by the following formula:
meanwhile, the iterative polling times are increased by 1, i ═ i + 1.
(3) And judging whether the stop condition is met, namely judging that i is greater than Iter.
1) If not, the status field S is updated according to the behavior state estimation process in step S6(i)The input data is historical flow data o and current model parametersThe output data being an update state field s(i)Wherein the initial state field of the (1) th step of the step of estimating the behavior state flow in the step S6 uses the current state field S(i-1)。
And (4) returning to the step (2).
2) If yes, the final model parameter Ω is output as { a, α, B }.
According to the above steps, the model parameters Ω ═ { a, α, B } trained on the historical traffic data can be used as the traffic behavior model.
Step S4, obtaining a flow behavior model
The model parameters mainly comprise omega ═ { A, α, B }, the change of the time dimension of the network flow behavior mode can be described by using a time hidden state transition matrix A, the interaction relation of space nodes is described by using a space state field parameter α, and the relation between an observed value and a hidden state is described by using an output probability matrix B.
The flow behavior model works in a flow analysis center, and can be flexibly applied according to the actual network management requirements. For different types of historical traffic data obtained in step S2, such as information about number of bytes, packet arrival rate, IP address, application protocol, etc., different models, such as a packet arrival rate model, a traffic IP address model, an application protocol model, or a model fusing multiple types of traffic data, may be obtained by training in the traffic analysis center according to step S3.
According to the mode, the flow analysis center can obtain different flow behavior models for use in actual flow behavior analysis, and can provide multi-dimensional flow behavior analysis for users.
Step S5, collecting real-time flow data
According to the network traffic collection scheme deployed in step S1, real-time traffic data can be collected in practical applications.
In a flow analysis center, according to a deployment strategy of a network administrator, network probe data can be polled in real time or periodically to obtain network flow data to be analyzed, wherein the network flow data comprises information such as byte number, data packet arrival rate, IP address and application protocol, and the network flow data reflects flow appearance in the current network environment.
According to actual monitoring requirements, specific types of flow data are selected, and after discretization processing, the flow data are used as observed values and input into corresponding flow behavior models for further estimating network global flow behaviors.
According to the mode, the flow analysis center can acquire real-time flow data.
Step S6, estimating the network global flow behavior
In the traffic analysis center, according to the traffic behavior model obtained in step S4 and the network node real-time traffic data acquired in step S5, the internal behavior state of the network node traffic can be estimated, and a network administrator can accordingly obtain monitoring of the overall network behavior and guide the implementation of network management work such as resource scheduling and anomaly detection.
The above process is equivalent to estimating a configuration of the hidden state field given the model parameters Ω and the observation field oAccording to the MAP estimation criterion, the optimal hidden state field estimation value is foundEquivalent to solving the following equation:
according to Bayes' theorem, there areDue to Pr [ o]Is a constant, so Pr [ s | o, Ω)]∝Pr[o|s,Ω]·Pr[s|Ω]. Wherein the prior probability Pr [ s | omega [ ]]And likelihood probability Pr [ o | s, Ω)]Respectively calculated by formulas (1) and (2), namely obtained by the following formulas:
the method obtains a network global optimal state field by using an iterative computation mode, the flow of estimating the behavior state is shown in fig. 5, input data is real-time flow data o, model parameters omega are { A, α, B }, and output data is the estimated value of the network state fieldThe process for estimating the behavior state comprises the following steps:
(1) initializing iterative polling initial value i, initial state field s(0)Iteration stop condition Iter.
The iterative polling initial value i is initialized to 1. The state field is initialized according to prior knowledge of the relationship between the state field and the observation field, or the state field is initialized according to the observation field by using a clustering algorithm. The iteration stop condition is set to the iteration stop number Iter, and is preferably set to 3 to 5 times empirically.
(2) For each node in the network, traversing all possible state values, selecting the state value with the maximum probability as the current iteration round estimation result of the node according to the MAP estimation criterion, and equivalently solving the following formula:
meanwhile, the iterative polling times are increased by 1, i ═ i + 1.
(3) And judging whether the stop condition is met, namely judging that i is greater than Iter.
1) If not, returning to the step (2) and updating the state of each node again.
2) If yes, outputting the final state field estimation valueTherefore, the network global flow behavior state is obtained, and a network administrator can monitor the whole network behavior accordingly.
Particularly, the network is regarded as a whole in the method, the estimated network behavior state is a global optimal result and represents the most likely state of the network node, and tasks such as resource scheduling, anomaly detection and the like performed according to the state are an overall optimal scheme and strategy.
The meanings of corresponding behavior states are different aiming at different types of collected traffic data, so that the traffic behavior analysis method can provide multi-dimensional analysis on network traffic. For example, the arrival rate of data packets at network nodes is monitored, the traffic behavior state reflects the arrival mode of the data packets at the current node, and after the global behavior state of the network is estimated, the arrival modes of the data packets in different areas at different times in the network can be known. For example, network node application protocol network flow is monitored, the flow behavior state reflects current node application protocol component information, after the network global behavior state is estimated, the application protocol component information of different areas at different moments in the network can be obtained, and for some nodes with P2P flow occupying a large proportion, a network administrator can limit the flow rate under appropriate conditions to ensure normal service operation.
The method estimates the behavior state of the global network flow, and can carry out the flow scheduling of the whole network according to the behavior state, such as load balancing or green network application. And designing a scheduling algorithm from a global perspective according to the estimated flow size states of different nodes of the network, and scheduling the flow of the high-flow node to the low-flow node to realize the flow load balance of the whole network. When the nodes in the network are all in a low-flow state, on the premise of ensuring network connectivity and link utilization rate constraint, part of the nodes are closed, so that the energy consumption of the network is reduced to the maximum extent under the condition of ensuring the basic performance of the network, and green network application is realized.
According to the method, the network node flow space-time context information is utilized, so that the network global flow behavior state can be estimated, the relation between the network behavior state and the flow appearance can be disclosed, a network administrator can be helped to establish the view angle of the whole network, the situation and trend change of the network can be observed, and the information such as the network load situation and the use situation of network application resources can be mastered. Based on historical traffic data modeling, the distribution and trend characteristics of the network in time, space and flow direction can be established, a network administrator is helped to deeply mine service requirements, hot spots, trends and the like, and network planning and design are assisted. The model is established by taking the multi-dimensional flow data as an observation value, the space-time change of the flow behavior state reflects the space-time evolution process of the network flow, and a network global normal flow behavior model can be established by utilizing the space-time change condition of the network node state to assist in realizing the abnormal detection.
Examples
The embodiment illustrates the advantages of the method by taking the example of analyzing the packet arrival pattern of the network node. As shown in FIG. 6, an example network comprises 50 network nodes and 88 links, a topology source Germany research network (Germany research network topology information http:// sndlib. zip. de/home. action), different nodes capture traffic data in a domination network, the arrival rate of a data packet is taken as a node observation value, the arrival mode of the data packet of the node is taken as a traffic behavior state, the arrival mode of the data packet of the whole network in different time and different areas is sensed according to the distributed network traffic behavior analysis method, and the monitoring of the arrival mode of the data packet of the whole network is realized. According to the above implementation steps S1-S6, first, a traffic collection scheme is deployed throughout the network; secondly, collecting historical data packet arrival rate data at a network node; thirdly, training a model according to the historical data packet arrival rate data; then, the flow analysis center obtains a data packet arrival rate model; then, collecting real-time data packet arrival rate data at the network node, and inputting a data packet arrival rate model; and finally, obtaining a global state diagram depicting the network data packet arrival mode through iterative computation. The network administrator can observe the state and trend change of the network and master the network load condition, thereby realizing the applications of green network, resource scheduling and the like.
The state diagram of the arrival mode of the data packet of the network node at different time is shown in fig. 6, different colors represent different arrival modes of the data packet of the network node, only two of these arrival patterns are shown, the gray nodes corresponding to pattern 1 in fig. 7, the black nodes corresponding to pattern 2 in fig. 7, fig. 7 shows the fitted probability density function of the normalized observed values for these two packet arrival patterns, reflecting the output distribution of the observed values in different states, according to the estimated network global data packet arrival mode, a network administrator can know the data packet arrival mode of different nodes in the network in real time, sense the internal behavior state of the global network flow, know the relation between the network node data packet arrival mode and the data packet arrival rate, according to the behavior state, the node load condition of the whole network can be obtained, and the applications such as green network and resource scheduling are realized in an auxiliary mode.
As the method is an unsupervised learning process and belongs to a multi-classifier, the method is compared with a typical clustering method Kmeans. Estimating the arrival mode of the current network node according to the actual arrival rate of the data packets, wherein when the number of the arrival modes of the data packets is 5, the ratio of the global accuracy of the Kmeans and the method to the macro F1 value performance is shown in a table I, and the ratio of the accuracy, the recall rate and the F1 value performance in different arrival modes (states) is shown in a table II. In the aspect of performance evaluation, the method selects the overall accuracy, the macro F1 value, the accuracy, the recall rate and the F1 value as evaluation indexes. The precision rate (precision), the recall rate (recall) and the F1 value are common evaluation indexes in a binary classification problem, the precision rate P represents the proportion of the number of correctly estimated positive classes to all estimated positive classes, the recall rate R represents the proportion of the number of correctly estimated positive classes to all real positive classes, the F1 value is a harmonic mean value of the precision rate and the recall rate, namely F1 is 2PR/(P + R), the three indexes are used for measuring the performance of each state value estimated by the method, and the higher the index value is, the better the performance is. The overall accuracy represents the proportion of all correctly estimated samples to the total number of samples, the macro F1 value is the arithmetic mean of the F1 values for each state, these two indexes represent the overall performance of the model, and likewise, the higher the index value, the better the performance.
TABLE-Kmeans comparison of the overall accuracy of the method with the performance of the Macro F1 values
Experimental results show that each performance index of the method is superior to that of a Kmeans method, the reason for obtaining better performance of the method is that when network nodes are in different arrival modes, the same arrival rate can be generated, namely, observed values in different modes are overlapped, due to the fact that time continuity and space correlation exist in network node behaviors, the state information of network space-time neighbor nodes is considered in an HMRF model of the method, nodes belonging to the same state can be well distinguished, however, the Kmeans method directly divides the states according to observed data, and therefore a better effect cannot be obtained in the aspect of distinguishing the states of the network nodes, and the gain brought by introducing space-time behavior information into HMRF is for estimating the network behavior states. Therefore, the state value estimated by the method has a good effect, the state of the whole network is described, the method for estimating the behavior state according to the method is more accurate when being used for monitoring network flow data of each dimension, and accordingly, the scheme and the strategy for formulating network management are also integrally optimal.
Comparison of Kmeans with the accuracy, recall and F1 value of the method when the number of states in Table two is 5
It should be noted that this embodiment is only an example of the method, but the method is not limited to analyzing network traffic data of a single dimension, such as a packet arrival rate. The distributed network traffic behavior analysis method can provide multi-dimensional network traffic analysis, overall network traffic behavior state distribution is obtained by estimation by taking multiple kinds of dimensional traffic data as observed values, a network state represents a current network internal working mode, the internal traffic behavior states among network nodes and the overall traffic behavior states of the whole network can be revealed, a network administrator can obtain the overall network traffic data and corresponding working mode information, the space time distribution of the behavior states is analyzed, and the multi-dimensional monitoring of the network traffic behavior is realized.
It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (4)
1. A traffic behavior analysis method facing a distributed network is characterized by comprising the following steps:
a model training stage: collecting network historical traffic data as training data of a training model to obtain a network traffic behavior model;
a learning stage: and inputting the acquired real-time traffic data into a trained network traffic behavior model, and obtaining the network global traffic behavior through iterative computation by utilizing a maximum posterior estimation criterion.
2. The method according to claim 1, further comprising, before collecting the traffic data, deploying a network probe in the administration network to collect the traffic data, specifically deploying the probe on a network node, so as to collect the traffic data of different granularities and different protocol levels at the network node, and transmitting the traffic data to a traffic analysis center for data analysis.
3. The method according to claim 1, wherein the model training phase is implemented by: determining a network flow behavior model structure and estimation model parameters;
determining a network flow behavior model structure:
the distributed network traffic behavior information is divided into two layers: the system comprises a hidden state layer and an observation data layer, wherein the observation data layer is formed by network node flow data obtained by measuring through a network probe, and the hidden state layer is formed by a behavior mode of a network node, represents a driving factor in the network and directly drives the external appearance of the network node flow; random variables are used for representing the hidden state and the observed value, so that the hidden state layer and the observed data layer form two random fields, namely a hidden state field and an observed field;
defining mathematical symbols: in a network having N nodes,a collection of network nodes is represented as a collection of network nodes,indicating the nth node at the t-th time slot,whereinRepresents all the space-time position node sets, haveT is the number of time slots; using St,nRepresenting a node xt,nThe hidden state variable of (a) is,denotes a random variable St,nAn example of whereinRepresents a set of hidden states, thenThe representation is defined inA family of hidden state random variables; thus, can be represented as S at [1, T]A hidden-state field on the surface of the substrate,represents a configuration of S in whichRepresenting a hidden-state place possible configuration set; using a similar expression, Ot,nRepresenting a node xt,nIs compared with the observed value of (a) variable,denotes a random variable Ot,nAn example of whereinRepresenting a set of observations, thenThe representation is defined inA family of observed values random variables; therefore, the temperature of the molten metal is controlled,can be represented by O in [1, T ]]The field of view of (a) is,represents a configuration of O, whereinIndicating a set of possible configurations of observation sites;
the method comprises the steps of using an HMRF model to depict a space-time evolution relation between a hidden state field and an observation field;
for hidden state fields, one assumption is introduced: a node is only related to the state of a one-hop neighbor node in space and only related to the state of the previous time in time; based on the statistical learning method, the probability of the hidden state field can be obtained through the following formula:
wherein,indicates that node x is not includedt,nA set of network space-time location nodes of (a),andrespectively represent nodes xt,nThe space neighbor state and the time neighbor state of (1), and lambda represents a hidden state field parameter;
the local probability in formula (1) is obtained by the following formula:
where m represents the node state, time transition probabilityCalculating according to a time hidden state transition probability matrix A, wherein A represents a state transition probability matrix of a hidden state from the time t to the time t +1, and the time hidden state transition forms a first-order Markov chain; the A matrix is represented by:
wherein P isijSubscripts i and j of (a) respectively represent the hidden states of the node at the time t and t + 1; the spatial transition probability is obtained by the following formula:
wherein, Ut,n(m) represents an edge energy function, and representing a node xt,nThe spatial neighbor nodes of (a) are,representing a node xt,nThe number of spatial neighbor nodes of (2), wherein the potential function is defined as: vt,n(m) num · α, where the parameter α is used to delineate the strength of the relationship between the current node and its spatial neighboring nodes, and num represents the number of different states of the spatial neighboring nodes from the current node state;
for an observation field, network node observations are obtained by network probes, i.e. the observation field of the networkIs known data; setting the observation value of a node only related to the state of the node, and the output probability of the observation field under the drive of the hidden state through the following formulaObtaining:
wherein the continuous multiplication symbol subscript (t, n) representsPr[Ot,n＝k|St,n＝m,θm]Representing the probability that the observed value is k when the node n is in the state of m at the time t, and calculating the observed value O convenientlyt,nDiscretizing, using frequency to approximate the replacement probability, i.e. using the frequency distribution of the observed values in state m to approximate the conditional probability, parameter θmThe distribution parameter representing the observed value in a particular state, here represented by the output probability matrix B, referred to as the observed field parameter, is represented by the following equation:
wherein P ismkRepresenting the probability that the node outputs an observation value k in the state m;
thus, a network traffic behavior model structure is determined, the network traffic behavior model is characterized by an HMRF model, so that the model parameters are omega ═ { A, α, B },
estimating model parameters;
in order to facilitate practical engineering application, frequency approximation probability is used in the calculation process, and therefore an observed value O needs to be calculated before calculationt,nDiscretizing;
the training process inputs historical flow data o, namely network node observed values, and outputs model parameters omega as { A, α, B }, and the flow of estimating the model parameters comprises the following steps:
(3-1) initializing iteration polling initial value i, iteration stop condition Iter and initial hidden state field s(1)；
Wherein the initial value of iterative polling i isThe initialization is 1; the iteration stop condition is set as iteration stop times Iter, and is preferably set to 5-8 times according to experience; in addition, the iteration stopping condition can also be set as a parameter change range threshold value in the two iteration processes, and when the change range is smaller than a given threshold value, the iteration is stopped; initial hidden state field s(1)Initializing by using a clustering algorithm according to a historical traffic data observation value, determining the number of clustering categories according to actual network monitoring requirements, wherein the number of the categories corresponds to the number of behavior states of network nodes, so that the number of the categories reflects the granularity of network traffic behavior characterization, and the more the number of the behavior states, the finer the granularity of traffic behavior which can be characterized is;
(3-2) updating model parameters according to the configuration condition of the hidden state field, updating a time hidden state transition probability matrix A according to the frequency of time state jump, and setting the frequency of the transition from the time t to the state j when the time t is in the state i and the time t +1 as AijThen in A the probability of a state transition PijIs obtained by the following formula:
determining α value according to empirical formula, preferably between 0.5-10, wherein the more α, the larger the interaction between nodes is, the more the influence of the state of the neighbor node on the state of the current node is, and vice versa, updating output probability matrix B according to frequency distribution of corresponding state output observation value, and setting frequency of state m in sample and observation value k as BmkThen B outputs the probability PmkIs obtained by the following formula:
meanwhile, the iterative polling times are added with 1, namely i is i + 1;
(3-3) judging whether a stopping condition is met, namely judging that i is greater than Iter;
3-3-1) if the judgment is negative, updating the hidden state field s according to the estimated behavior state process(i)The input data is history flow data o and current modeForm parameterThe output data is an updated hidden state field s(i)Wherein the initial hidden state field of the process of estimating the behavior state uses the current hidden state field s(i-1)；
Returning to the step (3-2);
3-3-2), if the judgment result is yes, outputting a final model parameter omega, { a, α, B };
according to the above steps, the model parameters Ω ═ { a, α, B } trained on the historical flow data can be used as the flow behavior model.
4. The method according to claim 3, characterized in that the learning phase is carried out by:
according to the acquired traffic behavior model and the acquired real-time traffic data of the network node, the internal behavior state of the network node traffic can be estimated,
the above process is equivalent to estimating a configuration of the hidden state field given the model parameters Ω and the observation field oAccording to the MAP estimation criterion, the optimal hidden state field estimation value is foundEquivalent to solving the following equation:
according to Bayes' theorem, there areDue to Pr [ o]Is a constant, so Pr [ s | o, Ω)]∝Pr[o|s,Ω]·Pr[s|Ω](ii) a Wherein the prior probability Pr [ s | omega [ ]]And likelihood probability Pr [ o | s, Ω)]Calculated by the formulas (1) and (2), respectively, namely the followingObtaining a formula:
obtaining a network global optimal hidden state field by using an iterative computation mode, wherein the behavior state estimation process is to input data which is real-time flow data o and model parameters omega which are { A, α, B }, and output data which is a network hidden state field estimation valueThe process for estimating the behavior state comprises the following steps:
(4-1) initializing iterative polling initial value i and initial hidden state field s(0)Iteration stop condition Iter;
an iteration polling initial value i is initialized to 1; initializing the state field according to the prior knowledge of the relation between the state field and the observation field, or initializing the state field by using a clustering algorithm according to the observation field; the iteration stop condition is set as iteration stop times Iter, and is preferably set to 3-5 times according to experience;
(4-2) for each node in the network, traversing all possible state values, selecting the state value with the maximum probability as the estimation result of the current iteration turn of the node according to the MAP estimation criterion, and equivalently solving the following formula:
meanwhile, the iterative polling times are added with 1, namely i is i + 1;
(4-3) judging whether a stopping condition is met, namely judging that i is greater than Iter;
4-3-1), if the judgment is negative, returning to the step (4-2) and updating the state of each node again;
4-3-2) if the judgment is yes, outputting the final state field estimation valueTherefore, the network global flow behavior state is obtained, and a network administrator can monitor the whole network behavior accordingly.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810728186.0A CN108923975B (en) | 2018-07-05 | 2018-07-05 | Traffic behavior analysis method for distributed network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810728186.0A CN108923975B (en) | 2018-07-05 | 2018-07-05 | Traffic behavior analysis method for distributed network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108923975A true CN108923975A (en) | 2018-11-30 |
| CN108923975B CN108923975B (en) | 2021-08-10 |
Family
ID=64424625
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810728186.0A Expired - Fee Related CN108923975B (en) | 2018-07-05 | 2018-07-05 | Traffic behavior analysis method for distributed network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108923975B (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109831386A (en) * | 2019-03-08 | 2019-05-31 | 西安交通大学 | Optimal route selection algorithm based on machine learning under a kind of SDN |
| CN109951462A (en) * | 2019-03-07 | 2019-06-28 | 中国科学院信息工程研究所 | A kind of application software Traffic anomaly detection system and method based on holographic modeling |
| CN110139125A (en) * | 2019-06-18 | 2019-08-16 | 洛阳师范学院 | Video sharing method based on demand perception and caching resource under mobile radio network |
| CN110691003A (en) * | 2019-09-04 | 2020-01-14 | 北京天融信网络安全技术有限公司 | Network traffic classification method, device and storage medium |
| CN111224940A (en) * | 2019-11-15 | 2020-06-02 | 中国科学院信息工程研究所 | Anonymous service traffic correlation identification method and system nested in encrypted tunnel |
| CN111294284A (en) * | 2018-12-10 | 2020-06-16 | 华为技术有限公司 | Traffic scheduling method and device |
| CN111698269A (en) * | 2020-04-07 | 2020-09-22 | 中博信息技术研究院有限公司 | Network intrusion detection method based on Plackett-Luce model |
| CN112039906A (en) * | 2020-09-03 | 2020-12-04 | 华侨大学 | Cloud computing-oriented network flow anomaly detection system and method |
| CN112134738A (en) * | 2020-09-24 | 2020-12-25 | 中电科仪器仪表有限公司 | Network multidimensional data flow simulation device based on composite two-dimensional Sketch |
| CN112202593A (en) * | 2020-09-03 | 2021-01-08 | 深圳前海微众银行股份有限公司 | Data acquisition method, device, network management system and computer storage medium |
| CN112653588A (en) * | 2020-07-10 | 2021-04-13 | 深圳市唯特视科技有限公司 | Adaptive network traffic collection method, system, electronic device and storage medium |
| CN112769972A (en) * | 2020-12-22 | 2021-05-07 | 赛尔网络有限公司 | Flow analysis method and device for IPv6 network, electronic equipment and storage medium |
| CN112788066A (en) * | 2021-02-26 | 2021-05-11 | 中南大学 | Abnormal flow detection method and system for Internet of things equipment and storage medium |
| CN113569368A (en) * | 2021-09-17 | 2021-10-29 | 支付宝(杭州)信息技术有限公司 | Protocol-based modeling method and device |
| CN113783788A (en) * | 2021-09-16 | 2021-12-10 | 航天新通科技有限公司 | Network optimization system and method based on flow prediction |
| CN114039758A (en) * | 2021-11-02 | 2022-02-11 | 中邮科通信技术股份有限公司 | Network security threat identification method based on event detection mode |
| CN114338419A (en) * | 2021-12-15 | 2022-04-12 | 中电信数智科技有限公司 | IPv6 global networking edge node monitoring and early warning method and system |
| CN114598904A (en) * | 2020-11-20 | 2022-06-07 | 中国移动通信集团广东有限公司 | Fault positioning method and device for IPTV service |
| CN115277249A (en) * | 2022-09-22 | 2022-11-01 | 山东省计算中心（国家超级计算济南中心） | Network security situation perception method based on cooperation of multi-layer heterogeneous network |
| CN116471066A (en) * | 2023-04-06 | 2023-07-21 | 华能信息技术有限公司 | Flow analysis method based on flow probe |
| US11956117B1 (en) | 2023-05-22 | 2024-04-09 | Google Llc | Network monitoring and healing based on a behavior model |
| CN118488587A (en) * | 2024-07-12 | 2024-08-13 | 北京中网华通设计咨询有限公司 | Wireless communication network scheduling method and system based on artificial intelligence |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104753732A (en) * | 2013-12-27 | 2015-07-01 | 郭祖龙 | Distribution based network traffic analysis system and method |
| CN106612289A (en) * | 2017-01-18 | 2017-05-03 | 中山大学 | Network collaborative abnormality detection method based on SDN |
-
2018
- 2018-07-05 CN CN201810728186.0A patent/CN108923975B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104753732A (en) * | 2013-12-27 | 2015-07-01 | 郭祖龙 | Distribution based network traffic analysis system and method |
| CN106612289A (en) * | 2017-01-18 | 2017-05-03 | 中山大学 | Network collaborative abnormality detection method based on SDN |
Non-Patent Citations (1)
| Title |
|---|
| 谢逸 等: "A General Collaborative Framework for Modeling and Perceiving Distributed Network Behavior", 《 IEEE/ACM TRANSACTIONS ON NETWORKING 》 * |
Cited By (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111294284A (en) * | 2018-12-10 | 2020-06-16 | 华为技术有限公司 | Traffic scheduling method and device |
| CN111294284B (en) * | 2018-12-10 | 2022-04-26 | 华为技术有限公司 | Traffic scheduling method and device |
| CN109951462A (en) * | 2019-03-07 | 2019-06-28 | 中国科学院信息工程研究所 | A kind of application software Traffic anomaly detection system and method based on holographic modeling |
| CN109831386B (en) * | 2019-03-08 | 2020-07-28 | 西安交通大学 | Optimal path selection algorithm based on machine learning under SDN |
| CN109831386A (en) * | 2019-03-08 | 2019-05-31 | 西安交通大学 | Optimal route selection algorithm based on machine learning under a kind of SDN |
| CN110139125A (en) * | 2019-06-18 | 2019-08-16 | 洛阳师范学院 | Video sharing method based on demand perception and caching resource under mobile radio network |
| CN110691003A (en) * | 2019-09-04 | 2020-01-14 | 北京天融信网络安全技术有限公司 | Network traffic classification method, device and storage medium |
| CN111224940B (en) * | 2019-11-15 | 2021-03-09 | 中国科学院信息工程研究所 | Anonymous service traffic correlation identification method and system nested in encrypted tunnel |
| CN111224940A (en) * | 2019-11-15 | 2020-06-02 | 中国科学院信息工程研究所 | Anonymous service traffic correlation identification method and system nested in encrypted tunnel |
| CN111698269A (en) * | 2020-04-07 | 2020-09-22 | 中博信息技术研究院有限公司 | Network intrusion detection method based on Plackett-Luce model |
| CN112653588A (en) * | 2020-07-10 | 2021-04-13 | 深圳市唯特视科技有限公司 | Adaptive network traffic collection method, system, electronic device and storage medium |
| CN112202593A (en) * | 2020-09-03 | 2021-01-08 | 深圳前海微众银行股份有限公司 | Data acquisition method, device, network management system and computer storage medium |
| CN112202593B (en) * | 2020-09-03 | 2024-05-31 | 深圳前海微众银行股份有限公司 | Data acquisition method, device, network management system and computer storage medium |
| CN112039906B (en) * | 2020-09-03 | 2022-03-18 | 华侨大学 | Cloud computing-oriented network flow anomaly detection system and method |
| CN112039906A (en) * | 2020-09-03 | 2020-12-04 | 华侨大学 | Cloud computing-oriented network flow anomaly detection system and method |
| CN112134738A (en) * | 2020-09-24 | 2020-12-25 | 中电科仪器仪表有限公司 | Network multidimensional data flow simulation device based on composite two-dimensional Sketch |
| CN112134738B (en) * | 2020-09-24 | 2023-03-24 | 中电科思仪科技股份有限公司 | Network multidimensional data flow simulation device based on composite two-dimensional Sketch |
| CN114598904A (en) * | 2020-11-20 | 2022-06-07 | 中国移动通信集团广东有限公司 | Fault positioning method and device for IPTV service |
| CN114598904B (en) * | 2020-11-20 | 2023-06-30 | 中国移动通信集团广东有限公司 | Fault positioning method and device for IPTV service |
| CN112769972A (en) * | 2020-12-22 | 2021-05-07 | 赛尔网络有限公司 | Flow analysis method and device for IPv6 network, electronic equipment and storage medium |
| CN112769972B (en) * | 2020-12-22 | 2023-02-28 | 赛尔网络有限公司 | Flow analysis method and device for IPv6 network, electronic equipment and storage medium |
| CN112788066A (en) * | 2021-02-26 | 2021-05-11 | 中南大学 | Abnormal flow detection method and system for Internet of things equipment and storage medium |
| CN112788066B (en) * | 2021-02-26 | 2022-01-14 | 中南大学 | Abnormal flow detection method and system for Internet of things equipment and storage medium |
| CN113783788A (en) * | 2021-09-16 | 2021-12-10 | 航天新通科技有限公司 | Network optimization system and method based on flow prediction |
| CN113783788B (en) * | 2021-09-16 | 2022-06-17 | 航天新通科技有限公司 | Network optimization system and method based on flow prediction |
| CN113569368B (en) * | 2021-09-17 | 2022-01-11 | 支付宝(杭州)信息技术有限公司 | Protocol-based modeling method and device |
| CN113569368A (en) * | 2021-09-17 | 2021-10-29 | 支付宝(杭州)信息技术有限公司 | Protocol-based modeling method and device |
| CN114039758A (en) * | 2021-11-02 | 2022-02-11 | 中邮科通信技术股份有限公司 | Network security threat identification method based on event detection mode |
| CN114338419A (en) * | 2021-12-15 | 2022-04-12 | 中电信数智科技有限公司 | IPv6 global networking edge node monitoring and early warning method and system |
| CN114338419B (en) * | 2021-12-15 | 2024-04-16 | 中电信数智科技有限公司 | IPv6 global networking edge node monitoring and early warning method and system |
| CN115277249A (en) * | 2022-09-22 | 2022-11-01 | 山东省计算中心（国家超级计算济南中心） | Network security situation perception method based on cooperation of multi-layer heterogeneous network |
| CN115277249B (en) * | 2022-09-22 | 2022-12-20 | 山东省计算中心（国家超级计算济南中心） | Network security situation perception method based on cooperation of multi-layer heterogeneous network |
| CN116471066A (en) * | 2023-04-06 | 2023-07-21 | 华能信息技术有限公司 | Flow analysis method based on flow probe |
| US11956117B1 (en) | 2023-05-22 | 2024-04-09 | Google Llc | Network monitoring and healing based on a behavior model |
| CN118488587A (en) * | 2024-07-12 | 2024-08-13 | 北京中网华通设计咨询有限公司 | Wireless communication network scheduling method and system based on artificial intelligence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108923975B (en) | 2021-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108923975B (en) | Traffic behavior analysis method for distributed network | |
| Li et al. | Online distributed IoT security monitoring with multidimensional streaming big data | |
| Wang et al. | Recursive fact-finding: A streaming approach to truth estimation in crowdsourcing applications | |
| CN106612289A (en) | Network collaborative abnormality detection method based on SDN | |
| CN113079143A (en) | Flow data-based anomaly detection method and system | |
| US11722359B2 (en) | Drift detection for predictive network models | |
| Garg et al. | A probabilistic data structures-based anomaly detection scheme for software-defined Internet of vehicles | |
| Zhang et al. | Distributed intrusion detection based on clustering | |
| Tayfour et al. | Collaborative detection and mitigation of DDoS in software-defined networks | |
| Cerroni et al. | Decentralized detection of network attacks through P2P data clustering of SNMP data | |
| Yang et al. | Heavy hitter detection and identification in software defined networking | |
| Patcha et al. | Network anomaly detection with incomplete audit data | |
| Chaudhary et al. | Flow classification using clustering and association rule mining | |
| Liu et al. | A fast all-packets-based DDoS attack detection approach based on network graph and graph kernel | |
| Prasad et al. | Ensemble classifiers with drift detection (ECDD) in traffic flow streams to detect DDOS attacks | |
| Shamshirband et al. | Co-FQL: Anomaly detection using cooperative fuzzy Q-learning in network | |
| Kumarage et al. | Granular evaluation of anomalies in wireless sensor networks using dynamic data partitioning with an entropy criteria | |
| Ma et al. | Threat-event detection for distributed networks based on spatiotemporal markov random field | |
| Kadri et al. | Survey and classification of Dos and DDos attack detection and validation approaches for IoT environments | |
| Geepthi et al. | RETRACTED ARTICLE: Network traffic detection for peer-to-peer traffic matrices on bayesian network in WSN | |
| Campazas-Vega et al. | Analysis of netflow features’ importance in malicious network traffic detection | |
| Liu et al. | Topology sensing of non-collaborative wireless networks with conditional Granger causality | |
| Sun et al. | Deep learning-based anomaly detection in LAN from raw network traffic measurement | |
| Pekar et al. | Towards threshold‐agnostic heavy‐hitter classification | |
| Sahay et al. | Traffic convergence detection in IoT LLNs: a multilayer perceptron based mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210810 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |