CN114785707B - Hierarchical large-flow collaborative monitoring method - Google Patents
Hierarchical large-flow collaborative monitoring method Download PDFInfo
- Publication number
- CN114785707B CN114785707B CN202210526869.4A CN202210526869A CN114785707B CN 114785707 B CN114785707 B CN 114785707B CN 202210526869 A CN202210526869 A CN 202210526869A CN 114785707 B CN114785707 B CN 114785707B
- Authority
- CN
- China
- Prior art keywords
- flow
- task
- network
- network flow
- switch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
- H04L43/062—Generation of reports related to network traffic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/15—Correlation function computation including computation of convolution operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computational Mathematics (AREA)
- Signal Processing (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Algebra (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention belongs to the technical field of network measurement, and particularly provides a hierarchical large-flow collaborative monitoring method which is used for solving the problems that the existing centralized method is shifted to distributed collaborative monitoring and the sub-flow loss of the distributed lower hierarchical large-flow collaborative monitoring; according to the invention, the collaborative monitoring problem is solved by modeling and solving the task deployment and network flow selection combined optimization problem, and a better load balancing effect on the whole network switch is realized while a collaborative monitoring strategy is obtained; the invention provides a distributed hierarchical large flow measurement framework for solving the problem of sub-flow loss, wherein a bloom filter is used for supporting a full-network switch to cooperatively monitor hierarchical large flows, a large flow component is used for storing identifiers and count values of candidate large flows, a small flow component is used for storing count values of network flows thrown out by the large flow component, lost sub-flows are recovered through the small flow component, and the measurement accuracy of the hierarchical large flows is improved. Finally, the invention significantly improves the measurement accuracy of the hierarchical large flow.
Description
Technical Field
The invention belongs to the technical field of network measurement, and particularly provides a hierarchical large-flow collaborative monitoring method.
Background
The network measurement provides necessary information for network management application, and plays a role in ensuring the stability and safety of the network; hierarchical large flows (Hierarchical Heavy Hitters) are used as one of the network measurement tasks to identify large flows (Heavy letters) based on public IP prefix aggregation with network applications such as anomaly detection, DDoS detection, etc. Existing research uses a centralized algorithm on a single switch to identify hierarchical large flows in all network flows passing through it, but the continuous expansion of network traffic scale makes it difficult for the centralized algorithm to reach the expected measurement accuracy; in the measurement of the hierarchical large flow, the measurement error of the lower hierarchy can affect the accuracy of the higher hierarchy, which is called a dependency problem; the accuracy of performing hierarchical high-flow measurements on a single switch is very low due to the resource limitations and dependency problems described above.
With the proposal of software defined network and software defined measurement, the method supports the efficient deployment of measurement tasks on a plurality of switches in a distributed manner for collaborative monitoring of network flows; however, in a distributed environment, multiple sub-streams (with common IP prefix) from the same large stream are measured scattered across different switches, and part of the sub-streams will be ignored because they are too small, resulting in aggregation at a higher level, which is referred to herein as the sub-stream loss problem. The current research work mainly focuses on balancing the load of the switch, ignoring the characteristics of specific tasks, and particularly solving the problem of sub-stream loss in hierarchical large-stream collaborative monitoring; based on the method, the invention provides a hierarchical large-flow collaborative monitoring method.
Disclosure of Invention
The invention aims to solve the problems in the prior art and provides a hierarchical large-flow collaborative monitoring method; in order to solve the dependence problem of hierarchical large-flow measurement in a centralized environment, the invention provides collaborative monitoring of hierarchical large-flow under a distributed environment, wherein the hierarchical large-flow collaborative monitoring strategy formulation problem is expressed as a task deployment and network flow selection joint optimization problem, and then the task deployment and network flow selection strategy is approximately solved and acquired, so that the hierarchical large-flow collaborative monitoring is carried out by utilizing the resources of a full-network switch according to the strategy, and the problem of low precision caused by limited resources and dependence problems is alleviated; in order to solve the problem of sub-stream loss of distributed lower-level large stream collaborative monitoring, the invention provides a distributed level large stream measurement framework, each measurement task deployed on each switch under the framework has a compact data structure, and the framework consists of a Bloom filter (Bloom filter), a large stream component (Heavy part) and a small stream component (Light part), wherein the Bloom filter is used for enabling a plurality of switches to carry out hierarchical large stream collaborative monitoring according to a strategy of a joint optimization problem, the large stream component is used for storing identifiers and count values of candidate large streams, and the small stream component is used for storing count values of network streams thrown out by the large stream component and realizing data recovery.
In order to achieve the above purpose, the invention adopts the following technical scheme:
the hierarchical large-flow collaborative monitoring method is characterized by comprising the following steps of:
step 1, a control plane formulates a task deployment strategy and a network flow selection strategy and sends the task deployment strategy and the network flow selection strategy to a data plane;
and 3, carrying out data merging and data recovery on the measurement data uploaded by each switch in the data plane, and carrying out inquiry on the hierarchical large-flow identification result by the control plane to obtain a measurement report.
Further, in step 1, the task deployment policy and the network flow selection policy satisfy the following joint optimization constraint:
x＝(x jl ∈{0,1}:v j ∈V,t l ∈T)，
y＝(y ijl ∈{0,1}:f i ∈F,v j ∈V,t l ∈T)，
y ijl ＜x jl ,f i ∈F,v j ∈V,t l ∈T，
wherein V represents a switch set, F represents a network flow set, and T represents a task set; f (f) i Representing the ith network flow, v, in the set of network flows F j Represents the jth switch, F, in the switch set V j Pass-through switch v represented in network flow set F j V of network flows of (a) i Representing network flow f i A set of switches traversed, t l Representing the first task in the task set T, r l Representing task t l Occupy the size of the storage space, R j Representing a switch v j Is a storage space constraint of (1); x and y are both indication vectors, respectively indicate task deployment and network flow selection policies, x jl For task t l At the exchange v j An indication variable of whether or not to deploy: x is x jl =1 represents task t l At the exchange v j Is deployed in x jl =0 denotes task t l At the exchange v j Not deployed in (y) ijl For network flow f i Task t of (2) l At the exchange v j An indicator variable of whether or not to measure: y is ijl =1 denotes network flow f i Task t of (2) l At the exchange v j Middle measurement, y ijl =0 denotes network flow f i Task t of (2) l At the exchange v j Not measured.
Further, the solving process of the joint optimization constraint is as follows: linear relaxation is carried out on the joint optimization constraint and the solution is carried out, so that an indication vector is obtained
And->
For indication vectors
For->
Is->
With probability->
Will x jl Set to 1; setting an inlet switch to deploy all measurement tasks;
for indication vectors
Adopting off-line solution or on-line solution;
offline solution: for each network flow f i Each task t l According to probability
On its forwarding path is deployed task t l One of the internal switches of (a) is selected and the corresponding indicated variable y is to be used ijl Set to 1; if no internal switch is selected, selecting the corresponding inlet switch, and correspondingly indicating variable y ijl Set to 1;
on-line solution: using greedy ideas in network flow f i For each task t when first arriving l Select in network flow f i The forwarding path is deployed with a task t l The switch with the smallest current measurement load is measured, and the corresponding indication variable y is used for ijl Set to 1.
Further, in step 2, the data structure of each measurement task deployed on each switch in the data plane includes three parts: bloom filters, large flow components, and small flow components; the bloom filter is used for realizing the collaborative monitoring of the network flow by the whole network switch, and comprises the following components: realizing the indicated variable y ijl Inquiring generalized prefix of network flow corresponding to the task measured by the current exchanger; the large flow component is used for storing identifiers and count values of candidate large flows, and the small flow component is used for storing count values of network flows thrown out by the large flow component.
Further, in step 2, the specific process of task measurement is: for each network flow f i Is set to each measurement task t l In network flow f i Determining, by bloom filters, in each switch on the forwarding path whether task t should be performed at the current switch l Is a measurement of (2); if yes, according to granularity of task splitting, the current task t is processed l Corresponding network flow f i The generalized prefix of (1) is used as an identifier to be inserted into a large-flow component and a bloom filter, and the count value corresponding to the large-flow component is updated; and if the network flow f 'is thrown out of the large-flow component' i The network flow f i ' insert into the small flow component and update the corresponding calculated value of the small flow component.
Further, in step 3, the specific process of data synthesis is as follows: a list of candidate large flows is obtained from the large flow component of each switch in the data plane and the count values of network flows having the same identifier are combined as an estimate of the network flow size.
Further, in step 3, the specific process of data recovery is: inquiring bloom filters of each switch for identifiers of each network flow according to the candidate large flow list, and judging whether the current switch measures the network flow or not; inquiring the large flow component of the network flow when the inquiring result is true, judging whether the identifier of the network flow exists or not, returning to 0 when the identifier exists, inquiring the small flow component of the network flow when the identifier of the network flow does not exist, and returning to the count value; and accumulating the returned count value of the small flow component and the estimated value of the current network flow to be used as a new network flow estimated value, thereby completing the recovery process of the network flow measurement data.
Further, in step 3, the specific process of the result query is: the result inquiry starts from the bottommost layer, and for each network flow, whether the estimated value of the network flow is larger than a preset threshold value or not is judged; if yes, reporting the parent prefix as a large stream of the current level, and deducting count values of all parent prefixes by using the estimated value; the above operations are performed from bottom to top, and the large flows identified in each hierarchy are summarized together into hierarchical large flows identified in the whole network.
In summary, the beneficial effects of the invention are as follows: the hierarchical large-flow collaborative monitoring method mainly comprises the following steps:
1. task deployment and network flow selection joint optimization problem:
1) Deployment problem of hierarchical large-flow measurement tasks in distributed network environment
In hierarchical large-flow collaborative monitoring, each network flow needs to complete corresponding measurement in h hierarchies, and the measurement of each hierarchy l from bottom to top is regarded as a measurement task, and the h measurement tasks are deployed into a switch of the whole network under a certain constraint condition, so that the invention defines the h measurement tasks as task deployment problems;
2) Selective measurement of a flow through a network by a network measurement switch
In hierarchical large-flow collaborative monitoring, each measurement task of each network flow selects one switch to measure from switches through which the task is deployed, and the invention defines the task as a network flow selection problem;
3) Hierarchical large-flow measurement task deployment and network flow selection joint optimization problem
The invention obtains task deployment and network flow selection strategies by modeling task deployment and network flow selection joint optimization problems and approximately solving the problems; the approximate solution scheme has strong adaptability, so that the load of the whole network switch is more balanced, the utilization of resources is more fully and reasonably realized, and a better load balancing effect is realized; meanwhile, under the better load balancing effect, the precision of the hierarchical large-flow measurement is improved.
2. Distributed hierarchical large flow measurement framework:
1) Distributed collaborative monitoring using bloom filters
The bloom filter is used for realizing the collaborative monitoring of the switch of the whole network to the network flow, and mainly has two functions: for indicating variable y according to ijl Implementing a selection function for monitoring network flows on the switch, i.e. querying the current network flow f i Task t of (2) l Whether or not the switch v j Is measured; the method comprises the steps of inserting a generalized prefix of a corresponding network flow into a bloom filter in a task measurement process to inquire whether the network flow identified by the prefix is subjected to task measurement in the current switch according to the generalized prefix in a data recovery process;
2) Monitoring structural design of large-flow component and small-flow component combination
The large flow component is used for storing identifiers and count values of candidate large flows, most of flow can be filtered by setting the large flow component, and the overestimation problem of the small flow component is relieved; the large flow component may specifically be a counter-based algorithm (e.g., space save); the small flow component is used for storing the count value of the network flow thrown out by the large flow component, namely, the count value used for reserving the small flow is very important for recovering the lost count value; the streamlet component may specifically be a Sketch (Sketch) based algorithm (e.g., count-Min Sketch);
the distributed hierarchical large-flow measurement framework provided by the invention can support the full-network switch to cooperatively monitor the hierarchical large flow through the bloom filter, and can improve the measurement accuracy of the hierarchical large flow after the lost sub-flow is recovered through selecting proper data structures for measurement in the large-flow component and the small-flow component.
Drawings
FIG. 1 is a schematic diagram of a software defined measurement system according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of task deployment and network flow selection policies in an embodiment of the invention.
FIG. 3 is a schematic diagram of a distributed hierarchical high-flow measurement framework in an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the embodiments and the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present invention more apparent.
The invention aims to perform hierarchical large-flow collaborative monitoring on the whole network and make the measurement load of each switch equivalent; in short, each network flow can complete all measurement tasks through the cooperation of a plurality of switches on the forwarding path, and each switch only needs to deploy part of measurement tasks, so that the resource utilization rate is improved in a finer granularity mode; among these, it is important to ensure that each network flow can perform all measurement tasks through the cooperation of multiple switches on its forwarding path. In collaborative monitoring, on one hand, the measurement tasks of the hierarchical large flow are expected to be deployed to the switches of the whole network under the constraint of resources, and the method is called task deployment; on the other hand, a switch with a task deployed is selected from the switches on the forwarding path of each network flow for task measurement, which is called network flow selection.
Based on the above, the invention provides a hierarchical large-flow collaborative monitoring method, which comprises the following steps: task deployment, network flow selection and distributed hierarchical large flow measurement; the task deployment and network flow selection strategy is used for acquiring task deployment and network flow selection strategies for hierarchical large-flow collaborative monitoring; the distributed hierarchical large-flow measurement is used for carrying out collaborative monitoring of hierarchical large-flow according to a strategy and periodically carrying out merging, recovery and query of data; further details are provided below in connection with the examples.
Example 1
The embodiment provides a hierarchical large-flow collaborative monitoring method, which is implemented based on a software-defined measurement system, wherein the software-defined measurement system is shown in fig. 1, and specifically comprises the following steps: a control plane and a data plane, wherein the data plane is formed by a plurality of programmable switches; the control plane is responsible for formulating task deployment and network flow selection strategies, issuing the strategies to a programmable switch of the data plane, and merging, recovering and reporting results of information periodically uploaded by the data plane; the programmable switch of the data plane is responsible for measuring network flows according to the strategy of the control plane and periodically collecting data and uploading the data to the control plane.
In this embodiment, the hierarchical large-flow collaborative monitoring method based on the software defined measurement system includes the following steps:
step 1, a control plane formulates a task deployment strategy and a network flow selection strategy and sends the task deployment strategy and the network flow selection strategy to a data plane;
the task deployment strategy refers to: which measurement tasks are deployed on each switch, the network flow selection policy refers to which switch each measurement task of each network flow is selected to be performed on; the task deployment strategy and the network flow selection strategy are formulated as key points of collaborative monitoring, and therefore, the task deployment and network flow selection combined optimization problem is established as follows:
the constraint conditions are as follows:
x＝(x jl ∈{0,1}:v j ∈V,t l ∈T) (2)
y＝(y ijl ∈{0,1}:f i ∈F,v j ∈V,t l ∈T) (3)
y ijl ＜x jl ,f i ∈F,v j ∈V,t l ∈T (5)
wherein V represents a switch set, F represents a network flow set, and T represents a task set; f (f) i Representing the ith network flow, v, in the set of network flows F j Represents the jth switch, F, in the switch set V j Pass-through switch v represented in network flow set F j V of network flows of (a) i Representing network flow f i A set of switches traversed, t l Representing the first task in the task set T, r l Representing task t l Occupy the size of the storage space, R j Representing a switch v j Is a storage space constraint of (1);
in the task deployment and network flow selection combined optimization problem, the combined optimization target shown in the formula (1) is to minimize the maximum measurement load of the whole network switch, and find the task deployment and network flow selection strategy, so as to obtain x and y vector types; (2) X and y in the formula (3) are both indication vectors, and respectively indicate task deployment and network flow selection strategies, wherein x is as follows jl For task t l At the exchange v j An indication variable of whether or not to deploy: x is x jl =1 represents task t l At the exchange v j Is deployed in x jl =0 denotes task t l At the exchange v j Not deployed in (y) ijl For network flow f i Task t of (2) l At the exchange v j An indicator variable of whether or not to measure: y is ijl =1 denotes network flow f i Task t of (2) l At the exchange v j Middle measurement, y ijl =0 denotes network flow f i Task t of (2) l At the exchange v j Is not measured; a kind of electronic device with high-pressure air-conditioning system(4) For ensuring each network flow f i Each task t l Switch V capable of being on its forwarding path i Selecting one switch for measurement; equation (5) is used to ensure network flow f i Selected task t l Is a switch v of (1) j The task must be deployed; equation (6) is used to ensure deployment at switch v j The total memory size occupied by all measurement tasks in (a) should not exceed its memory capacity limit.
Further, since the above joint optimization problem is an integer programming problem, the present embodiment performs linear relaxation and solves it by replacing constraint equations (2) and (3) with the following equations:
x＝(x jl ∈[0,1]:v j ∈V,t l ∈T) (7)
y＝(y ijl ∈[0,1]:f i ∈F,v j ∈V,t l ∈T) (8)
solutions derived from linear relaxation (in order to respectively
And->
Representation) is of the floating point type, so the present embodiment obtains an integer solution by rounding according to probability, specifically:
1) Task deployment strategy: for the purpose of
Is->
With probability->
Will x jl Set to 1;
in particular, in order to ensure that each task of each network flow must find a switch to execute, for the result of the above approximate solution, an ingress switch is set to deploy all measurement tasks;
2) Network flow selection policy: an off-line or on-line mode is adopted;
offline: for each network flow f i Each task t l According to probability
On its forwarding path is deployed task t l One of the internal switches of (a) is selected and the corresponding indicated variable y is to be used ijl Set to 1; if no internal switch is selected, selecting the corresponding inlet switch, and correspondingly indicating variable y ijl Set to 1;
on-line: using greedy ideas in network flow f i For each task t when first arriving l Select in network flow f i The forwarding path is deployed with a task t l The switch with the least current measurement load is used for measurement, namely the corresponding indication variable y ijl Set to 1.
further, in the data plane, the data structure of each measurement task deployed on each switch includes three parts: bloom filters, large flow components, and small flow components; wherein, the liquid crystal display device comprises a liquid crystal display device,
the bloom filter is used for realizing the collaborative monitoring of the switch of the whole network to the network flow, and mainly has two functions: for indicating variable y according to ijl Implementing a selection function for monitoring network flows on the switch, i.e. querying the current network flow f i Task t of (2) l Whether or not the switch v j Is measured; the method comprises the steps of inserting a generalized prefix of a corresponding network flow into a bloom filter in a task measurement process to inquire whether the network flow identified by the prefix is subjected to task measurement in the current switch according to the generalized prefix in a data recovery process;
the large flow component is used for storing identifiers and count values of candidate large flows, and most of flow can be filtered by setting the large flow component, so that the overestimation problem of the small flow component is relieved; the large flow component may specifically be a counter-based algorithm (e.g., space save);
the small flow component is used for storing the count value of the network flow thrown out by the large flow component, namely, the count value of the small flow is reserved, and the recovery of the lost count value is very important; the streamlet component may specifically be a Sketch (Sketch) based algorithm (e.g., count-Min Sketch);
further, the specific process of task measurement is as follows: for each network flow f i Is set to each measurement task t l In network flow f i Determining, by bloom filters, in each switch on the forwarding path whether task t should be performed at the current switch l Is a measurement of (2); if yes, according to granularity of task splitting, the current task t is processed l Corresponding network flow f i The generalized prefix of (1) is used as an identifier to be inserted into a large-flow component and a bloom filter, and the count value corresponding to the large-flow component is updated; and if the network flow f is thrown out of the large-flow component i ' the network flow f is then set i ' insert into the small flow component and update the corresponding calculated value of the small flow component.
Step 3, the control plane performs data merging and data recovery on the measurement data uploaded by each switch in the data plane, and performs inquiry on the hierarchical large-flow identification result to obtain a measurement report;
further, the specific process of the data set is as follows: obtaining a candidate large flow list from a large flow part of each switch in a data plane, and combining count values of network flows with the same identifier as an estimated value of the network flow size;
the specific process of data recovery is as follows: inquiring bloom filters of each switch for identifiers of each network flow according to the candidate large flow list, and judging whether the current switch measures the network flow or not; querying its large flow component when the query result is true, determining whether there is an identifier of the network flow (no operation is required if the query result is not true), returning 0 when there is, indicating that there is no count value on the current switch that may be lost, querying its small flow component and returning the count value when there is no; accumulating the returned count value of the small flow component and the estimated value of the existing network flow to be used as a new network flow estimated value, thereby completing the recovery process of the network flow measurement data;
the specific process of the result query is as follows: the result inquiry starts from the bottommost layer, and for each network flow, whether the estimated value of the network flow is larger than a preset threshold value or not is judged; if yes, reporting the parent prefix as a large stream of the current level, and deducting the count values of all the parent prefixes by using the estimated value (if not, not performing any operation); the operations are performed from bottom to top, and the identified streamlets in each hierarchical report are summarized together into a hierarchical streamlet identified in the entire network.
Fig. 2 is a schematic diagram of a task deployment policy and a network flow selection policy in this embodiment, where the network topology of the data plane is composed of 5 switches: v 1 、v 2 、v 3 、v 4 And v 5 A total of 6 network flows: f (f) 1 、f 2 、f 3 、f 4 、f 5 And f 6 Each network flow needs to complete 5 levels of measurement tasks t 1 、t 2 、t 3 、t 4 And t 5 Each switch can only deploy 3 measurement tasks under the constraint of resources; therefore, it is impossible for any one network flow to complete all measurement tasks on only one switch; as can be seen from fig. 2, the data plane has deployed on each switch a partial measurement task, e.g., task t, according to a task deployment policy from the control plane 2 、t 4 And t 5 Deployed at switch v 1 On task t 1 、t 3 And t 5 Deployed at switch v 2 Applying; there are two possible forwarding paths a-v between hosts a and B 1 -v 2 -v 3 -B and A-v 1 -v 4 -v 3 B, according to the task deployment strategy given in FIG. 2, the network flows on any forwarding path can complete all measurement tasks because theyMeasurements can be provided for all 5 tasks in concert by multiple switches; likewise, all measurement tasks can be on path A-v between hosts A and C 1 -v 4 -v 5 Complete at-C. In order that each network flow can complete all measurement tasks under the cooperation of a plurality of switches, a corresponding switch for completing each measurement task is selected for each network flow according to a network flow selection strategy in the figure, wherein the strategy can ensure that each network flow completes all measurement tasks under the cooperation of a plurality of switches, and the measurement load of each switch is 6; for example, network flow t 4 Task t of (2) 4 At the exchange v 1 Upper measurement, task t 1 、t 3 And t 5 At the exchange v 2 Upper measurement, task t 2 At the exchange v 3 Upper measurement; and for network flow f 2 Task t of it 2 And t 5 Select at switch v 1 Upper measurement, task t 1 And t 4 Select at switch v 3 Upper measurement, task t 3 Then select at switch v 4 And (5) measuring. Therefore, compared with the strategy that all measurement tasks are completed under the same switch by network flows (assuming that all measurement tasks can be deployed on one switch, at least one switch needs to measure all tasks of two network flows, namely, the measurement load of at least one switch is 10), the invention completes the measurement tasks through cooperative monitoring of the fine-grained switches, the measurement load of the switches is more balanced, and the utilization of resources is more sufficient.
FIG. 3 is a schematic diagram of a distributed hierarchical large flow measurement framework in which the large flow component selects Space save and the small flow component selects CM Sketch (Count-Min Sketch); the data plane contains three switches, and the current data stream is updated according to the update algorithm of the corresponding structure and the update process of the framework. The control plane obtains candidate streamlets { (10.1.1..40), (10.1.2..9), (10.2.1..19) } from streamlet components of three switches in the data plane; at this time, it can be known from the candidate large flow list that the identifiers of the two network flows whose count values are the largest are (10.1.1.+ -.) and (10.2.1.+ -.), respectively, given by the accurate results(10.1.1) and (10.1.2) are not identical; further, from the candidate large stream list, recovery of the missing values from the small stream unit is started: network flows (10.1.1.) are at switch v due to the query results according to the bloom filter 2 Measured in (b), but at v 2 No corresponding record exists in the large stream component, so that the small stream component is queried, and the possible lost count value is 8; similarly, for network flows (10.1.2.) it exists at switch v 1 And thus a possible missing count value of 20 can be obtained; however, for network flows (10.2.1.) it is recorded at switch v 3 The count value that may be lost is thus 0; after recovering the lost count value, new candidate large flows are available at this point as { (10.1.1..48), (10.1.2..29), (10.2.1..19) }, where the identifiers of the two network flows with the largest count values are (10.1.1..and (10.1.2.), respectively, consistent with accurate results.
While the invention has been described in terms of specific embodiments, any feature disclosed in this specification may be replaced by alternative features serving the equivalent or similar purpose, unless expressly stated otherwise; all of the features disclosed, or all of the steps in a method or process, except for mutually exclusive features and/or steps, may be combined in any manner.
Claims (5)
1. The hierarchical large-flow collaborative monitoring method is characterized by comprising the following steps of:
step 1, a control plane formulates a task deployment strategy and a network flow selection strategy and sends the task deployment strategy and the network flow selection strategy to a data plane;
the task deployment policy and the network flow selection policy satisfy the following joint optimization constraints:
x＝(x jl ∈{0,1}:v j ∈V,t l ∈T)，
y＝(y ijl ∈{0,1}:f i ∈F,v j ∈V,t l ∈T)，
y ijl ＜x jl ,f i ∈F,v j ∈V,t l ∈T，
wherein V represents a switch set, F represents a network flow set, and T represents a task set; f (f) i Representing the ith network flow, v, in the set of network flows F j Represents the jth switch, F, in the switch set V j Pass-through switch v represented in network flow set F j V of network flows of (a) i Representing network flow f i A set of switches traversed, t l Representing the first task in the task set T, r l Representing task t l Occupy the size of the storage space, R j Representing a switch v j Is a storage space constraint of (1); x and y are both indication vectors, respectively indicate task deployment and network flow selection policies, x jl For task t l At the exchange v j An indication variable of whether or not to deploy: x is x jl =1 represents task t l At the exchange v j Is deployed in x jl =0 denotes task t l At the exchange v j Not deployed in (y) ijl For network flow f i Task t of (2) l At the exchange v j An indicator variable of whether or not to measure: y is ijl =1 denotes network flow f i Task t of (2) l At the exchange v j Middle measurement, y ijl =0 denotes network flow f i Task t of (2) l At the exchange v j Is not measured;
step 2, each switch in the data plane performs task deployment according to a task deployment strategy, and simultaneously performs task measurement of network flows according to a network flow selection strategy, and periodically uploads measurement data to the control plane;
the data structure of each measurement task deployed on each switch in the data plane contains three parts: bloom filters, large flow components, and small flow components; the bloom filter is used for realizing the collaborative monitoring of the network flow by the whole network switch, and comprises the following components: realizing the indicated variable y ijl Inquiring generalized prefix of network flow corresponding to the task measured by the current exchanger; the large flow component is used for storing identifiers and count values of candidate large flows, and the small flow component is used for storing count values of network flows thrown out by the large flow component;
the specific process of task measurement is as follows: for each network flow f i Is set to each measurement task t l In network flow f i Determining, by bloom filters, in each switch on the forwarding path whether task t should be performed at the current switch l Is a measurement of (2); if yes, according to granularity of task splitting, the current task t is processed l Corresponding network flow f i The generalized prefix of (1) is used as an identifier to be inserted into a large-flow component and a bloom filter, and the count value corresponding to the large-flow component is updated; and if the network flow f is thrown out of the large-flow component i ' the network flow f is then set i ' insert into the small flow component, and update the corresponding calculated value of the small flow component;
and 3, carrying out data merging and data recovery on the measurement data uploaded by each switch in the data plane, and carrying out inquiry on the hierarchical large-flow identification result by the control plane to obtain a measurement report.
2. The hierarchical large-flow collaborative monitoring method according to claim 1, wherein in step 3, the specific process of data merging is: a candidate large flow list is obtained from the large flow component of each switch in the data plane and the count values of network flows having the same identifier are combined as an estimate of the network flows.
3. The hierarchical large-flow collaborative monitoring method according to claim 1, wherein in step 3, the specific process of data recovery is: inquiring bloom filters of each switch for identifiers of each network flow according to the candidate large flow list, and judging whether the current switch measures the network flow or not; inquiring the large flow component of the network flow when the inquiring result is true, judging whether the identifier of the network flow exists or not, returning to 0 when the identifier exists, inquiring the small flow component of the network flow when the identifier of the network flow does not exist, and returning to the count value; and accumulating the returned count value of the small flow component and the estimated value of the current network flow to serve as a new network flow estimated value, namely finishing the data recovery of the network flow measurement data.
4. The hierarchical large-flow collaborative monitoring method according to claim 1, wherein in step 3, the specific process of the result query is: the result inquiry starts from the bottommost layer, and for each network flow, whether the estimated value of the network flow is larger than a preset threshold value or not is judged; if yes, reporting the parent prefix as a large stream of the current level, and deducting count values of all parent prefixes by using the estimated value; the operations described above are performed from bottom to top, summarizing the large flows identified in each hierarchy into hierarchical large flows identified in the entire network.
5. The hierarchical large-flow collaborative monitoring method according to claim 1, wherein the solving process of the joint optimization constraint is as follows: linear relaxation is carried out on the joint optimization constraint and the solution is carried out, so that an indication vector is obtained
And->
For indication vectors
For->
Is->
With probability->
Will x jl Set to 1; setting an inlet switch to deploy all measurement tasks;
for indication vectors
Adopting off-line solution or on-line solution;
offline solution: for each network flow f i Each task t l According to probability
On its forwarding path is deployed task t l One of the internal switches of (a) is selected and the corresponding indicated variable y is to be used ijl Set to 1; if no internal switch is selected, selecting the corresponding inlet switch, and correspondingly indicating variable y ijl Set to 1;
on-line solution: using greedy ideas in network flow f i For each task t when first arriving l Select in network flow f i The forwarding path is deployed with a task t l The switch with the smallest current measurement load is measured, and the corresponding indication variable y is used for ijl Set to 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210526869.4A CN114785707B (en) | 2022-05-16 | 2022-05-16 | Hierarchical large-flow collaborative monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210526869.4A CN114785707B (en) | 2022-05-16 | 2022-05-16 | Hierarchical large-flow collaborative monitoring method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114785707A CN114785707A (en) | 2022-07-22 |
| CN114785707B true CN114785707B (en) | 2023-06-20 |
Family
ID=82437999
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210526869.4A Active CN114785707B (en) | 2022-05-16 | 2022-05-16 | Hierarchical large-flow collaborative monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114785707B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116032829B (en) * | 2023-03-24 | 2023-07-14 | 广东省电信规划设计院有限公司 | SDN network data stream transmission control method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106357547A (en) * | 2016-09-08 | 2017-01-25 | 重庆邮电大学 | Software-defined network congestion control algorithm based on stream segmentation |
| CN113839835A (en) * | 2021-09-27 | 2021-12-24 | 长沙理工大学 | Top-k flow accurate monitoring framework based on small flow filtering |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8276148B2 (en) * | 2009-12-04 | 2012-09-25 | International Business Machines Corporation | Continuous optimization of archive management scheduling by use of integrated content-resource analytic model |
| US9252972B1 (en) * | 2012-12-20 | 2016-02-02 | Juniper Networks, Inc. | Policy control using software defined network (SDN) protocol |
| CN104301246A (en) * | 2014-10-27 | 2015-01-21 | 盛科网络（苏州）有限公司 | Large-flow load balanced forwarding method and device based on SDN |
| CN107395693A (en) * | 2017-07-04 | 2017-11-24 | 大连工业大学 | The hospital clinical operation data selection equipment for the size stream classification applied in cloud data center system |
| CN112367217B (en) * | 2020-10-20 | 2021-12-17 | 武汉大学 | Cooperative type large flow detection method and system oriented to software defined network |
-
2022
- 2022-05-16 CN CN202210526869.4A patent/CN114785707B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106357547A (en) * | 2016-09-08 | 2017-01-25 | 重庆邮电大学 | Software-defined network congestion control algorithm based on stream segmentation |
| CN113839835A (en) * | 2021-09-27 | 2021-12-24 | 长沙理工大学 | Top-k flow accurate monitoring framework based on small flow filtering |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114785707A (en) | 2022-07-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108600102B (en) | Flexible data transmission system based on intelligent cooperative network | |
| CN1925437B (en) | System and method for detecting status changes in a network | |
| Papagiannaki et al. | A distributed approach to measure IP traffic matrices | |
| Sachidananda et al. | Quality of information in wireless sensor networks: A survey | |
| US7403988B1 (en) | Technique for autonomous network provisioning | |
| CN112564964B (en) | Fault link detection and recovery method based on software defined network | |
| US9203857B2 (en) | Method and system for detecting anomaly of user behavior in a network | |
| CN114785707B (en) | Hierarchical large-flow collaborative monitoring method | |
| CN101536428B (en) | Method for tracking network parameters | |
| CN107911242A (en) | A kind of cognitive radio based on industry wireless network and edge calculations method | |
| CN109034562A (en) | A kind of social networks node importance appraisal procedure and system | |
| CN108601047B (en) | Measurement method of opportunistic network key node | |
| Huang et al. | Dynamic coverage in ad-hoc sensor networks | |
| CN111130928A (en) | Network measurement method based on in-band detection in wide area network | |
| CN117729567B (en) | Optimization method and system for wireless ad hoc network | |
| CN104539471A (en) | Bandwidth metering method and device and computer equipment | |
| CN102082701B (en) | Method for storing network element positional information and apparatus for same | |
| CN114915995A (en) | Network slice monitoring method, system and storage medium based on in-band network telemetry | |
| CN109952743B (en) | System and method for low memory and low flow overhead high flow object detection | |
| CN116455729A (en) | Fault link detection and recovery method based on link quality assessment model | |
| CN113507396B (en) | Network state analysis method, device, equipment and machine-readable storage medium | |
| CN112994970A (en) | In-band network telemetry INT method and system based on capture and arrangement | |
| JP4342795B2 (en) | Supervisory control device | |
| CN115085985B (en) | Memory high-efficiency range base number measuring method for network security monitoring | |
| KR102492409B1 (en) | Multi/hybrid cloud real-time monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |