CN110572332B - Network equipment message observation data acquisition task dividing method - Google Patents

Network equipment message observation data acquisition task dividing method Download PDF

Info

Publication number
CN110572332B
CN110572332B CN201910816830.4A CN201910816830A CN110572332B CN 110572332 B CN110572332 B CN 110572332B CN 201910816830 A CN201910816830 A CN 201910816830A CN 110572332 B CN110572332 B CN 110572332B
Authority
CN
China
Prior art keywords
observation data
message
tcp
communication
message observation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910816830.4A
Other languages
Chinese (zh)
Other versions
CN110572332A (en
Inventor
罗寿西
邢焕来
李可
戴朋林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Yuxin Intelligent Technology Co ltd
Original Assignee
Southwest Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southwest Jiaotong University filed Critical Southwest Jiaotong University
Priority to CN201910816830.4A priority Critical patent/CN110572332B/en
Publication of CN110572332A publication Critical patent/CN110572332A/en
Application granted granted Critical
Publication of CN110572332B publication Critical patent/CN110572332B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/12—Avoiding congestion; Recovering from congestion
    • H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Environmental & Geological Engineering (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention relates to network measurement task allocation in the communication field, in particular to a network equipment message observation data acquisition task dividing method, which comprises the following operations: counting all network equipment which can generate message observation data for the TCP stream; setting a communication pair; sequencing all communication pairs from large to small according to the number of the TCP streams contained in estimation; sequentially determining the measurement task load values corresponding to the communication pairs according to the sequence obtained after sequencing and distributing the measurement tasks to different network devices; constructing a directed acyclic graph for all paths consisting of communication pairs which have equivalent paths, simultaneously have a plurality of transmission paths and pass through each transmission path and have known TCP stream number; and measuring the communication node on a link to obtain a measured value, and sequentially selecting specific measuring nodes according to the measured value. By adopting the method, repeated measurement can be avoided, and the measurement task load on each input port of each network device is balanced.

Description

Network equipment message observation data acquisition task dividing method
Technical Field
The invention relates to network measurement task allocation in the communication field, in particular to a network equipment message observation data acquisition task division method.
Background
In application scenarios such as data centers, the network is better managed. Network managers need to monitor and count the size of individual TCP flows (i.e., the amount of data transmitted) in the network in order to detect network attacks, diagnose network failures, optimize network routing, etc. For such a flow measurement requirement, the measurement tools supported by the existing network devices are mainly netflow and sflow. netflow requires complex data processing on network devices and special hardware support on high-speed networks, and therefore currently, support is not widespread. To reduce the amount of data that needs to be collected and processed, sflow randomly samples the data that passes through before being sent to a remote server for further analysis. The design reduces the extra overhead introduced by the test, but the measurement accuracy is limited and the method is not friendly to burst small flow. In addition, some research schemes adopt a selective data mirroring scheme to send part or all of data mirrors to a remote server for processing, and the problems of high test overhead also exist in the research schemes; other research schemes perform sketch calculation on the data, report sketch results to control, and have the problem of measurement precision loss. In summary, the TCP packet observation data acquisition scheme of the existing network device is complex in calculation and processing, requires high-performance network hardware to implement measurement in a high-speed network, or has coarse measurement accuracy, is not friendly to measurement of burst streamlets, has large bandwidth overhead and CPU processing overhead introduced by measurement, has weak expansibility, and is unbalanced in task processing amount in the existing network device, thereby increasing the load to be borne by individual network devices.
Disclosure of Invention
The invention mainly aims to provide a network equipment message observation data acquisition task dividing method for reducing bandwidth overhead.
In order to achieve the above object, the technical solution adopted by the present application is a method for dividing a task of acquiring packet observation data of a network device, comprising the following operations:
counting all network equipment which can generate message observation data for the TCP stream;
setting a group of communication requirements with the same sending node and receiving node as a communication pair;
sequencing all communication pairs from large to small according to the number of the TCP streams contained in estimation;
according to the sequence obtained after sequencing, sequentially determining the message observation data acquisition tasks corresponding to each communication pair, and distributing the message observation data acquisition tasks to different network equipment;
constructing a directed acyclic graph by using all paths which are simultaneously provided with a plurality of transmission paths and formed by communication pairs with known TCP (transmission control protocol) stream numbers passing through each transmission path, wherein one directed edge of the directed acyclic graph corresponds to one unidirectional link on the transmission path; and measuring the message observation data acquisition tasks of the communication node transmission paths on the links to obtain measured values, and sequentially selecting specific message observation data acquisition nodes for the TCP streams on each transmission path of each communication pair according to the measured values on each link.
That is, a plurality of devices in the network have fine-grained TCP message observation data acquisition capability, the controller can divide and distribute message observation data acquisition tasks of all TCP streams to different devices, and processing burden of each device is reduced. By adopting the message observation data acquisition task division method, repeated measurement can be avoided, and the load of the message observation data acquisition tasks of each network device can be balanced.
Further, the communication pair is a transmission path from one subnet or host to another subnet or host.
Further, all the communication pairs are sorted from large to small according to the number of the TCP flows contained in the estimation, and the communication pairs also comprise information of the number of the TCP flows from one subnet or host to another subnet or host.
Further, only when a certain TCP flow is set to detect on one of the network devices, the network device performs packet detection processing on the TCP flow and generates packet observation data as needed.
Further, for a group of communication pairs with a sending node s and a receiving node d, selecting a specific message observation data acquisition node comprises the following steps:
1) setting a weight for each edge in the directed acyclic graph, and taking a value to represent an estimated value of the number of TCP streams required to be processed by a message observation data acquisition task arranged on the link;
2) using a minimum cut algorithm to calculate a minimum edge cut set from s to d in the directed acyclic graph, so that the s to d can not be reached any more after the directed edges in the set are deleted; if a plurality of minimum edge cut sets exist, one minimum edge cut set is randomly selected;
3) and for the directed edge u-v from the node u to the node v in the selected directed edge set E, setting the node v as a message observation data acquisition node of a flow contained in a communication pair from s to d, accumulating the TCP flow number passing through a link u-v in the communication pair to a weight value corresponding to the directed edge u-v in the network, and updating.
Further, the directed edge set E is a directed edge set in which the sum of directed edge weights in the selected edge cut sets is the minimum.
The invention also provides a TCP flow measurement method which can obviously reduce the communication overhead between the network equipment and the controller and provide fine-grained measurement capability.
In order to achieve the purpose, the technical scheme adopted by the application is an event-driven fine-grained TCP flow measurement method, which comprises the steps of identifying and acquiring TCP message observation data on network equipment, and analyzing and estimating the transmitted data volume of TCP on a remote logic integrated controller;
after receiving the specified type of TCP message, the network device performs the following operations: caching the observation data of the message in local network equipment;
if the position to be cached has the content, the message observation data content caching fails; directly sending the observation data content of the message to be cached and the content I of the corresponding position in the cache to a controller, and accommodating the cache position to be empty;
and if the position to be cached does not have the content, the message successfully observes the caching of the data content.
Further, when the content of the message observation data to be cached and the content of the corresponding position in the cache are directly sent to the controller, the network device periodically reports the content of the message observation data and the content of the corresponding position in the cache to the controller, so that the frequency of content reporting and the transmission overhead are reduced.
Furthermore, when a plurality of network devices in the network all have TCP message observation data acquisition capacity, the remote controller divides the message observation data acquisition tasks of different TCP streams according to the routing of the TCP streams and distributes the tasks to different devices for processing.
Further, after the network device receives a TCP packet of a specific type, the type of the TCP packet, the arrival time of the packet, the quintuple information, and the SEQ value, that is, the observation data of the packet are sent to the control plane of the network device.
Further, the network device control plane sends the message observation data and the device identity thereof to a remote controller; and the remote controller calculates the size information of each TCP stream according to the message observation data fed back by all the network equipment.
Further, when the remote controller calculates the size information of each TCP flow according to the packet observation data fed back by all the network devices, the remote controller does not repeatedly perform calculation on the packet observation data with the same quintuple information that repeatedly appears within the set time period T.
Further, the method for measuring flow of event-driven fine-grained TCP further includes collecting message observation data on the network device, and includes the following operations:
after receiving a TCP packet, if the current packet is any one of a SYN packet, a FIN packet, or a packet with SEQ ═ 0, the network device data forwarding plane generates observation data for the TCP packet, and sends the observation data to the control plane of the network device;
the control plane of the network equipment caches the message observation data submitted by the data forwarding plane;
after receiving the message observation data, the control plane performs the following processing on the message observation data:
s1, performing hash calculation on a quintuple field in the message observation data to obtain an integer number f between 0 and N-1;
s2, when the message observation data is cached in the fth unit in the cache, sending the observation data cached in the unit and the current message observation data to a remote controller together, and emptying the cached content in the fth unit;
when the f-th unit in the cache does not cache the message observation data, caching the current message observation data into the f unit;
and S3, finishing the processing.
Compared with other message observation data acquisition methods, the method is a measurement method which supports fine granularity, has low overhead and is extensible, can accurately measure the transmission data volume of the TCP stream including the small stream, and can be realized on the existing hardware.
Further, the message observation data includes a type of the TCP message, a message arrival time, quintuple information, and an SEQ value.
Further, after receiving the message observation data, the control plane scans all the cached message observation data at intervals, sends the message observation data to the remote controller, and clears the cached observation data of the control plane.
Further, when the control plane sends data to the remote controller, the data includes the identity number of the network device.
Further, the control plane of the network device caches the message observation data submitted by the data forwarding plane through a continuous memory.
Further, the continuous memory includes N storage units, and the content stored in each of the N storage units includes the type of the cached observation data, the packet arrival time, the TCP five-tuple information, and the SEQ value.
Further, the types of the observation data cached in the memory include four types, which are:
NUL (0), representing uncached content;
SYN (1), which represents that the cached content is the observation data information of SYN message;
SEQ ═ 0(2) indicating observation data information of the cached message whose content is SEQ ═ 0;
and the FIN (3) represents that the cached content is observation data information of the FIN message.
Further, after the packet observation data on the network device is collected, estimating the transmitted data volume of each TCP flow, including:
splitting observation data of the TCP message according to the membership relation with the flow;
and (4) the transmitted data quantity of the flow is back estimated according to the TCP message observation data belonging to each flow.
Further, the above mentioned message observation data includes the type of TCP message, the message arrival time, the quintuple information and the SEQ value, and the TCP flow transmitted data amount estimation method includes the following operation steps:
s1, a controller receives TCP message observation data reported by acquisition equipment and groups the message observation data according to a quintuple value of the message observation data;
s2, for each group of message observation data, if a plurality of observation data have the same type and SEQ value and the difference of the capturing time (namely the message arrival time) is less than the set time T, the plurality of message observation data are observation data of retransmission data or observation data repeatedly captured by the same message on different equipment, and only one observation data is reserved;
s3, selecting observation data with the type of SYN from the redundancy-removed message observation data, and sequencing the observation data according to the sequence of the capturing time of the observation data;
s4, dividing time into a plurality of intervals according to the capturing time of the SYN type message observation data, dividing TCP message observation data in one time interval into one stream, and dividing the message observation data in the last interval into the last stream;
and S5, the controller re-estimates the transmitted data volume of the TCP stream according to the message observation data of the membership and each stream.
Compared with other methods, the method can correctly identify the wrapping which is possibly generated by the SEQ, can eliminate the interference introduced by route change and repeated message observation, and improves the accuracy of the estimation of the transmitted data quantity of the TCP.
Further, the method for estimating the transmitted data volume of the TCP stream of the remote controller, wherein the message observation data comprises the type of the TCP message, the message arrival time, quintuple information and the SEQ value, comprises the following operation steps:
s1, a controller receives message observation data and groups the message data according to a quintuple value of the message observation data;
s2, for each group of message observation data, if a plurality of message observation data have the same type and SEQ value and the difference of the capture time is less than the set time T, the plurality of message observation data are observation data of retransmission data or observation data repeatedly captured by the same message on different equipment, and only one observation data is reserved; the time T is a set threshold value;
s3, selecting observation data with the type of SYN from all the observation data of the messages after redundancy removal, and sequencing the observation data according to the sequence of the capturing time of the observation data;
s4, dividing the message observation data in a time interval into a stream, and dividing the message observation data in the last interval into the last stream;
and S5, the controller estimates the sent data quantity of the TCP stream according to the message observation data of the subordinate TCP stream and each TCP stream.
Further, the controller, when estimating the amount of data sent by the controller for the packet observation data belonging to each TCP flow, includes the following operations:
setting the SEQ in the SYN message as SEQ1, the SEQ value in the FIN message as SEQ2, and the occurrence frequency of the message with the value of SEQ ═ 0 as k;
the following calculations were performed:
1) if the flow has FIN message observation data, the size of the flow is SEQ2-SEQ1+ k 2^ 32;
2) if the flow has no FIN message observation data and k is greater than 0, the flow size is k × 2^32-SEQ1, and the estimation is inaccurate;
3) if the flow has no FIN packet observation data and k is 0, no estimation is performed.
Further, dividing the message observation data in a time interval into a TCP flow, specifically:
and the controller selects the message observation data with the type of SYN from all the message observation data after redundancy removal, and sorts the message observation data according to the sequence of the capturing time.
Further, the sorting specifically includes:
a plurality of SYN message observation data are arranged, after sequencing, the time is divided into a plurality of sections according to the capture time of the SYN message observation data, a plurality of continuously captured message observation data in a time interval are divided into an independent TCP stream, and the message observation data in the last interval are divided into the last stream.
Further, the specific sorting steps include:
1) setting m observation data of SYN message, and setting their capture time as t1, t2, t.i.t.m;
2) the controller divides all message observation data in a time interval [ T [ i ] -T1-T2, T [ i +1] -T1-T2) into the ith flow, wherein i is 1, 2. For all message observations after T [ m ] -T1-T2, it is considered to belong to the last flow, i.e., the mth flow.
The invention is further described with reference to the following figures and detailed description. Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description. Or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to assist in understanding the invention, and are included to explain the invention and their equivalents and not limit it unduly. In the drawings:
fig. 1 is a general flowchart for explaining an event-driven fine-grained TCP flow measurement method in the present embodiment;
fig. 2 is a flowchart for explaining a network device message observation data acquisition task dividing method in the event-driven fine-grained TCP flow measurement method according to the present embodiment;
fig. 3 is a flowchart for explaining data acquisition in the event-driven fine-grained TCP flow measurement method in the present embodiment;
fig. 4 is a flowchart for explaining a controller in the event-driven fine-grained TCP flow measurement method in this embodiment splitting observed data according to a flow membership;
fig. 5 is a schematic diagram of a network in which communication pairs for dividing the collection task are located in the present embodiment;
fig. 6 is a directed acyclic graph for explaining the processing of dividing the collection task in the present embodiment when the communication pair b-h requests;
fig. 7 is a directed acyclic graph illustrating a-g request processing by dividing the collection task in this embodiment.
Detailed Description
The present invention will now be described more fully hereinafter with reference to the accompanying drawings. Those skilled in the art will be able to implement the invention based on these teachings. Before the present invention is described in detail with reference to the accompanying drawings, it is to be noted that:
the technical solutions and features provided in the present invention in the respective sections including the following description may be combined with each other without conflict.
Furthermore, the embodiments of the invention described in the following description are generally only examples of a part of the invention, and not all embodiments. Therefore, all other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative effort shall fall within the protection scope of the present invention.
With respect to terms and units in the present invention. The term "comprises" and any variations thereof in the description and claims of this invention and the related sections are intended to cover non-exclusive inclusions.
The contents of the present invention have been explained above. Those skilled in the art will be able to practice the invention based on these descriptions. All other embodiments, which can be derived by a person skilled in the art from the above description without inventive step, shall fall within the scope of protection of the present invention.
The invention will be described more fully hereinafter with reference to the accompanying drawings. Those skilled in the art will be able to practice the invention based on these descriptions. Before the present invention is described in detail with reference to the accompanying drawings, it is to be noted that:
the technical solutions and features provided in the present invention in the respective sections including the following description may be combined with each other without conflict.
Moreover, the embodiments of the present invention described in the following description are generally only examples of a part of the present invention, and not all examples. Therefore, all other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative effort shall fall within the protection scope of the present invention.
With respect to terms and units in the present invention. The term "comprises" and any variations thereof in the description and claims of this invention and the related sections are intended to cover non-exclusive inclusions.
The contents of the present invention have been explained above. Those skilled in the art will be able to practice the invention based on these descriptions. All other embodiments, which can be derived by a person skilled in the art from the above description without inventive step, shall fall within the scope of protection of the present invention.
The embodiment adopts an event-driven fine-grained TCP flow measurement method, TCP message observation data are collected on network equipment, and the transmitted data volume of TCP flow in the network is analyzed and estimated by a remote logic integrated controller;
after receiving the specified type of TCP packet, the network device performs the following operations: generating message observation data for the message and caching the message observation data in the local network equipment;
if the position to be cached has the content, the message observation data content caching fails; directly sending the observation data content of the message to be cached and the content I of the corresponding position in the cache to a controller, and accommodating the cache position to be empty;
and if the position to be cached does not have the content, the message successfully observes the caching of the data content.
The data plane functionality of the present method can be implemented on existing programmable switches such as P4.
As in fig. 1, the steps are carried out:
s1, dividing the measurement and acquisition tasks of the network equipment;
s2, collecting the message observation data by the data forwarding plane of the network equipment;
s3, analyzing and estimating the transmitted data volume of the TCP stream on the remote controller;
in the fine-grained TCP flow measurement method of the embodiment, a network device message observation data acquisition task division method is also adopted, so that repeated measurement can be avoided, and the measurement task load of each network device can be balanced. It is contemplated herein that the controller knows the transmission path in the network from one subnet or host to another subnet or host, and the corresponding TCP flow number scenario (not necessarily very accurate).
We call such a set of communication requirements a communication pair.
As shown in fig. 2, the overall steps are:
s101, counting all network equipment capable of generating message observation data for the TCP stream;
s102, setting a group of communication requirements with the same sending node and receiving node as a communication pair;
s103, sequencing all communication pairs from large to small according to the number of the estimated TCP streams;
and S104, sequentially determining the message observation data acquisition tasks corresponding to the communication pairs according to the sequence obtained after sequencing, and distributing the message observation data acquisition tasks to different network devices.
A plurality of devices in the network have message observation data acquisition capacity, so that the controller can divide the acquisition task of the message observation data and distribute the task to different devices, and the processing burden of each device is reduced. At this time, only when a certain TCP is set to detect on one device, the device performs the above-mentioned packet detection and observation packet generation processing on the TCP.
Correspondingly, the specific division method of each message observation data acquisition task is as follows:
1) sequencing all communication pairs from large to small according to the number of TCP streams contained in the estimated value;
2) and sequentially determining the arrangement of the message observation data acquisition tasks corresponding to each communication pair according to the sequence obtained after sequencing.
For each communication pair, assuming src to dst (where s and d may be a host IP address or a subnet address prefix), the node pair forms a directed acyclic graph (possibly more than one because equal cost multipath routing may be used) on all the feasible routes in the network. For example, referring to FIG. 5, src is labeled a, dst is labeled g:
there are two communication pairs a-g and b-h in the network, where no measurement task is scheduled on each directed edge in the network.
Wherein the number of streams that communication pairs a through g contain together is 3. Of these, approximately 1.5 is transmitted via path a- > c- > e- > g, and 1.5 is transmitted via path a- > d- > f- > g (since the number of streams is an estimate, the fractional number can occur);
the number of streams that a communication pair b to h collectively contains is 5, where about 3 is transmitted via path b- > c- > e- > h, and further the 2 is transmitted via path b- > d- > f- > h.
src corresponds to node a in the graph and dst corresponds to node g in the graph.
The task allocation process is as follows:
and sequencing the communication pairs according to the number of the included streams, and processing more than the communication pairs and less than the communication pairs, so that the communication pairs are processed firstly and the task distribution of the b-h message observation data acquisition is processed.
When processing a b-h request, the corresponding directed acyclic graph is FIG. 6.
At this time, b-h in the directed acyclic graph has a plurality of minimum cuts, and any one of the minimum cuts can be used.
Suppose the minimal cut algorithm used exactly chooses edges b- > c and d- > f.
Then, the nodes c and f are respectively used as the nodes for collecting the message observation data of the communication pairs b-h, the weight of the directed edge b- > c is updated to 3, and the weight of the directed edge d- > f is updated to 2.
Then, processing a task of acquiring observation data of the communication pairs a-g, wherein a directed acyclic graph corresponding to the task is shown in FIG. 7;
in this case, a-g in the directed acyclic graph has a plurality of minimum partitions, and any one of the minimum partitions may be used.
Assume that the minimal cut algorithm used exactly selects edges a- > c and a- > d.
Then, the nodes c and d are respectively used as the nodes for collecting the message observation data of the communication pairs a-g, the weight of the directed edge a- > c is updated to 1.5, and the weight of the directed edge a- > d is updated to 1.5.
The selection decision process of the message observation data acquisition node is as follows:
1) setting a weight for each directed edge in the directed acyclic graph, and taking a value to represent an estimated value of the number of TCP streams to be measured on a corresponding input port on a next hop node corresponding to the directed edge (namely a link);
2) finding a directional edge set with the minimum weight, and assuming the directional edge set as E, so that a to g can not be reached any more after deleting the directional edge in the set; if a plurality of directional edge sets exist at the same time, one directional edge set is randomly selected;
3) and E, taking the down-hop network equipment corresponding to the directed edge in the E as a message observation data acquisition node of the TCP flow passing through the link in the communication pair, and updating the load weight value of the measurement task corresponding to the directed edge.
And sequentially processing the requirements of all communication pairs, and finally selecting a message observation data acquisition node for each requirement.
Therefore, the task division method can achieve the purposes of avoiding repeated measurement and balancing the load of the data acquisition task observed by each network device.
As shown in fig. 3, for each received TCP packet, the data forwarding plane of the network device collects packet observation data, and the overall flow is as follows:
s201, when a network equipment data forwarding plane receives a TCP message;
s201a, if the current packet is any one of a SYN packet, a FIN packet, or a packet with SEQ ═ 0, generating observation data for the TCP packet, and sending the observation data to a control plane of the network device, and performing S202;
s201b, if the current message is not any of a SYN message, a FIN message, or a message with SEQ ═ 0, jumping to S206;
s202, caching message observation data submitted by a data forwarding plane by a control plane of the network equipment; the control surface receives the message observation data;
s204, the control surface performs hash calculation on the quintuple field in the message observation data to obtain an integer number f from 0 to N-1;
s205, when the message observation data is cached in the fth unit in the cache, sending the observation data cached in the unit and the current message observation data to a remote controller together, and emptying the cached content in the fth unit;
when the message observation data is not cached in the unit f in the cache, caching the current message observation data into the unit f;
and S206, ending the processing.
The specific process can be carried out as follows:
1) after receiving a message, judging whether the current message is a SYN message or a FIN message in a TCP message or a message with SEQ ═ 0, if so, performing 2), and if not, performing 3);
2) generating observation data for the TCP message, and sending the observation data to a control plane of the network equipment;
3) the process is ended.
The control plane of the network equipment caches the message observation data information submitted by the data plane by using a continuous memory, and the control plane has N storage units, and the content stored by each unit comprises:
1) header type (2 bit): indicating that there are 4 possible types of stored content, NUL (0), indicating that the content is not cached; SYN (1), which represents that the cached content is the observation data information of SYN message; SEQ ═ 0(2) represents the observation data information of the cached message SEQ ═ 0; and FIN (3) which represents that the cached content is observation data information of the FIN message;
2) message arrival time (32 bit);
3) TCP quintuple information (104 bit for IPv 4); namely, a source IP address (32bit), a destination IP address (32bit), a source port number (16bit), a destination port number (16bit), a protocol number (8 bit);
4) SEQ value (32bit)
The control plane corresponds to two processes:
after receiving message observation data sent by a data plane, the control plane:
1) performing hash calculation on the five-tuple field of the message to obtain an integer number between 0 and N-1 as an identity identifier of the flow in the network equipment, assuming that f is the integer number;
2) judging whether observation data are cached in the f unit in the cache, if so, skipping to 3), and if not, skipping to 4)
3) Sending the 'message type, arrival time, quintuple information and SEQ value' cached in the unit and the 'message type, arrival time, quintuple information and SEQ value' of the current message to a remote controller, and emptying the content cached in the unit; jump to 5);
4) caching the message type, arrival time, quintuple information and SEQ value of the current message into the f unit;
5) finishing the treatment;
and (II) periodically scanning all the cached message observation data once at intervals of T0, sending the message observation data to a remote controller and emptying the message observation data.
When the network equipment control surface sends data to the remote controller, the identity of the equipment is contained at the same time, so that the controller can identify and distinguish the data.
Compared with other message observation data acquisition methods, the method is a measurement method which supports fine granularity, has low overhead and is extensible, can accurately measure the transmission data volume of the TCP stream including the small stream, and can be realized on the existing hardware.
The message observation data received by the controller contains the following information: message type, arrival time, quintuple information, SEQ value, and serial number of the detection device.
Multiple communications may be made between the same sending host and receiving host of the same pair over a long period of time, and multiple TCP flows occurring at different times may use the same port number.
Accordingly, the controller needs to group the packet observation data having the same five tuple, so that the packet observation data in each group corresponds to one flow.
Since the clocks of different devices in the network cannot be identical, and some packets of a TCP flow may be captured by different devices due to a change in routing. To reduce the effect of time errors between time devices, all devices in the network need to be time synchronized. Here, it is assumed that the maximum difference between clocks between different devices is T1 and the maximum one-way delay of the network is T2.
In addition, TCP may trigger data retransmissions, and the controller may also need to relieve redundancy for message observation data of such retransmitted data.
In the event-driven fine-grained TCP flow measurement method, after data acquisition is completed, a TCP flow transmitted data volume estimation method can be adopted for a remote controller, and the method specifically comprises the following operations:
the remote controller classifies the obtained observation data, and the controller firstly splits the observation data according to the membership of the flow, and referring to fig. 4, the processing flow is as follows:
s301, grouping all the received observation data according to the five-tuple values of the observation data by the controller;
s302, regarding each group of message observation data, if a plurality of message observation data have the same quintuple information and the capturing time is less than T3 (the value can be configured), the message observation data are regarded as message observation data of retransmission data, or only one message observation data is reserved for the message observation data repeatedly captured by different equipment on the same data;
and S303, selecting the message observation data with the type of SYN from all the message observation data after redundancy removal, and sequencing according to the sequence of the capture time of the message observation data. Assuming that a total of m SYN message observation data, the capture time of the m SYN message observation data after sequencing is t [1], t [2],. once, t [ i ],. once, t [ m ];
s304, the controller divides TCP message observation data corresponding to the observation data of all messages in a time interval [ T [ i ] -T1-T2, T [ i +1] -T1-T2) into a single stream, wherein i is 1, 2. For all message observations after T [ m ] -T1-T2, it is considered to belong to the last flow.
S305, the controller estimates the transmitted data quantity of the TCP flow according to the message observation data belonging to each flow.
Let SEQ in the SYN message observation data be SEQ1, SEQ value of the FIN message observation data be SEQ2, and the number of occurrences of the message observation data of SEQ ═ 0 be k. The calculation is then as follows:
1) if the flow has FIN message observation data, the size of the flow is SEQ2-SEQ1+ k 2^ 32;
2) if the flow has no FIN message observation data and k is greater than 0, the flow size is k x 2^32-SEQ1, and the flow is not accurately estimated;
3) if the flow has no FIN packet observation data and k is 0, no estimation is performed.
The contents of the present invention have been explained above. Those skilled in the art will be able to implement the invention based on these teachings. Based on the above description of the present invention, all other embodiments obtained by a person of ordinary skill in the art without any creative effort shall fall within the protection scope of the present invention.

Claims (5)

1. The method for dividing the message observation data acquisition task of the network equipment is characterized by comprising the following operations:
counting all network equipment which can generate message observation data for the TCP stream;
setting a group of communication requirements with the same sending node and receiving node as a communication pair;
sequencing all communication pairs from large to small according to the number of the TCP streams contained in estimation;
according to the sequence obtained after sequencing, determining the message observation data acquisition tasks corresponding to the communication pairs in sequence, and distributing the message observation data acquisition tasks to different network equipment;
constructing a directed acyclic graph by using all paths composed of communication pairs which simultaneously have a plurality of transmission paths and pass through each transmission path and have known TCP stream number, wherein one directed edge of the directed acyclic graph corresponds to one unidirectional link on the transmission path; measuring the message observation data acquisition tasks of the communication pair transmission paths on the links to obtain measurement values, and sequentially selecting specific message observation data acquisition nodes for TCP streams on each transmission path of each communication pair according to the measurement values on each link;
for a group of communication pairs with a sending node being s and a receiving node being d, selecting a specific message observation data acquisition node comprises the following steps:
1) setting a weight for each edge in the directed acyclic graph, and taking a value to represent an estimated value of the number of TCP streams required to be processed by a message observation data acquisition task arranged on the link;
2) using a minimum cut algorithm to calculate a minimum edge cut set from s to d in the directed acyclic graph, so that the s to d can not be reached any more after the directed edges in the set are deleted; if a plurality of minimum edge cut sets exist, one minimum edge cut set is randomly selected;
3) and for the directed edge u-v from the node u to the node v in the selected directed edge set E, setting the node v as a message observation data acquisition node of a flow contained in a communication pair from s to d, accumulating the TCP flow number passing through a link u-v in the communication pair to a weight value corresponding to the directed edge u-v in the network, and updating.
2. The method for partitioning data collection tasks for observing messages by network equipment according to claim 1, wherein the communication pair is a transmission path from one subnet or host to another subnet or host.
3. The method as claimed in claim 2, wherein the communication pair further includes information about the number of TCP flows from the one sub-network or host to another sub-network or host.
4. The method according to claim 1, wherein the network device performs packet inspection processing on a TCP stream and generates packet observation data as needed only when a TCP stream is set to be inspected on one of the network devices.
5. The method for partitioning network device message observation data collection tasks according to claim 1, wherein the directed edge set E is a directed edge set in which a sum of directed edge weights in the selected edge cut sets is the smallest.
CN201910816830.4A 2019-08-30 2019-08-30 Network equipment message observation data acquisition task dividing method Active CN110572332B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910816830.4A CN110572332B (en) 2019-08-30 2019-08-30 Network equipment message observation data acquisition task dividing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910816830.4A CN110572332B (en) 2019-08-30 2019-08-30 Network equipment message observation data acquisition task dividing method

Publications (2)

Publication Number Publication Date
CN110572332A CN110572332A (en) 2019-12-13
CN110572332B true CN110572332B (en) 2022-08-26

Family

ID=68777104

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910816830.4A Active CN110572332B (en) 2019-08-30 2019-08-30 Network equipment message observation data acquisition task dividing method

Country Status (1)

Country Link
CN (1) CN110572332B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111740857B (en) * 2020-05-29 2023-12-26 新华三信息安全技术有限公司 Method and device for issuing Network Quality Analysis (NQA) configuration

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101834763A (en) * 2010-06-25 2010-09-15 山东大学 Multiple-category large-flow parallel measuring method under high speed network environment
CN102437959A (en) * 2011-12-20 2012-05-02 东南大学 Stream forming method based on dual overtime network message
CN103716208A (en) * 2013-12-31 2014-04-09 北京邮电大学 Network management method and system supporting elephant flow, interchanger and network
CN107342906A (en) * 2016-04-29 2017-11-10 华为技术有限公司 A kind of detection method, equipment and the system of elephant stream

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107370676A (en) * 2017-08-03 2017-11-21 中山大学 Fusion QoS and load balancing demand a kind of route selection method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101834763A (en) * 2010-06-25 2010-09-15 山东大学 Multiple-category large-flow parallel measuring method under high speed network environment
CN102437959A (en) * 2011-12-20 2012-05-02 东南大学 Stream forming method based on dual overtime network message
CN103716208A (en) * 2013-12-31 2014-04-09 北京邮电大学 Network management method and system supporting elephant flow, interchanger and network
CN107342906A (en) * 2016-04-29 2017-11-10 华为技术有限公司 A kind of detection method, equipment and the system of elephant stream

Also Published As

Publication number Publication date
CN110572332A (en) 2019-12-13

Similar Documents

Publication Publication Date Title
CN108028778B (en) Method, system and apparatus for generating information transmission performance warning
US8811395B2 (en) System and method for determination of routing information in a network
EP3496346B1 (en) Data message forwarding method and apparatus
US20110063979A1 (en) Network traffic management
KR20060044844A (en) High-speed traffic measurement and analysis methodologies and protocols
EP3357196B1 (en) Analysis of network performance
CN103959713A (en) Measurement on a data flow in a communication network
CN110557302B (en) Network equipment message observation data acquisition method
CN110572332B (en) Network equipment message observation data acquisition task dividing method
US8169932B2 (en) QoS degradation point estimation method, QoS degradation point estimation device, and program
EP1986378B1 (en) Quality-degraded portion estimating device, quality-degraded portion estimating method, and quality-degraded portion estimating program
CN110572300B (en) Method for estimating transmitted data volume of TCP stream
CN110545217B (en) Event-driven fine-grained TCP flow measurement method
CN113132179A (en) Measuring packet residence and propagation times
Elattar et al. Reliable multipath communication approach for internet-based cyber-physical systems
JP4222567B2 (en) Congestion control method and congestion control apparatus
CN112910795B (en) Edge load balancing method and system based on many sources
CN113767597B (en) Network device, system and method for cycle-based load balancing
CN113812119B (en) Network node for performance measurement
CN114157595A (en) Communication system, data processing method and related equipment
CN107592269B (en) Method and network node for transmitting load information of path
Ervasti A survey on network measurement: Concepts, techniques, and tools
GB2338147A (en) Compution of traffic flow by scaling sample packet data
CN117376261A (en) Link and equipment state weighted load balancing method based on P4
CN117978739A (en) Message sending method and device, storage medium and electronic device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20230919

Address after: 518000 1104, Building A, Zhiyun Industrial Park, No. 13, Huaxing Road, Henglang Community, Longhua District, Shenzhen, Guangdong Province

Patentee after: Shenzhen Hongyue Information Technology Co.,Ltd.

Address before: 610031 No. 111, North Section of Second Ring Road, Chengdu City, Sichuan Province

Patentee before: SOUTHWEST JIAOTONG University

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20240208

Address after: A808, 40th Avenue, No. 87-1 Qingdao Middle Road, Huancui District, Weihai City, Shandong Province, 264200

Patentee after: Shandong Yuxin Intelligent Technology Co.,Ltd.

Country or region after: China

Address before: 518000 1104, Building A, Zhiyun Industrial Park, No. 13, Huaxing Road, Henglang Community, Longhua District, Shenzhen, Guangdong Province

Patentee before: Shenzhen Hongyue Information Technology Co.,Ltd.

Country or region before: China

TR01 Transfer of patent right