CN107196826A - A kind of network flow programming method algorithm based on sampling - Google Patents
A kind of network flow programming method algorithm based on sampling Download PDFInfo
- Publication number
- CN107196826A CN107196826A CN201710564141.XA CN201710564141A CN107196826A CN 107196826 A CN107196826 A CN 107196826A CN 201710564141 A CN201710564141 A CN 201710564141A CN 107196826 A CN107196826 A CN 107196826A
- Authority
- CN
- China
- Prior art keywords
- sampling
- new stream
- stream
- network flow
- bloom filter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/022—Capturing of monitoring data by sampling
- H04L43/024—Capturing of monitoring data by sampling by adaptive sampling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a kind of network flow programming method algorithm based on sampling, comprise the following steps：(1) parameter to Counting Bloom Filter structures carries out reasonable disposition；(2) whether network flow described in the packet arrived using Counting Bloom Filter structural determinations is new stream；(3) if the packet arrived is judged as new stream, the new stream is inserted into Counting Bloom Filter, flow counter adds 1, according to the stream number calculation error probability P inserted, while adjustment sampling frequency isF is the sampling frequency of whole algorithm, and random sampling is carried out to new stream with the frequency after adjustment, and continues with next packet, and repeat step (2) is continued cycling through；(4) if it is determined that being reached without new stream, then next packet is continued with, repeat step (2) is continued cycling through.The present invention realizes the sampling with equal probability to network flow, accomplishes that network flow data can be reduced, and the characteristic information of network flow data can be retained again, saves the storage resource of system.
Description
Technical field
The present invention relates to network flow programming method field of engineering technology, especially a kind of network flow programming method based on sampling is calculated
Method.
Background technology
Due to the continuous improvement of network link rates and sharply increasing for network data flow, each message information or stream are captured
Information is stored and analyzed become impossible, and the introducing of sampling techniques effectively solves the bottleneck problem, based on sampling
Network flow programming method algorithm become one of the primary study content in network flow programming method field in recent years.
Although sampling techniques improves the validity and feasibility of flow measurement to a certain extent, due to express network stream
Amount is big, the increasingly reinforcement of complexity and isomerism, and the deficiency of network processes and storage resource, the basic methods of sampling has been difficult to full
The need for foot reality, the sampling measurement based on stream, the measurement of sampling and Hash combination have then been derived and based on Bloom
The flow measurement algorithm such as Filter sampling measurement.
The basic methods of sampling is widely present to the not enough shortcoming of rill sampling of data, causes a large amount of flow distribution information to be lost
Lose, it is impossible to the true distribution situation of network flow is effectively obtained, therefore, it is necessary to be taken out in more fair mode to network flow
Sample, the present invention devises the network flow sampling with equal probability algorithm based on Counting Bloom Filter structures.
The content of the invention
The technical problems to be solved by the invention are there is provided a kind of network flow programming method algorithm based on sampling, can
The sampling with equal probability to network flow is realized, accomplishes that network flow data can be reduced, network flow data can be retained again
Characteristic information, save system storage resource.
In order to solve the above technical problems, the present invention provides a kind of network flow programming method algorithm based on sampling, including it is as follows
Step：
(1) parameter to Counting Bloom Filter structures carries out reasonable disposition；
(2) whether the packet belonging network stream arrived using Counting Bloom Filter structural determinations is new
Stream；
(3) if the packet arrived is judged as new stream, the new stream is inserted into Counting Bloom Filter
In, flow counter adds 1, according to the stream number calculation error probability P inserted, while adjustment sampling frequency isF is whole
The sampling frequency of algorithm, carries out random sampling, and continue with next packet, repeat step with the frequency after adjustment to new stream
(2) continue cycling through；
(4) if it is determined that being reached without new stream, then next packet is continued with, repeat step (2) is continued cycling through.
It is preferred that, in step (1), if each Counter in Counting Bloom Filter structures distributes 4,
When Counter values reach 16, the probability that just overflows is：F≤1.37×10-15× m, m are Counter numbers, are each
Counter distributes 4；As hash function number k=(ln2) (m/n), error rate is minimum, wherein, n is Counting
The element number of the represented set of Bloom Filter, in the case where error rate is not more than E, to ensure in Counter arrays
At least half is that 0, m should meet condition：m≥n·log2(1/E)·log2E, is nlog21.44 times of (1/E).
It is preferred that, in step (2), when one be grouped into up to when, its traffic identifier is parsed first, the k of its traffic identifier is calculated
Hash function value, if corresponding k Counter values are all higher than or equal to 1 in Counting Bloom Filter structures, sentences
It is set to no new stream to reach, is otherwise determined as there is new stream arrival.
It is preferred that, in step (3), if the packet arrived is judged as new stream, first the stream is inserted into
In Counting Bloom Filter, corresponding k Counter values Jia 1 respectively, and flow counter adds 1, according to the stream inserted
Number calculation error probability P；The probability of error is Counting Bloom Filter False Rate：N is the network flow number inserted in formula, then adjusts sampling frequency and isProtect
Demonstrate,prove and f is equal to the sampling frequency of any new stream, random sampling is carried out to new stream with the frequency after adjustment, complete to continue after sampling
The next packet of processing, repeats second step and continues cycling through.
Beneficial effects of the present invention are：The present invention realizes the sampling with equal probability to network flow, and accomplishing can be to network traffics
Data are reduced, and the characteristic information of network flow data can be retained again, save the storage resource of system.
Brief description of the drawings
Fig. 1 is circuit theory schematic diagram of the invention.
Fig. 2 is algorithm flow schematic diagram of the invention.
Embodiment
As shown in figure 1, a kind of network flow programming method algorithm based on sampling, comprises the following steps：
(1) parameter to Counting Bloom Filter structures carries out reasonable disposition；
(2) whether the packet belonging network stream arrived using Counting Bloom Filter structural determinations is new
Stream；
(3) if the packet arrived is judged as new stream, the new stream is inserted into Counting Bloom Filter
In, flow counter adds 1, according to the stream number calculation error probability P inserted, while adjustment sampling frequency isF is whole
The sampling frequency of algorithm, carries out random sampling, and continue with next packet, repeat step with the frequency after adjustment to new stream
(2) continue cycling through；
(4) if it is determined that being reached without new stream, then next packet is continued with, repeat step (2) is continued cycling through.
In step (1), if each Counter in Counting Bloom Filter structures distributes 4, when
Counter values reach the probability just overflowed when 16：F≤1.37×10-15× m, m are Counter numbers, are each Counter
Distribution 4；As hash function number k=(ln2) (m/n), error rate is minimum, wherein, n is Counting Bloom
The element number of the represented set of Filter, in the case where error rate is not more than E, to ensure at least one in Counter arrays
Half should meet condition for 0, m：m≥n·log2(1/E)·log2E, is nlog21.44 times of (1/E).
In step (2), when one be grouped into up to when, its traffic identifier is parsed first, k hash function of its traffic identifier is calculated
Value, if corresponding k Counter values are all higher than or equal to 1 in Counting Bloom Filter structures, is determined as not having
New stream is reached, and is otherwise determined as there is new stream arrival.
In step (3), if the packet arrived is judged as new stream, the stream is first inserted into Counting Bloom
In Filter, corresponding k Counter values Jia 1 respectively, and flow counter adds 1, according to the stream number calculation error probability inserted
P；The probability of error is Counting Bloom Filter False Rate：N is in formula
The network flow number inserted, then adjusting sampling frequency isEnsure to be equal to f to the sampling frequency of any new stream, with
Frequency after adjustment carries out random sampling to new stream, completes to continue with next packet after sampling, repeats second step and continues cycling through.
As shown in figure 1, a kind of network flow programming method algorithm based on sampling that the present invention is provided, algorithm is by Counting
Bloom Filter modules, error absorption module and stochastic sampling module composition.Counting Bloom Filter modules are used for
Determine whether there is new stream arrival；Error absorption module then is used to record the stream quantity currently reached, calculates in sampling process and produces
The probability of error, while adjust sampling frequency；Stochastic sampling module is then with the probability after adjustment to Counting Bloom
The new stream that Filter assert is sampled.
As shown in Fig. 2 a kind of network flow programming method algorithm based on sampling, comprises the following steps：
Step 1, it is that each Counter in Counting Bloom Filter structures distributes 4, of hash function
The size m that number k is taken as 10, Counter arrays is set to total 20 times of actual stream.
Step 2, using Counting Bloom Filter structural determinations arrive packet belonging network stream whether be
New stream：When one be grouped into up to when, its traffic identifier is parsed first, k hash function value of its traffic identifier is calculated, if Counting
Corresponding k Counter values are all higher than or equal to 1 in Bloom Filter structures, then are determined as that no new stream is reached, otherwise sentence
Being set to has new stream arrival.
Step 3, if the packet arrived is judged as new stream, the stream is first inserted into Counting Bloom
In Filter, flow counter adds 1, according to the stream number calculation error probability P inserted, while adjustment sampling frequency isf
For the sampling frequency of whole algorithm, random sampling is carried out to new stream with the frequency after adjustment, and continues with next packet, is repeated
Step 2 is continued cycling through：If the packet arrived is judged as new stream, the stream is first inserted into Counting Bloom
In Filter, corresponding k Counter values Jia 1 respectively, and flow counter adds 1, according to the stream number calculation error probability inserted
P.Here the probability of error is Counting Bloom Filter False Rate：Formula
Middle n is the network flow number inserted.Then adjustment sampling frequency isEnsure to be equal to the sampling frequency of any new stream
F, random sampling is carried out with the frequency after adjustment to new stream, continues with next packet after completing sampling, repeat step 2 continues to follow
Ring.
Step 4, if it is decided that reached for no new stream, then continue with next packet, repeat step 2 is continued cycling through.
Although the present invention is illustrated and described with regard to preferred embodiment, it is understood by those skilled in the art that
Without departing from scope defined by the claims of the present invention, variations and modifications can be carried out to the present invention.
Claims (4)
1. a kind of network flow programming method algorithm based on sampling, it is characterised in that comprise the following steps：
(1) parameter to Counting Bloom Filter structures carries out reasonable disposition；
(2) whether the packet belonging network stream arrived using Counting Bloom Filter structural determinations is new stream；
(3) if the packet arrived is judged as new stream, the new stream is inserted into Counting Bloom Filter,
Flow counter adds 1, according to the stream number calculation error probability P inserted, while adjustment sampling frequency isF is whole calculation
The sampling frequency of method, carries out random sampling, and continue with next packet, repeat step (2) with the frequency after adjustment to new stream
Continue cycling through；
(4) if it is determined that being reached without new stream, then next packet is continued with, repeat step (2) is continued cycling through.
2. the network flow programming method algorithm as claimed in claim 1 based on sampling, it is characterised in that in step (1), if
Each Counter in Counting Bloom Filter structures distributes 4, then is just overflowed when Counter values reach 16
Probability is：F≤1.37×10-15× m, m are Counter numbers, are that each Counter distributes 4；As hash function number k=
(ln2) error rate is minimum when (m/n), wherein, n is the element number of the represented set of Counting Bloom Filter,
In the case that error rate is not more than E, to ensure that at least half is that 0, m should meet condition in Counter arrays：m≥n·log2
(1/E)·log2E, is nlog21.44 times of (1/E).
3. the network flow programming method algorithm as claimed in claim 1 based on sampling, it is characterised in that in step (2), when one
Be grouped into up to when, its traffic identifier is parsed first, k hash function value of its traffic identifier is calculated, if Counting Bloom
Corresponding k Counter values are all higher than or equal to 1 in Filter structures, then are determined as that no new stream is reached, are otherwise determined as having
New stream is reached.
4. the network flow programming method algorithm as claimed in claim 1 based on sampling, it is characterised in that in step (3), if arrived
The packet come is judged as new stream, and first the stream is inserted into Counting Bloom Filter, corresponding k
Counter values Jia 1 respectively, and flow counter adds 1, according to the stream number calculation error probability P inserted；The probability of error is
Counting Bloom Filter False Rate：N is the network flow inserted in formula
Number, then adjusting sampling frequency isEnsure to be equal to f to the sampling frequency of any new stream, with the frequency pair after adjustment
New stream carries out random sampling, completes to continue with next packet after sampling, repeats second step and continues cycling through.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710564141.XA CN107196826A (en) | 2017-07-12 | 2017-07-12 | A kind of network flow programming method algorithm based on sampling |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710564141.XA CN107196826A (en) | 2017-07-12 | 2017-07-12 | A kind of network flow programming method algorithm based on sampling |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107196826A true CN107196826A (en) | 2017-09-22 |
Family
ID=59883662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710564141.XA Pending CN107196826A (en) | 2017-07-12 | 2017-07-12 | A kind of network flow programming method algorithm based on sampling |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107196826A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107948007A (en) * | 2017-10-10 | 2018-04-20 | 东南大学 | Long stream recognition method based on sampling and two-stage CBF |
| CN112800250A (en) * | 2021-02-05 | 2021-05-14 | 联想(北京)有限公司 | Multimedia data stream processing method and device and electronic equipment |
| CN114826955A (en) * | 2022-05-26 | 2022-07-29 | 电子科技大学 | Dynamic grouping sampling method for traffic flow in IPv6 network |
| CN115051940A (en) * | 2022-05-26 | 2022-09-13 | 电子科技大学 | IPv6 network flow measuring method based on bloom filter |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1901545A (en) * | 2006-07-17 | 2007-01-24 | 北京邮电大学 | Stream sampling device and method for detecting high speed network super connection host |
| US8131841B2 (en) * | 2007-07-27 | 2012-03-06 | Hewlett-Packard Development Company, L.P. | Method and apparatus for detecting predefined signatures in packet payload |
-
2017
- 2017-07-12 CN CN201710564141.XA patent/CN107196826A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1901545A (en) * | 2006-07-17 | 2007-01-24 | 北京邮电大学 | Stream sampling device and method for detecting high speed network super connection host |
| US8131841B2 (en) * | 2007-07-27 | 2012-03-06 | Hewlett-Packard Development Company, L.P. | Method and apparatus for detecting predefined signatures in packet payload |
Non-Patent Citations (3)
| Title |
|---|
| 严华云、关佶红，: "Bloom Filter 研究进", 《电信科学》 * |
| 王洪波等: "高速网络超连接主机检测中的流抽样算法研究", 《电子学报》 * |
| 胡广昌: "基于Bloom Filters流抽样算法的研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107948007A (en) * | 2017-10-10 | 2018-04-20 | 东南大学 | Long stream recognition method based on sampling and two-stage CBF |
| CN107948007B (en) * | 2017-10-10 | 2021-09-10 | 东南大学 | Long flow identification method based on sampling and two-stage CBF |
| CN112800250A (en) * | 2021-02-05 | 2021-05-14 | 联想(北京)有限公司 | Multimedia data stream processing method and device and electronic equipment |
| CN114826955A (en) * | 2022-05-26 | 2022-07-29 | 电子科技大学 | Dynamic grouping sampling method for traffic flow in IPv6 network |
| CN115051940A (en) * | 2022-05-26 | 2022-09-13 | 电子科技大学 | IPv6 network flow measuring method based on bloom filter |
| CN114826955B (en) * | 2022-05-26 | 2023-03-21 | 电子科技大学 | Dynamic grouping sampling method for service flow in IPv6 network |
| CN115051940B (en) * | 2022-05-26 | 2023-05-30 | 电子科技大学 | IPv6 network flow measurement method based on bloom filter |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107196826A (en) | A kind of network flow programming method algorithm based on sampling | |
| CN1677940B (en) | High-speed traffic measurement and analysis methodologies and protocols | |
| US20110261710A1 (en) | Analysis apparatus and method for abnormal network traffic | |
| CN110519290A (en) | Anomalous traffic detection method, device and electronic equipment | |
| CN107508719A (en) | Measure the method, apparatus and network node of network delay | |
| CN105357137B (en) | Message filtering method and the FPGA being applicable in, intelligent substation | |
| CN106559407A (en) | A kind of Network traffic anomaly monitor system based on SDN | |
| CN109558727B (en) | Routing security detection method and system | |
| CN110049061A (en) | Lightweight ddos attack detection device and detection method on high speed network | |
| CN106326461A (en) | Real time processing guarantee method and system based on network signaling record | |
| CN100352208C (en) | Detection and defence method for data flous of large network station | |
| CN111181930A (en) | DDoS attack detection method, device, computer equipment and storage medium | |
| CN108809752B (en) | Adaptive monitoring method and device for network traffic, NPB (network provider node B) equipment and medium | |
| CN107920024A (en) | A kind of SDN controllers dynamic stream table management method | |
| Li et al. | Ladderfilter: Filtering infrequent items with small memory and time overhead | |
| CN111080362A (en) | Advertisement monitoring system and method | |
| Tang et al. | Intelligent awareness of delay-sensitive internet traffic in digital twin network | |
| CN112367322B (en) | Power station industrial control system abnormal flow identification method based on bubbling sequencing method | |
| CN105307212B (en) | A kind of processing method of base station data on flows and base station | |
| CN106911807A (en) | A kind of vehicle-mounted CAN FD network design space search methods that safety is perceived | |
| CN114531401A (en) | Token bucket flow limiting method based on priority | |
| CN103595632A (en) | Rate-adjustable user-defined multi-message sending system and realizing method thereof | |
| CN114363178B (en) | Block chain network optimization method, device and system | |
| JP5508310B2 (en) | Loop detection apparatus and loop detection method | |
| CN117081853B (en) | New energy edge side communication protocol conversion module with time wheel dynamic authentication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170922 |
|
| RJ01 | Rejection of invention patent application after publication |