CN105516012A - Load balancing method and system for extra large network traffic processing - Google Patents
Load balancing method and system for extra large network traffic processing Download PDFInfo
- Publication number
- CN105516012A CN105516012A CN201410773879.3A CN201410773879A CN105516012A CN 105516012 A CN105516012 A CN 105516012A CN 201410773879 A CN201410773879 A CN 201410773879A CN 105516012 A CN105516012 A CN 105516012A
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- balance policy
- common task
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a load balancing method and system for extra large network traffic processing. The method comprises the steps of initializing an IP address classifier and a network protocol classifier, creating a packet processing queue, classifying captured packets through the classifiers, assigning the packets meeting the classifier classification rules to a specific task queue for processing, distributing other packets through a selected balancing strategy to a common task queue for processing, meanwhile, evaluating the balancing effects of the common task queue periodically, and reselecting a balancing strategy suitable for a current network if the balancing effect is not good. The method can be used for solving the technical problems of poor data processing capability of a network device, too centralized processing of task queues and poor multithread processing.
Description
Technical field
The present invention relates to computer network security technology field, particularly relate to the load-balancing method to the process of super large network traffics and system.
Background technology
Along with disparate networks safety means are in the extensive use in anti-virus field, the Internet, its network environment tackled is also complicated all the more, this makes the requirement also corresponding raising of the disposal ability of disparate networks safety means when network data, network traffics at a high speed determine Network Security Device to be needed to adopt multithreading operation to make load balancing, but the load balancing of correspondence also has problems, such as: 1, the balance policy of some Network Security Devices is more single, cause the process of some task queue too concentrated, multiple threads poor effect; 2, increase after balance policy, the position disposed due to Network Security Device varies, and cannot allow load-balancing method self adaptation, even needs to revise source code after replacing applied environment; 3, balance policy select improper, cause each queue need when usage data use thread lock protect.These problems above-mentioned all can cause Network Security Device concurrent tasks treatment effeciency lower.
Summary of the invention
For above-mentioned technical problem, the invention provides the load-balancing method to the process of super large network traffics and system, the method comprises IP address sort device and procotol grader, by grader the message satisfied condition delivered in particular task queue and process, avoid the appearance of thread lock between queue, other messages are distributed to common task queue by the balance policy selected and process, realize load balancing, periodical evaluation portfolio effect simultaneously, when portfolio effect is not good, reselect the balance policy of applicable current network.
To the load-balancing method of super large network traffics process, comprising:
Initialization IP address sort device and procotol grader, IP address sort device is for identifying the message with particular ip address section, and procotol grader is for identifying the message with specific network protocols transmission;
Create Message processing queue, comprise particular task queue and common task queue;
Catch message;
Utilize IP address sort device and procotol grader classification message, judge whether message meets classifying rules, if the particular task queue then message being distributed to corresponding classification processes, otherwise message is distributed to common task queue according to balance policy processes;
The queue of monitoring common task, the portfolio effect of periodical evaluation current equalization strategy, if portfolio effect is not good, reselects the balance policy of applicable current network.
Further, the mode of catching message described in is that pcap, zero-copy or specialized hardware catch bag card.
Further, balance policy comprises (1) four-tuple balance policy: strategy is based on source/destination port and source/destination IP address, calculated by hash algorithm, obtain corresponding hash value, mould is asked again with common task number of queues, obtain corresponding queue sequence number, this is the balance policy of system default; (2) IP bis-tuple balance policy: strategy, based on source/destination IP address, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number; (3) port two tuple balance policy: strategy, based on source/destination port, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number.
Further, before message is distributed to common task queue, use often kind of balance policy to carry out simulation distribution respectively, the result of distribution is strategically numbered with queue number message stored count, assess the portfolio effect of often kind of balance policy respectively, choose the balance policy that portfolio effect is best.
To the SiteServer LBS of super large network traffics process, comprising:
Grader initialization module, for initialization IP address sort device and procotol grader, IP address sort device is for identifying the message with particular ip address section, and procotol grader is for identifying the message with specific network protocols transmission;
Creating Queue module, for creating Message processing queue, comprising particular task queue and common task queue;
Catch message module, for catching message;
Message classification module, for utilizing IP address sort device and procotol grader classification message, judge whether message meets classifying rules, if the particular task queue then message being distributed to corresponding classification processes, otherwise message is distributed to common task queue according to balance policy processes;
Monitoring Queue module, for monitoring common task queue, the portfolio effect of periodical evaluation current equalization strategy, if portfolio effect is not good, reselects the balance policy of applicable current network.
Further, the acquisition mode of catching message module described in is that pcap, zero-copy or specialized hardware catch bag card.
Further, the balance policy of monitoring Queue module comprises (1) four-tuple balance policy: strategy is based on source/destination port and source/destination IP address, calculated by hash algorithm, obtain corresponding hash value, mould is asked again with common task number of queues, obtain corresponding queue sequence number, this is the balance policy of system default; (2) IP bis-tuple balance policy: strategy, based on source/destination IP address, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number; (3) port two tuple balance policy: strategy, based on source/destination port, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number.
Further, before message is distributed to common task queue, use often kind of balance policy to carry out simulation distribution respectively, the result of distribution is strategically numbered with queue number message stored count, assess the portfolio effect of often kind of balance policy respectively, choose the balance policy that portfolio effect is best.
The present invention relates to the method to super large network flow data equilibrium treatment, embedded IP address sort device and procotol grader in this method, by reaching condition in early stage to the initialization of grader with the queue of establishment Message processing, grader by the message classification of catching to particular task queue and common task queue processing, and the message of common task queue is distributed according to balance policy, overcome task queue process in super large network traffics too concentrated, the problem of multiple threads poor effect, and then improve network data flow multiple threads efficiency.
Accompanying drawing explanation
In order to be illustrated more clearly in technical scheme of the present invention, be briefly described to the accompanying drawing used required in embodiment below, apparently, the accompanying drawing that the following describes is only some embodiments recorded in the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the load-balancing method embodiment flow chart to the process of super large network traffics provided by the invention;
Fig. 2 is the SiteServer LBS example structure figure to the process of super large network traffics provided by the invention.
Embodiment
The present invention gives the load-balancing method to the process of super large network traffics and system, technical scheme in the embodiment of the present invention is understood better in order to make those skilled in the art person, and enable above-mentioned purpose of the present invention, feature and advantage become apparent more, below in conjunction with accompanying drawing, technical scheme in the present invention is described in further detail:
The present invention provide firstly the load-balancing method to the process of super large network traffics, as shown in Figure 1, comprising:
S101 initialization IP address sort device and procotol grader;
IP address sort device is for identifying the message with particular ip address section, and the IP of grader identification is any IP(source IP and object IP), procotol grader is for identifying the message with specific network protocols transmission;
S102 creates Message processing queue, comprises particular task queue and common task queue;
Task queue quantity is formulated according to configuration parameter, and particular task number of queues is consistent with the classification quantity of grader;
S103 catches message;
Described mode of catching message is that pcap, zero-copy or specialized hardware catch bag card;
S104 utilizes IP address sort device and procotol grader classification message;
S105 judges whether message meets classifying rules, if then perform S106, otherwise performs S107;
The particular task queue that message is distributed to corresponding classification by S106 processes, and terminates;
Message is distributed to common task queue according to balance policy and processes by S107;
Balance policy comprises (1) four-tuple balance policy: strategy is based on source/destination port and source/destination IP address, calculated by hash algorithm, obtain corresponding hash value, then ask mould with common task number of queues, obtain corresponding queue sequence number, this is the balance policy of system default; (2) IP bis-tuple balance policy: strategy, based on source/destination IP address, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number; (3) port two tuple balance policy: strategy, based on source/destination port, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number;
User can specify the balance policy of acquiescence voluntarily, also can use other balance policy;
S108 monitors common task queue, and the portfolio effect of periodical evaluation current equalization strategy, if portfolio effect is not good, reselects the balance policy of applicable current network;
When load differs by more than 20% between queue, then think that portfolio effect is not good;
Before message is distributed to common task queue, often kind of balance policy is used to carry out simulation distribution respectively, the result of distribution strategically to be numbered with queue number message stored count, assesses the portfolio effect of often kind of balance policy respectively, chooses the balance policy that portfolio effect is best;
In the operation phase, user can arrange timer, and system regularly will carry out portfolio effect assessment.
Present invention also offers the SiteServer LBS to the process of super large network traffics, as shown in Figure 2, comprising:
Grader initialization module 201, for initialization IP address sort device and procotol grader, IP address sort device is for identifying the message with particular ip address section, and procotol grader is for identifying the message with specific network protocols transmission;
Creating Queue module 202, for creating Message processing queue, comprising particular task queue and common task queue;
Catch message module 203, for catching message;
Message classification module 204, for utilizing IP address sort device and procotol grader classification message, judge whether message meets classifying rules, if the particular task queue then message being distributed to corresponding classification processes, otherwise message is distributed to common task queue according to balance policy processes;
Monitoring Queue module 205, for monitoring common task queue, the portfolio effect of periodical evaluation current equalization strategy, if portfolio effect is not good, reselects the balance policy of applicable current network.
In sum, the present invention relates to the load-balancing method of super large network traffics.This method is classified by built-in IP address sort device and procotol grader the message captured, the message meeting classifying rules is put in the particular task queue created and processes, other messages are then distributed in common task queue by balance policy and process, system can according to the portfolio effect of timer periodical evaluation common task queue, when portfolio effect is not good, reselect the balance policy of applicable current network.Traditional balance policy Measures compare is single, causes the process of some task queue too concentrated, multiple threads poor effect, and varying of disposing of Network Security Device, cannot allow the problems such as load-balancing method self adaptation.Instant invention overcomes these shortcomings, classified by grader and formulate balance policy, the operation of thread-data independence and cross-thread can be made not interfere with each other.The flow rate capacity of final raising Network Security Device also improves the adaptivity of load balance.
Above embodiment is unrestricted technical scheme of the present invention in order to explanation.Do not depart from any modification or partial replacement of spirit and scope of the invention, all should be encompassed in the middle of right of the present invention.
Claims (8)
1. the load-balancing method of pair super large network traffics process, is characterized in that:
Initialization IP address sort device and procotol grader, IP address sort device is for identifying the message with particular ip address section, and procotol grader is for identifying the message with specific network protocols transmission;
Create Message processing queue, comprise particular task queue and common task queue;
Catch message;
Utilize IP address sort device and procotol grader classification message, judge whether message meets classifying rules, if the particular task queue then message being distributed to corresponding classification processes, otherwise message is distributed to common task queue according to balance policy processes;
The queue of monitoring common task, the portfolio effect of periodical evaluation current equalization strategy, if portfolio effect is not good, reselects the balance policy of applicable current network.
2. the method for claim 1, is characterized in that, described in catch message mode be that pcap, zero-copy or specialized hardware catch bag card.
3. the method for claim 1, it is characterized in that, balance policy comprises (1) four-tuple balance policy: strategy is based on source/destination port and source/destination IP address, calculated by hash algorithm, obtain corresponding hash value, ask mould with common task number of queues again, obtain corresponding queue sequence number, this is the balance policy of system default; (2) IP bis-tuple balance policy: strategy, based on source/destination IP address, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number; (3) port two tuple balance policy: strategy, based on source/destination port, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number.
4. the method as described in claim 1 or 3, it is characterized in that, before message is distributed to common task queue, often kind of balance policy is used to carry out simulation distribution respectively, the result of distribution is strategically numbered with queue number message stored count, assess the portfolio effect of often kind of balance policy respectively, choose the balance policy that portfolio effect is best.
5. the SiteServer LBS of pair super large network traffics process, is characterized in that:
Grader initialization module, for initialization IP address sort device and procotol grader, IP address sort device is for identifying the message with particular ip address section, and procotol grader is for identifying the message with specific network protocols transmission;
Creating Queue module, for creating Message processing queue, comprising particular task queue and common task queue;
Catch message module, for catching message;
Message classification module, for utilizing IP address sort device and procotol grader classification message, judge whether message meets classifying rules, if the particular task queue then message being distributed to corresponding classification processes, otherwise message is distributed to common task queue according to balance policy processes;
Monitoring Queue module, for monitoring common task queue, the portfolio effect of periodical evaluation current equalization strategy, if portfolio effect is not good, reselects the balance policy of applicable current network.
6. system as claimed in claim 5, is characterized in that, described in catch message module acquisition mode be that pcap, zero-copy or specialized hardware catch bag card.
7. system as claimed in claim 5, it is characterized in that, the balance policy of monitoring Queue module comprises (1) four-tuple balance policy: strategy is based on source/destination port and source/destination IP address, calculated by hash algorithm, obtain corresponding hash value, ask mould with common task number of queues again, obtain corresponding queue sequence number, this is the balance policy of system default; (2) IP bis-tuple balance policy: strategy, based on source/destination IP address, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number; (3) port two tuple balance policy: strategy, based on source/destination port, is calculated by hash algorithm, obtains corresponding hash value, then ask mould with common task number of queues, obtains corresponding queue sequence number.
8. the system as described in claim 5 or 7, it is characterized in that, before message is distributed to common task queue, often kind of balance policy is used to carry out simulation distribution respectively, the result of distribution is strategically numbered with queue number message stored count, assess the portfolio effect of often kind of balance policy respectively, choose the balance policy that portfolio effect is best.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410773879.3A CN105516012B (en) | 2014-12-16 | 2014-12-16 | To the load-balancing method and system of the processing of super large network flow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410773879.3A CN105516012B (en) | 2014-12-16 | 2014-12-16 | To the load-balancing method and system of the processing of super large network flow |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105516012A true CN105516012A (en) | 2016-04-20 |
| CN105516012B CN105516012B (en) | 2019-07-26 |
Family
ID=55723663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410773879.3A Active CN105516012B (en) | 2014-12-16 | 2014-12-16 | To the load-balancing method and system of the processing of super large network flow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105516012B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109246023A (en) * | 2018-11-16 | 2019-01-18 | 锐捷网络股份有限公司 | Flow control methods, the network equipment and storage medium |
| CN110417675A (en) * | 2019-07-29 | 2019-11-05 | 广州竞远安全技术股份有限公司 | The network shunt method, apparatus and system of high-performance probe under a kind of SOC |
| CN110661719A (en) * | 2019-09-26 | 2020-01-07 | 杭州安恒信息技术股份有限公司 | Flow load balancing method and device |
| CN111897496A (en) * | 2020-07-28 | 2020-11-06 | 上海德拓信息技术股份有限公司 | Method for improving network IO read-write performance in distributed system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1728698A (en) * | 2004-07-30 | 2006-02-01 | 国家数字交换系统工程技术研究中心 | Parallel structured order preserved flow equilibrium system, and method for dispatching message based on sorted sream |
| CN1822567A (en) * | 2005-12-23 | 2006-08-23 | 清华大学 | Multi-domain net packet classifying method based on network flow |
| US20090003271A1 (en) * | 2007-06-28 | 2009-01-01 | Mohammad Riaz Khawer | Multi-link load balancing for reverse link backhaul transmission |
| CN101977162A (en) * | 2010-12-03 | 2011-02-16 | 电子科技大学 | Load balancing method of high-speed network |
-
2014
- 2014-12-16 CN CN201410773879.3A patent/CN105516012B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1728698A (en) * | 2004-07-30 | 2006-02-01 | 国家数字交换系统工程技术研究中心 | Parallel structured order preserved flow equilibrium system, and method for dispatching message based on sorted sream |
| CN1822567A (en) * | 2005-12-23 | 2006-08-23 | 清华大学 | Multi-domain net packet classifying method based on network flow |
| US20090003271A1 (en) * | 2007-06-28 | 2009-01-01 | Mohammad Riaz Khawer | Multi-link load balancing for reverse link backhaul transmission |
| CN101977162A (en) * | 2010-12-03 | 2011-02-16 | 电子科技大学 | Load balancing method of high-speed network |
Non-Patent Citations (1)
| Title |
|---|
| 孟宪福: ""基于优先级的任务调度与负载均衡模型研究"", 《万方数据库》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109246023A (en) * | 2018-11-16 | 2019-01-18 | 锐捷网络股份有限公司 | Flow control methods, the network equipment and storage medium |
| CN110417675A (en) * | 2019-07-29 | 2019-11-05 | 广州竞远安全技术股份有限公司 | The network shunt method, apparatus and system of high-performance probe under a kind of SOC |
| CN110417675B (en) * | 2019-07-29 | 2020-12-01 | 广州竞远安全技术股份有限公司 | Network shunting method, device and system of high-performance probe under SOC (System on chip) |
| CN110661719A (en) * | 2019-09-26 | 2020-01-07 | 杭州安恒信息技术股份有限公司 | Flow load balancing method and device |
| CN111897496A (en) * | 2020-07-28 | 2020-11-06 | 上海德拓信息技术股份有限公司 | Method for improving network IO read-write performance in distributed system |
| CN111897496B (en) * | 2020-07-28 | 2023-12-19 | 上海德拓信息技术股份有限公司 | Method for improving network IO read-write performance in distributed system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105516012B (en) | 2019-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9986563B2 (en) | Dynamic allocation of network bandwidth | |
| US9838308B2 (en) | Improving the architecture of middleboxes or service routers to better consolidate diverse functions | |
| US20210336885A1 (en) | Phantom queue link level load balancing system, method and device | |
| US10153979B2 (en) | Prioritization of network traffic in a distributed processing system | |
| CN104579962B (en) | A kind of method and device of qos policy that distinguishing different messages | |
| US20170195292A1 (en) | Sequentially serving network security devices using a software defined networking (sdn) switch | |
| Aujla et al. | Adaptflow: Adaptive flow forwarding scheme for software-defined industrial networks | |
| CN101106518B (en) | Service denial method for providing load protection of central processor | |
| CN105516012A (en) | Load balancing method and system for extra large network traffic processing | |
| US20180198717A1 (en) | A smart flow classification method/system for network and service function chaining | |
| US20180145904A1 (en) | System of hierarchical flow-processing tiers | |
| US20160119227A1 (en) | Distributed system and method for flow identification in an access network | |
| CN102132511A (en) | Virtual switch quality of service for virtual machines | |
| CN105763606B (en) | A kind of method and system of service chaining agent polymerization | |
| JP2016528630A (en) | Application-aware network management | |
| CN102158398A (en) | Method and device for forwarding messages | |
| US10530681B2 (en) | Implementing forwarding behavior based on communication activity between a controller and a network device | |
| CN106534394B (en) | Apparatus, system, and method for managing ports | |
| CN103023804A (en) | Method, device and network equipment for polymerization chain self-adaption flow load balance | |
| US20160344611A1 (en) | Method and control node for handling data packets | |
| CN106533946B (en) | Message forwarding method and device | |
| CN104618253A (en) | Dynamically changed transmission message processing method and device | |
| Abdollahi et al. | Flow-aware forwarding in SDN datacenters using a knapsack-PSO-based solution | |
| CN103685321B (en) | Packet forwards and safety protection detection, load-balancing method and device | |
| KR101191251B1 (en) | 10 Gbps scalable flow generation and control, using dynamic classification with 3-level aggregation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information |
Address after: 100080 Beijing city Haidian District minzhuang Road No. 3, Tsinghua Science Park Building 1 Yuquan Huigu a Applicant after: Beijing ahtech network Safe Technology Ltd Address before: 100080 Haidian District City, Zhongguancun, the main street, No. 1 Hailong building, room 1415, room 14 Applicant before: Beijing Antiy Electronic Installation Co., Ltd. |
|
| CB02 | Change of applicant information | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |