CN112565101A - Data packet distribution method - Google Patents
Data packet distribution method Download PDFInfo
- Publication number
- CN112565101A CN112565101A CN202011206311.5A CN202011206311A CN112565101A CN 112565101 A CN112565101 A CN 112565101A CN 202011206311 A CN202011206311 A CN 202011206311A CN 112565101 A CN112565101 A CN 112565101A
- Authority
- CN
- China
- Prior art keywords
- thread
- data packet
- packet
- analysis
- analysis thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/74—Address processing for routing
- H04L45/745—Address table lookup; Address filtering
- H04L45/7453—Address table lookup; Address filtering using hashing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data packet distribution method, wherein after a data packet analysis engine is started, data packets received by each packet capturing thread are actively and sequentially read, and when the number of the received data packets is lower than a certain number, namely the network flow is small, the active and sequential reading is changed into a passive waiting request mode, so that the received data packets can be ensured to be timely read, and the resource utilization is optimized. Firstly, data packets of the same session are distributed to the same analysis thread, and service continuity is kept; secondly, for the analysis thread in the overload state and the analysis thread receiving a larger conversation, the receiving of a new conversation is suspended, and the efficiency reduction caused by data backlog is reduced.
Description
Technical Field
The invention belongs to the technical field of network security, and particularly relates to a method for capturing a data packet and distributing the data packet to a corresponding analysis thread for analysis so as to improve the working efficiency of equipment.
Background
The role of computers and networks in enterprise and organizational services is becoming increasingly important. With the continuous increase of network traffic, the network security problem becomes more and more prominent, and the security detection range and depth of the network data packet need to be continuously strengthened. Various network security devices and systems need to analyze data packets, and under the conditions of high concurrency and big data, the security protection efficiency is affected by the bottleneck of data packet processing and analyzing performance.
The data packet capturing and distributing are all allocated among parallel threads, so the detection accuracy and efficiency are directly influenced by the quality of load balancing. Only the average distribution of the system load is static load distribution, and the current load condition of the system is considered in the distribution process, namely dynamic load distribution; the dynamic adjustment strategy can obviously improve the load balance. In the prior art, the data packet capturing and distributing allocation needs to be further optimized.
Disclosure of Invention
Based on the above background, the present invention is directed to provide a method for distributing data packets, which adjusts a scheduling mode of a packet capturing thread according to the number of data packets, optimizes a dynamic load balancing manner for an analysis thread, and improves data packet processing efficiency.
The specific technical scheme of the invention is that the data packet distribution method comprises the following steps: after the engine is started, data packets received by each packet capturing thread are sequentially and actively read; when the total number of the captured data packets in a specific time period is lower than a specified threshold value, stopping the sequential reading, and actively sending a reading request to an engine when the data packets received by a packet capturing thread reach a preset number; extracting quintuple information of the data packet as a session identifier, and adding the data packet into a queue to be distributed; and acquiring the data packet from the queue to be distributed and distributing the data packet to the analysis thread corresponding to the session identifier.
Preferably, the five-tuple Hash value of the data packet is used as the session identifier, and the data packet is distributed to a preset analysis thread corresponding to the Hash value.
Further, after the engine is started, the state of each packet capturing thread is inquired regularly, and when a new packet is inquired to exist in the packet capturing thread, reading operation is carried out; and after stopping reading in sequence, when the cache data packets of each packet capturing thread reach the preset number, sending a reading request to the engine, and starting the reading of the data packets cached by the packet capturing thread by the engine.
Preferably, the distributing the data packet to the analysis thread corresponding to the session identifier comprises presetting a distribution table in which a corresponding relation between a quintuple Hash value and the analysis thread is stored; searching whether a five-tuple Hash value of a data packet taken out from a queue to be distributed exists in the distribution table, and if so, sending the data packet to a corresponding analysis thread; and if the analysis thread with the least conversation exists, the data packet is sent to the analysis thread with the least current data packet, and if the analysis thread with the least conversation and data packet exists, the data packet is sent to the analysis thread with the least historical conversation and/or historical data packet.
Further, when the data packet is distributed to the analysis thread corresponding to the five-tuple Hash value, the method further includes: and if the corresponding analysis thread is in an overload state currently, the analysis thread does not receive the data packet of the new session within a preset time period. Judging whether the analysis thread is in an overload state or not, comprising the following steps: when the current conversation quantity of the analysis thread and/or the data packet quantity reach a specified threshold value, judging the current conversation quantity is in an overload state; and adding the overloaded analysis thread into the overload list, and deleting the overloaded analysis thread from the overload list after the time period is reached. And if the current data packet quantity of the analysis thread reaches a specified threshold value, adding the data packets sent to the analysis thread into a queue to wait for analysis. For the analysis threads located in the overload list, no packets for the new session are distributed.
And counting the data packets of each session in real time, and when the number of the data packets of the same session reaches a preset value, stopping receiving the data packets of a new session by an analysis thread corresponding to the session within a preset time period.
Preferably, the data packets in the queue to be distributed and the queuing queue are sorted according to a time line.
Compared with the prior art, the invention adopting the technical scheme at least has the following beneficial effects: after the data packet analysis engine is started, the data packets received by each packet capturing thread are actively and sequentially read, and when the number of the received data packets is lower than a certain number, namely the network flow is small, the active and sequential reading is changed into a passive waiting request mode, so that the received data packets can be ensured to be timely read, and the resource utilization is optimized. Firstly, data packets of the same session are distributed to the same analysis thread, and service continuity is kept; secondly, for the analysis thread in the overload state and the analysis thread receiving a larger conversation, the receiving of a new conversation is suspended, and the efficiency reduction caused by data backlog is reduced.
Drawings
Fig. 1 is a schematic diagram of a work flow of an embodiment of a data packet distribution method according to the present invention;
fig. 2 is a schematic diagram illustrating a deployment process of the packet capturing thread in fig. 1.
Detailed Description
The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for distributing a data packet includes: the engine reads the data packet received by each packet capturing thread, extracts quintuple information of the data packet as a session identifier, and adds the data packet into a queue to be distributed; and acquiring the data packet from the queue to be distributed and distributing the data packet to the analysis thread corresponding to the session identifier.
Preferably, the quintuple Hash value of the data packet is used as the session identifier, and the data packet is distributed to a preset analysis thread corresponding to the Hash value, that is, the data packets of the same session are distributed to the same analysis thread for analysis processing. The specific process comprises the following steps:
presetting a distribution table in which the corresponding relation between the five-tuple Hash value and the analysis thread is stored;
searching whether a five-tuple Hash value of a data packet taken out from a queue to be distributed exists in the distribution table, and if so, sending the data packet to a corresponding analysis thread;
if the analysis thread with the least conversation exists, sending the data packet to the analysis thread with the least conversation; further, if more than two analysis threads with the least current conversation and data packet exist, the data packet is sent to the historical conversation and/or the analysis thread with the least historical data packet.
The above-mentioned distributing the data packet to the preset analysis thread corresponding to the Hash value further includes: and if the corresponding analysis thread is in an overload state currently, the analysis thread does not receive the data packet of the new session within a preset time period.
And, judging whether the analysis thread is in an overload state includes: when the current conversation quantity of the analysis thread and/or the data packet quantity reach a specified threshold value, judging the current conversation quantity is in an overload state; and adding the overloaded analysis thread into the overload list, and deleting the overloaded analysis thread from the overload list after the time period is reached.
And if the current data packet quantity of the analysis thread reaches a specified threshold value, adding the data packets sent to the analysis thread into a queue to wait for analysis.
For the analysis thread in the overload list, not distributing the data packet of the new session; and counting the data packets of each session in real time, and when the number of the data packets of the same session reaches a preset value, stopping receiving the data packets of a new session by an analysis thread corresponding to the session within a preset time period.
Preferably, the data packets in the queue to be distributed and the queuing queue are sorted according to a time line.
As shown in fig. 2, as a preferred embodiment, after the engine is started, the engine actively reads the data packets received by each packet capturing thread in sequence; stopping the sequential reading when the total number of the captured data packets in a specific time period is lower than a specified threshold; and actively initiating a reading request to the engine only when the data packets received by the packet capturing thread reach a preset number.
The method specifically comprises the following steps: after the engine is started, regularly inquiring the state of each packet capturing thread, and performing reading operation when a new data packet exists in the packet capturing thread; and after stopping reading in sequence, when the cache data packets of each packet capturing thread reach the preset number, sending a reading request to the engine, and starting the reading of the data packets cached by the packet capturing thread by the engine.
Example one
The following describes a data packet distribution technical scheme of the present invention by taking an information auditing system as an example.
After an engine of the auditing system is started, an engine active reading mode is defaulted, whether each packet capturing thread receives a data packet or not is inquired regularly, if yes, the data packet is read, quintuple information in the data packet, namely a source IP, a destination IP, a source port, a destination port and a transport layer protocol, is extracted, a session identifier is formed by quintuples, and the data packets of the same quintuple belong to the same session.
Setting: and if the sum of the number of the data packets continuously read by all the packet capturing threads in 5 seconds is less than 100000, stopping active reading by the engine. Each packet capturing thread is provided with a cache for temporarily storing received data packets, for example, the upper limit of the cache of each packet capturing thread is set to 100 data packets, when the number of the cached data packets reaches the upper limit number, a read request is initiated to the engine, the engine sequentially takes out the data packets from the cache of the packet capturing thread for reading, and the cache of the packet capturing thread is cleared after the reading is completed, and certainly, if the number of the caches of a plurality of packet capturing threads reaches the upper limit, the engine can read in parallel.
And continuously counting the number of the data packets continuously read by all the packet capturing threads, if the number of the data packets is more than 100000 within 5 seconds, actively inquiring the state of each packet capturing thread by the engine again, and reading the received data packets. Therefore, the processor and the memory of the engine can be released for other processing operations, and system resources are saved.
And the data packet for extracting the quintuple information is put into a queue to be distributed and waits for being distributed to an analysis thread.
The specific data packet distribution process comprises the following steps:
firstly, a distribution table for storing a five-tuple information Hash value and corresponding to an analysis thread is preset, a data packet in a queue to be distributed is taken out, whether the five-tuple Hash value exists in the distribution table is judged, and if the five-tuple Hash value exists, the data packet is sent to the analysis thread corresponding to the five-tuple Hash value for analysis processing.
If no corresponding analysis thread exists, a corresponding analysis thread needs to be created for the session represented by the five-tuple Hash value, and the specific process may be as follows: sending the data packet to the analysis thread with the least conversation, and if more than two analysis threads with the least conversation exist, sending the data packet to the analysis thread with the least conversation; further, if more than two analysis threads with the least current conversation and data packet exist, the data packet is sent to the historical conversation and/or the analysis thread with the least historical data packet. And subsequently, the data packets with the same quintuple information are received and can be directly distributed to the corresponding analysis threads. The phenomenon that analysis thread queues are idle and inconsistent due to different conversations is avoided, and the conversations are reasonably distributed to the analysis threads.
Meanwhile, whether a corresponding analysis thread exists or not, when the current conversation quantity of the analysis thread and/or the data packet quantity reach a specified threshold value, the overload state is judged; and adding the overloaded analysis thread into the overload list, and deleting the overloaded analysis thread from the overload list after the time period is reached.
And if the current data packet quantity of the analysis thread reaches a specified threshold value, adding the data packets sent to the analysis thread into a queue to wait for analysis.
For the analysis thread in the overload list, not distributing the data packet of the new session; and counting the data packets of each session in real time, when the number of the data packets of the same session reaches a preset value, the session is larger, packet capturing and analysis can last for a longer time, the corresponding analysis thread also stops receiving the data packets of the new session within a preset time period so as to process the larger session by centralized resources, and the preset time period can be only the time for completing the analysis of the data packets of the session.
According to the technical scheme, data packets of the same session are distributed to the same analysis thread, so that service continuity is maintained; secondly, for the analysis thread in the overload state and the analysis thread receiving a larger conversation, the receiving of a new conversation is suspended, and the efficiency reduction caused by data backlog is reduced.
Claims (10)
1. A method for distributing a data packet, comprising:
after the engine is started, data packets received by each packet capturing thread are sequentially and actively read; when the total number of the captured data packets in a specific time period is lower than a specified threshold value, stopping the sequential reading, and actively sending a reading request to an engine when the data packets received by a packet capturing thread reach a preset number;
extracting quintuple information of the data packet as a session identifier, and adding the data packet into a queue to be distributed;
and acquiring the data packet from the queue to be distributed and distributing the data packet to the analysis thread corresponding to the session identifier.
2. The packet distribution method according to claim 1, wherein a five-tuple Hash value of the packet is used as a session identifier, and the packet is distributed to a preset analysis thread corresponding to the Hash value.
3. The data packet distribution method according to claim 1,
after the engine is started, regularly inquiring the state of each packet capturing thread, and performing reading operation when a new data packet exists in the packet capturing thread;
and after stopping reading in sequence, when the cache data packets of each packet capturing thread reach the preset number, sending a reading request to the engine, and starting the reading of the data packets cached by the packet capturing thread by the engine.
4. The data package distribution method according to claim 3, wherein the distributing the data package to the analysis thread corresponding to the session identifier thereof comprises,
presetting a distribution table in which the corresponding relation between the five-tuple Hash value and the analysis thread is stored;
searching whether a five-tuple Hash value of a data packet taken out from a queue to be distributed exists in the distribution table, and if so, sending the data packet to a corresponding analysis thread; and if the analysis thread with the least conversation exists, the data packet is sent to the analysis thread with the least current data packet, and if the analysis thread with the least conversation and data packet exists, the data packet is sent to the analysis thread with the least historical conversation and/or historical data packet.
5. The method according to claim 4, wherein when distributing the packet to the analysis thread corresponding to the five-tuple Hash value, the method further comprises: and if the corresponding analysis thread is in an overload state currently, the analysis thread does not receive the data packet of the new session within a preset time period.
6. The method of claim 5, wherein determining whether the analysis thread is in an overload state comprises: when the current conversation quantity of the analysis thread and/or the data packet quantity reach a specified threshold value, judging the current conversation quantity is in an overload state; and adding the overloaded analysis thread into the overload list, and deleting the overloaded analysis thread from the overload list after the time period is reached.
7. The method of claim 6, wherein if the number of current packets in the analysis thread reaches a predetermined threshold, the packets sent to the analysis thread are queued for analysis.
8. The method of claim 6, wherein no packets of a new session are dispatched for an analysis thread located in the overload list.
9. The method according to claim 6, wherein the data packets of each session are counted in real time, and when the number of data packets of the same session reaches a preset value, the analysis thread corresponding to the session stops receiving the data packets of a new session within a preset time period.
10. The method according to claim 1 or 6, wherein the data packets in the queue to be distributed and the queuing queue are sorted according to time line.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011206311.5A CN112565101A (en) | 2020-11-03 | 2020-11-03 | Data packet distribution method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011206311.5A CN112565101A (en) | 2020-11-03 | 2020-11-03 | Data packet distribution method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112565101A true CN112565101A (en) | 2021-03-26 |
Family
ID=75041744
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011206311.5A Pending CN112565101A (en) | 2020-11-03 | 2020-11-03 | Data packet distribution method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112565101A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2807656C1 (en) * | 2023-08-31 | 2023-11-21 | Акционерное общество "Научно-производственное предприятие "Цифровые решения" | Balancing method while maintaining integrity of data flows |
-
2020
- 2020-11-03 CN CN202011206311.5A patent/CN112565101A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2807656C1 (en) * | 2023-08-31 | 2023-11-21 | Акционерное общество "Научно-производственное предприятие "Цифровые решения" | Balancing method while maintaining integrity of data flows |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109257293B (en) | Speed limiting method and device for network congestion and gateway server | |
| US8365173B2 (en) | Method and apparatus for on-demand resource allocation and job management | |
| CN102487494B (en) | Short message flow control method and system | |
| US10554430B2 (en) | Systems and methods for providing adaptive flow control in a notification architecture | |
| CN101577671A (en) | Method and system for automatically controlling flow of peer-to-peer networking service | |
| CN106230997A (en) | A kind of resource regulating method and device | |
| CN113347111B (en) | Flow adjusting method and management system based on user experience | |
| CN110868323B (en) | Bandwidth control method, device, equipment and medium | |
| Desmouceaux et al. | SRLB: The power of choices in load balancing with segment routing | |
| CN113938435A (en) | Data transmission method, data transmission device, electronic device, storage medium, and program product | |
| CN112565101A (en) | Data packet distribution method | |
| US20230337266A1 (en) | Method, and electronic device for allocating routing resources of wifi6 router | |
| WO2020063661A1 (en) | Flow congestion monitoring method and device | |
| CN111464453A (en) | Message forwarding method and device | |
| CN114143263B (en) | Method, equipment and medium for limiting current of user request | |
| CN112055382B (en) | Service access method based on refined distinction | |
| JP2007329617A (en) | Communication control process executing method and system, and program therefor | |
| CN112436979A (en) | Cloud network traffic acquisition method and system | |
| US20050044168A1 (en) | Method of connecting a plurality of remote sites to a server | |
| CN117201202B (en) | Reflection amplification Flood attack flow storage method | |
| CN105812437A (en) | Business distribution method, system and related device | |
| CN113194158B (en) | Information storage method, device, equipment and computer readable storage medium | |
| CN109639802B (en) | Link statistics management method and device | |
| GB2577526A (en) | Network slice management | |
| CN112272210B (en) | Message caching method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210326 |