CN106059957B - Quickly flow stream searching method and system under a kind of high concurrent network environment - Google Patents

Abstract

The invention relates to a fast flow table search method and system in a high concurrent network environment. The method comprises: 1) counting the traffic entering the network interface, and setting the buffer window of the buffer according to the current traffic condition of the statistics; 2) according to the size of the buffer window set, utilizing quintuple information to process 3) Schedule each cached group according to the preset scheduling strategy, and send each group to the connection management module in turn; 4) The connection management module extracts the quintuple information of each group and performs flow table search process, find the corresponding flow entry, and use the data packets in the group to update the information of the flow entry. The invention is mainly applicable to the high-speed network flow processing system of the backbone link, can optimize the access cost of the connection management module under the high-speed network environment, and improve the access efficiency of the flow table.

Description

A fast flow table lookup method and system in a high-concurrency network environment

technical field

The invention belongs to the technical field of network security, and in particular relates to a fast flow table search method and system for a high-concurrency network environment.

Background technique

In a high-speed network environment, efficient connection management has become a key module of existing network traffic processing systems (such as intrusion detection, traffic accounting, etc. systems). Usually, the traffic processing system architecture is mainly divided into three modules: traffic acquisition, Connection management, business processing. Connection management provides flow traceability for business processing, including three operations: search, update, and delete. In order to accurately record each connection, the connection management module must maintain a connection table (or session table), wherein each connection table item traces a connection in the network, and is responsible for recording the identification ID, status and other related information of the connection, wherein the connection identification It is globally unique and generally consists of five-tuple information in the TCP/IP header.

The existing traffic processing system adopts a single-packet scheduling strategy: data packets are first buffered in the buffer of the network card, and then sent to the connection management module in order of arrival in the buffer to perform connection status update and maintenance operations. In a high-speed network environment, single-packet processing will not only bring a lot of function callback overhead, but also lead to performance bottlenecks in flow table access. As the number of concurrent connections increases, the size of the connection table continues to increase. Due to the limitations of hardware resources and the structure of the hash table, the number of slots in the hash table needs to be preset and it is very difficult to adjust dynamically. The growth of conflict chains reduces the search efficiency of the single-packet flow table. In the existing high-speed network with 10Gbps traffic, the packet speed reaches 10Mpps or even higher, and most packets in the network need to perform a flow table lookup operation. The flow table lookup frequency is equivalent to the packet arrival rate, and the flow table lookup efficiency has become One of the important performance of stream processing system. Based on this, it is necessary to design a scalable and efficient flow table lookup method to cope with the high-speed concurrent environment of the backbone network.

The current flow table lookup operation is divided into three types of implementation methods: hash table, Bloom filter, and content addressable memory. For a flow table using a hash table structure, a hash table lookup includes two steps: calculation of hash value and comparison of conflict chains. The worst-case performance of hash collisions handled by the join method is very poor: all N keys are inserted into the same slot, resulting in a linked list of length N, and the worst search length at this time is O(N) .

Based on this, a lot of work is focused on making the conflict chain lengths on each slot as equal as possible to ensure that the average search length is close to the best case O(1+α), where α is the loading factor. To achieve this, a good hash algorithm is required. Although complex cryptographic hash methods (MD5, SHA-1) can be used to achieve a balanced distribution of the lengths of all conflicting chains in the hash table, good hash functions usually Consumes a lot of CPU.

Compared with the single hash mentioned above, the effect of multi-hashing will be better. Multi-hashing will calculate the hash value multiple times, and finally insert it into the shortest one of the multiple sub-tables, but this makes it necessary to search for multiple conflict chains for each search, which will bring a large number of packets in the backbone network with dense packets. lookup cost.

In terms of optimizing the search operation with the help of network locality, using FPGA and SRAM to realize the high-speed cache of the flow table can speed up the access speed. Limited by the circuit complexity of the FPGA and the capacity limitation of the SRAM, the size of the flow table is limited by the storage capacity. In a high-concurrency network environment, affected by traffic fluctuations and burst traffic, a large number of active connections will be forced to be replaced, resulting in missed detection by the system.

Contents of the invention

In order to optimize the access overhead of the connection management module in a high-speed network environment, based on the high concurrency and slow update of the backbone link, there is a certain degree of localized traffic characteristics, the present invention provides a fast flow table lookup method and system, which is mainly applicable to In the high-speed network traffic processing system of the backbone link.

The main contents of the present invention include: (1) an efficient network flow grouping algorithm, grouping the data packets in the network card buffer according to the flow label; (2) threshold scheduling strategy, scheduling the grouped data packets; (3) Flow table lookup.

The core of the fast flow table lookup method of the present invention is to send the data packets in the network to the connection management module, so as to reduce the comparison times and callback overhead of the flow table lookup. The more packets arriving per connection after grouping, the more positive the fast lookup method will be. Therefore, efficient grouping algorithms are the basis of fast flow table lookup methods. Based on this, the design of the grouping algorithm mainly includes the following aspects:

1) The basis for grouping is the five-tuple information in the TCP/IP header. The connection in the connection management module is uniquely determined by the five-tuple information of source IP, destination IP, source port, destination port and transport layer communication protocol type in network communication.

2) Efficiency and flexibility of the grouping algorithm. Grouping operations will introduce a certain amount of time overhead, and a good data structure can greatly reduce the overhead of grouping operations, making the fast search method bring more positive effects.

3) For each group of data packets after grouping, their quintuples are the same and come from the same connection, and need to be efficiently indexed so that the scheduling strategy can efficiently schedule and maintain each group.

4) The operation object of the grouping algorithm is the data packet in the buffer of the network card, and the size of the buffer window needs to be considered in compromise. If the window is too large, it will not only consume a certain amount of memory space, but also increase the delay from capture to processing of data packets; if the window is too small, the number of packets cached by each connection will be too small, and the positive effect will be limited.

For the data packets that have been grouped, a certain scheduling strategy is required to send each group to the connection management module. A good scheduling strategy can not only enable the data packets in each group to get a fair scheduling opportunity, but also provide for quick search. method brings more positive effects. The design of the scheduling strategy mainly includes the following design contents:

1) For cached packets, packets with a large number of packets should be scheduled first, and packets with a small number of packets should be scheduled later to wait for more data packets to be cached, which can reduce more flow table access overhead.

2) There should be no starvation phenomenon in the delayed scheduling group, that is, no scheduling opportunity for a long time, which will make the system miss detection.

The fast flow table lookup solution provided by the present invention sends the data packets in the network to the connection management module to reduce the comparison times and callback overhead of the flow table lookup, and through a reasonable scheduling strategy, the data packets in each group are Can get a fair scheduling opportunity. This method is mainly applicable to the high-speed network traffic processing system of the backbone link, and can optimize the access cost of the connection management module in the high-speed network environment, and improve the access efficiency of the flow table.

Description of drawings

Fig. 1 is a schematic diagram of the system structure of the present invention.

Fig. 2 is a schematic diagram of a data stream packet structure.

Figure 3 is a schematic diagram of Q1Q2 queue migration.

Figure 4 is a comparison of flow table access time in scenario A.

Figure 5 is a comparison chart of flow table lookup lengths in scenario A.

Figure 6 is a comparison of flow table access time in scenario B.

Fig. 7 is a comparison diagram of flow table lookup length in scenario B.

Detailed ways

The present invention will be further described below through specific embodiments and accompanying drawings.

The general frame of the present invention is as shown in Figure 1, is made up of six parts of network interface, buffer window management module, data stream grouping module, starvation avoiding module, packet scheduler, connection management module, and operation steps are as follows:

1) When the traffic enters the network interface, the traffic situation is counted, and the traffic statistics are sent to the buffer window management module; the buffer window management module selects one from the preset window sizes according to the current traffic situation;

2) According to the set window size, the data stream grouping module performs a grouping operation on the arriving data packets, and when the scheduling opportunity arrives, triggers the grouping scheduler;

3) After receiving the trigger instruction, the packet scheduler schedules each cached packet according to the scheduling strategy, and sends each packet to the connection management module in turn;

4) The starvation avoidance module is responsible for collecting scheduling information from the packet scheduler, and triggering the packet scheduler to schedule unscheduled packets in due course;

5) The connection management module extracts the quintuple information of each group sent by the packet scheduler, performs a real flow table lookup process, finds the corresponding flow table entry, and then uses the data packets in the group to update the flow table entry in turn. Status and other information.

Next, the operation steps will be described in detail.

Buffer window management module: By collecting the current packet arrival speed and packet interval information, combined with the delay tolerance of the system, an appropriate window size K is selected. For example, the window size is preset with three values of 64, 256, and 512. The choice is 256, and its unit is "piece", which is used to describe how many data packets can be accommodated. The delay tolerance refers to the data packet processing delay that can be tolerated by the system or project. If the system delay tolerance is within 100us, the window selection is K=64; if the system delay tolerance is 100-500us, the window selection is K=256; otherwise, the window selection is K=512.

Data stream grouping module: as shown in Figure 2, in order to achieve efficient data stream grouping, the algorithm uses a hash table (PT in Figure 2) as the main structure to group data packets, and introduces index queues (Q1 and Q2 in Figure 2 ) to index the packet that has been grouped, and establish a bidirectional index relationship between the hash table and the index queue, the specific steps are as follows:

1) Whenever a data packet x arrives, the total number of packets cached_num in the current buffer is increased by 1, and the five-tuple information of x is extracted to represent its flow label x.fid, and the position of x in PT is calculated by x.fid j. Go to 2).

2) If PT[j] is empty, it means that x does not belong to any of the currently cached groups, you need to execute 3) Create a new group and establish a bidirectional index between PT and Q1; if PT[j] is not empty, Execute 4).

3) Store x and its quintuple information in PT[j], increase the number of packets PT[j].pkt_count that PT[j] has cached by 1, and add an item j at the tail t of Q1 to establish Q1 To the unidirectional index of PT[j], the number of elements in Q1 is increased by one, and the total number of packets Q1.pkt_count in Q1 is increased by 1; at the same time, Q1 and t information are added at the position of PT[j]: PT[j]. Q=Q1, PT[j].idx=t, a unidirectional index from PT to Q1 is established, and at this time, a bidirectional index between PT[j] and Q1 is established successfully. Go to 6).

4) PT[j] is not empty, indicating that position j of PT has maintained a group and index information. Compare PT[j].fid with x.fid, if they are equal, it means that x belongs to the group maintained by PT[j], go to 5); if they are not equal, it means that there is a hash conflict in the grouping process, and mark the conflict as submitflag set to 1, turn to 6).

5) Store x in PT[j], the number of packets PT[j].pkt_count already cached by PT[j] increases by 1, and the total number of packets Q.pkt_count in Q (Q1 or Q2) corresponding to j increases by 1; Go to 6).

6) If submitflag is 1, set submitflag to 0, trigger the packet scheduler to schedule, after scheduling, the position of PT[j] is empty, and execute 4); or, if the total number of packets in the current buffer cached_num=K, it means buffering If the zone is full, the packet scheduler should also be triggered for scheduling; after that, continue back to 1).

Packet scheduler:

1) For 4) and 5) in the data grouping module, if there is a hash collision in the grouping process, or if PT[j].pkt_count exceeds the historical average value, then perform the process 2) transfer PT[j] from Q1 to in Q2;

2) As shown in Figure 3. The figure (a) is to migrate PT[2] in Q1 to Q2, the dotted line in the figure indicates the index to be added in Q2, and PT[2] in Q1 will be removed). (b) The figure shows the index situation after the migration operation is completed. The one-way index indicated by the dotted line has been updated to the two-way index indicated by the solid line. At the same time, the PT[1] at the end of Q1 fills the removed PT[2]. The index Q1:3 page corresponding to PT[1] is updated to Q1:2, an element j is added at the tail t of Q2, the number of elements of Q2 is increased by 1, and a one-way index from Q2 to PT[j] is established. Q2.pkt_count plus PT[j].pkt_count. Through the one-way index from PT[j] to Q1, find its position PT[j].idx in Q1, find PT[i] through the element i stored in the tail t of Q1, and set PT[i].idx to PT [j].idx have the same value, and at the same time set the element at position PT[j].idx in Q1 to i, reduce the number of elements in Q1 by 1, and subtract PT[j] from the total package Q1.pkt_count in Q1. pkt_count, so far the operation of deleting PT[j] from Q1 is completed. Set PT[j].idx to t, PT[j].Q to Q2, and create an index from PT[j] to Q2.

3) If the scheduling is triggered by the data stream grouping module, submit the Q2 indexed groups to the connection management module; if the scheduling is triggered by the starvation avoidance module, then submit the Q1 indexed groups to the connection management module.

Starvation avoidance module:

1) By collecting the trigger information of 6) in the data flow grouping module, the packet scheduler is triggered to submit the group indexed in Q1 in good time.

2) Design grouping module conflict counter C1 and Q2 scheduling counter C2. The former is responsible for recording the number of hash collisions of the grouping module. Every time a conflict occurs, C1 increases by 1, and whenever the buffer is full, C1 decreases by 1 (if C1 is less than 0, set it to 0 ); the latter is responsible for recording the total number of packets scheduled by Q2, each time Q2 is scheduled, C2 accumulates Q2.pkt_count, when Q1 is scheduled, C2 is set to 0;

3) Whenever Q1.pkt_count=K or C1 exceeds 3 or the value of C2 exceeds 10 times of Q1.pkt_count, the scheduler is triggered to schedule Q1, and the value of the counter is cleared to 0.

Connection management module:

1) For the Q (Q1 or Q2) sent by the scheduler module, for each item i in Q, find a pair of PT[i], and use the first packet in PT[i] to search the real flow table process.

2) If the corresponding flow entry is found, each packet in PT[i] is used to update the flow entry in turn. If not found, create a new flow entry and update the flow state in sequence.

Two data sets are used to evaluate the present invention, and the basic information of the data sets is shown in Table 1.

Table 1. Basic information of the dataset

The effect of the present invention is evaluated by comparing whether to use the fast flow table lookup method under different time scales. The evaluation index adopts two dimensions: the average lookup length of the flow table and the average access time of the flow table. The results are shown in Figure 4-7, where Figure 4 is a comparison of flow table access time in scenario A, Figure 5 is a comparison of flow table lookup length in scenario A, Figure 6 is a comparison of flow table access time in scenario B, and Figure 7 is a comparison of flow table access time in scenario A Comparison of flow table lookup lengths in B. Experimental results show that under various traffic environments, the fast flow table lookup method proposed by the present invention has a good performance improvement, and can improve the access efficiency of the flow table.

Concrete steps of the present invention have used the realization mode of hash table and queue, but not limited to these two kinds of data structures, also can use other linear structure (as stack) to replace queue, can realize key-value mapping class data structure with other ( Such as red-black tree) instead of hash table structure.

The above embodiments are only used to illustrate the technical solution of the present invention and not to limit it. Those of ordinary skill in the art can modify or equivalently replace the technical solution of the present invention without departing from the spirit and scope of the present invention. The scope of protection should be determined by the claims.

Claims (10)

1. quick flow stream searching method under a kind of high concurrent network environment, which comprises the following steps:

1) flow for entering network interface is counted, according to the buffering window of the current traffic conditions setting buffer area of statistics Mouthful；

2) according to the size of the buffer window of setting, division operation is executed using data packet of the five-tuple information to arrival；

3) grouping of each caching is scheduled according to preset scheduling strategy, connection management mould successively is sent in each grouping Block；

4) connection management module extracts the five-tuple information of each grouping, carries out flow stream searching process, finds corresponding flow table , and use the information of the data packet update flow entry in grouping.

2. the method as described in claim 1, which is characterized in that step 1) passes through to current packet arrival rate and inter-packet gap information It is acquired, selects suitable buffer window size in conjunction with the delay degrees of tolerance of system.

3. the method as described in claim 1, which is characterized in that step 2) uses Hash table to carry out dividing for data packet for main structure Group, while introducing index queue and grouped data packet is indexed, and foundation is double between Hash table and index queue To index relative.

4. the method as described in claim 1, which is characterized in that the step 3) scheduling strategy includes:

A) for the grouping cached, the packet priority more than packet number is scheduled, and scheduling is delayed in the few grouping of packet number, waits and caching more More data packet, to reduce flow table access expense；

B) hunger phenomenon should not occur in the grouping for being delayed scheduling, i.e., do not obtain dispatcher meeting for a long time, to avoid system is generated System missing inspection.

5. quick flow stream searching system under a kind of high concurrent network environment using claim 1 the method, which is characterized in that Module and connection management module are avoided including buffer window management module, data stream packet module, packet scheduler, starvation；

The buffer window management module sets buffer area for caching traffic statistics, and according to current traffic conditions Buffer window；

The data stream packet module executes division operation to the data packet of arrival, and work as according to the buffer window size of setting Scheduling occasion triggers the packet scheduler when reaching；

After the packet scheduler receives triggering command, the grouping of each caching is scheduled according to preset scheduling strategy, The connection management module successively is sent in each grouping；

The starvation avoids module from being responsible for from the packet scheduler collection scheduling information, and triggers the packet scheduler in due course Unscheduled grouping is scheduled；

The connection management module extracts the five-tuple information for each grouping that packet scheduler is sent, and carries out flow stream searching mistake Journey finds corresponding flow entry, and the information of flow entry is updated using the data packet in grouping.

6. system as claimed in claim 5, which is characterized in that the buffer window management module is by reaching current packet Speed and inter-packet gap information are acquired, and select suitable window size in conjunction with the delay degrees of tolerance of system.

7. such as system described in claim 5 or 6, which is characterized in that the data stream packet module is tied using based on Hash table Structure carries out the grouping of data packet, while introducing index queue and being indexed to grouped data packet, and in Hash table and rope Draw and establishes two-way index relative between queue.

8. system as claimed in claim 7, which is characterized in that the step of data stream packet module is grouped is as follows:

1) when being reached a data packet x, total packet number cached_num of current buffer increases by 1, extracts the five-tuple letter of x Breath represents its number of failing to be sold at auction x.fid, and calculates position j of the x in Hash table PT by x.fid, turns 2)；

If 2) PT [j] is sky, show that x is not belonging to the grouping that any one has currently been cached, 3) execution establishes a new grouping simultaneously Establish the two-way index between PT and index queue Q1；If PT [j] is not sky, execute 4)；

3) x and its five-tuple information being stored in the position PT [j], PT [j] buffered packet number PT [j] .pkt_count increases by 1, And increase a j at the t of the tail portion Q1, it is established that the unidirectional index of Q1 to PT [j]；Total packet number Q1.pkt_count in Q1 increases 1, while increasing Q1 and t information: PT [j] .Q=Q1, PT [j] .idx=t, it is established that the single way cable of PT to Q1 in the position [j] PT Draw, the two-way index between PT [j] and Q1 is successfully established at this time, is turned 6)；

4) PT [j] for sky, show the position j of PT maintained one grouping and index information, by PT [j] .fid with X.fid is compared, if equal, indicated that x belongs to the grouping of PT [j] maintenance, is executed 5)；If unequal, it is meant that grouping process hair Hash-collision has been given birth to, conflict label submitflag is set 1, is turned 6)；If hash-collision has occurred in grouping process, if PT [j] .pkt_count is more than history average, then PT [j] is transferred to rope from index queue Q1 by the packet scheduler Draw in queue Q2；

5) by x storage in PT [j], PT [j] buffered packet number PT [j] .pkt_count increases total in Q corresponding to 1, j Packet number Q.pkt_count increases by 1, and wherein Q is Q1 or Q2；Turn 6)；

6) if submitflag is 1, submitflag is set 0, the packet scheduler is triggered and is scheduled, PT after scheduling The position of [j] is sky, is executed 4)；Alternatively, indicate that buffer area has been expired if total packet number cached_num=K of current buffer, Also the packet scheduler is triggered to be scheduled；Later, it continues back at 1).

9. system as claimed in claim 8, which is characterized in that the step of packet scheduler is scheduled is as follows:

1] for the step 4) that is grouped in data grouping module and 5), if hash-collision, Huo Zheruo has occurred in grouping process PT [j] .pkt_count is more than history average, then implementation procedure 2] PT [j] is transferred to index queue from index queue Q1 In Q2；

2] element number for increasing element a j, Q2 at the tail portion t of Q2 increases by 1, it is established that the unidirectional index of Q2 to PT [j], Q2.pkt_count adds PT [j] .pkt_count；The unidirectional index that Q1 is arrived by PT [j], finds its position PT in Q1 [j] .idx finds PT [i] by the element i that the tail portion Q1 t is stored, sets PT [j] .idx identical value for PT [i] .idx, simultaneously I is set by the element at position PT [j] .idx in Q1, the element number of Q1 is subtracted 1, Q1.pkt_count is always wrapped in Q1 and is subtracted PT [j] .pkt_count is removed, the operation for deleting PT [j] from Q1 is so far completed；T, PT [j] .Q are set by PT [j] .idx It is set as Q2, establishes the index that PT [j] arrives Q2；

3] if scheduling is triggered by data stream packet module, the grouping of Q2 index is submitted to connection management module by group；If adjusting Degree avoids module from triggering by starvation, then the grouping indexed in Q1 is submitted to connection management module by group.

10. system as claimed in claim 9, which is characterized in that the starvation avoids the treatment process of module are as follows:

1] it by the triggering information of step 6) in acquisition data stream packet module, triggers and is indexed in packet scheduler submission Q1 in due course Grouping；

2] grouping module collision counter C1 and Q2 schedule counter C2 are designed, the hash-collision that C1 is responsible for recording grouping module is secondary Number, every primary conflict C1 of generation increase by 1, and C1 subtracts 1 when buffer area is full, sets 0 if C1 is less than 0；It is accumulative that C2 is responsible for record Q2 Total packet number of scheduling, when dispatching Q2 every time, C2 adds up Q2.pkt_count, and when Q1 is scheduled, C2 is set to 0；

3] it when the value that Q1.pkt_count=K or C1 is more than 3 or C2 is more than 10 times of Q1.pkt_count, just triggers Scheduler schedules Q1, and the value for emptying counter is 0.