RU2790636C1 - Method for detecting duplicate packets in a network traffic flow - Google Patents

Abstract

FIELD: data networks.

SUBSTANCE: maximum time during which packets with the same descriptor value are considered duplicates (LIMIT_T) is set. If it is determined in the duplicate search block that the parameters of the analyzed packet from the descriptor do not match the parameters from the RAM memory, then the information about the packet is overwritten in the RAM memory, and the packet is transferred to the output interface. If it is determined in the duplicate search block that the parameters of the analysed packet from the descriptor match the parameters of the packet from RAM memory, and the time differenceΔT is greater than or equal to the set time LIMIT_T, then the packet parameters are overwritten in RAM-memory, and the packet is transmitted to the output interface. If it is determined in the duplicate search block that the parameters of the analysed packet from the descriptor match the parameters of the packet from RAM memory, and the time differenceΔT is less than the set time LIMIT_T, then the packet is removed from the packet buffer.

EFFECT: increasing the performance of systems for deep analysis of network traffic by reducing the load through eliminating duplicate packets.

1 cl, 3 dwg, 3 ex

Description

The invention relates to the field of building systems for monitoring and analyzing data transmission networks and can be used in devices for pre-processing data packets for systems for in-depth analysis of network traffic that support the TCP/IP protocol stack.

The closest analogue to the proposed method is the traffic deduplication method described in US patent 10091075 B2 (class H04L 12.26, 02.10.2018). The known method includes receiving the input stream of packets through the input interface, passing it to the packet analysis unit, in which the Ethernet, IP and TCP / UDP headers are extracted from the packet and the descriptor is formed on their basis, the packets are stored in the packet buffer, and the descriptors are stored in descriptor buffer.

The disadvantage of the known method is the inability to ignore individual fields of Ethernet, IP and TCP / UDP headers, such as MAC addresses, VLAN, Traffic class, IP checksum, TTL, which can change during the packet passing through the network.

The technical problem solved by the claimed invention is to improve the performance of systems for deep analysis of network traffic by reducing the load by eliminating duplicate packets.

The technical result consists in reducing the load on deep analysis systems by reducing the time spent on analyzing a data packet, as well as improving the accuracy of detecting duplicate data packets, including those with changing header fields in Ethernet, IP, TCP/UDP packets.

The technical result is achieved by the fact that in the claimed method for determining duplicate packets in the network traffic stream, the input stream of packets is received through the input interface, transferred to the packet analysis unit, in which the Ethernet, IP and TCP / UDP headers are extracted from the packet and formed on them descriptor based, store packets in a packet buffer and descriptors in a descriptor buffer, by suggestion

LIMIT_T is set via a configuration interface not connected to the network in the duplicate search block and remains unchanged until it is changed by the operator,

in the packet analysis block, the time of receipt of the packet is recorded,

descriptors are transferred from the descriptor buffer to the HASH calculation block,

in the HASH calculation block based on the packet header fields, including the destination IP address, source IP address, IP DSCP, IP ID packet identifier, IP OFFSET fragment offset, L4 protocol code, destination TCP/UDP port, TCP/UDP port the sender, the sequence number of the packet in TCP, the HASH sum is calculated using the counter algorithm with Galois authentication;

then the descriptor with the calculated HASH sum enters the duplicate search block, which loads the descriptor from RAM memory at the address equal to the value of the HASH sum of the descriptor of the incoming packet, after which the packet parameters from the descriptors are compared, the time difference (ΔТ) is determined from the descriptor of the analyzed packet and a descriptor loaded from RAM memory,

if it is determined in the duplicate search block that the parameters of the analyzed packet from the descriptor do not match the parameters of the packet from the descriptor stored in RAM memory, regardless of the value of the time difference ΔТ, then the information about the packet is rewritten to the RAM memory, the command is sent to the packet deletion block to transfer a packet from the packet buffer to the output interface, and the packet analysis unit is given a command to receive a new packet from the input interface,

if it is determined in the duplicate search block that the parameters of the analyzed packet from the descriptor match the parameters of the packet from the descriptor stored in RAM memory, and the time difference ΔT is greater than or equal to the specified time LIMIT_T, then the packet parameters in RAM memory are overwritten by the packet deletion block a command is sent to transfer a packet from the packet buffer to the output interface, and the packet analysis unit receives a command to receive a new packet from the input interface,

if the duplicate search block determines that the parameters of the packet being analyzed from the descriptor match the parameters of the packet from the descriptor stored in RAM memory, and the time difference ΔT is less than the specified time LIMIT_T, then the command to delete the packet from the packet buffer is sent to the packet removal block, and the analysis block packet – command to receive a new packet from the input interface.

The maximum time that packets with the same descriptor value are considered duplicates, LIMIT_T, ranges from 10ms to 30s. LIMIT_T is set by the operator via a non-networked configuration interface in the duplicate finder and remains unchanged until it is changed by the operator.

The claimed invention is illustrated by the following graphics:

fig. 1 is a diagram illustrating the functional blocks involved in determining duplicate packets in a network traffic flow,

fig. 2 is a flowchart for detecting duplicate packets in a network traffic stream,

fig. 3 is a diagram illustrating the storage structure of packet information in a descriptor and a RAM memory.

The system for detecting duplicate packets in a network traffic flow (Fig. 1) contains an input interface (101), a packet analysis unit (102), a packet buffer (103), a descriptor buffer (104), a HASH calculation unit (105), a duplicate search unit ( 106), RAM memory block (107), packet deletion block (108), output interface (109).

The input interface (101) receives the network packet, checks for compliance with the requirements of the Ethernet protocol (preamble, start of the packet, checksum).

The packet analysis block (102) extracts network headers from the received packet: Ethernet, IP and TCP / UDP headers, registers the time of packet reception and generates a special information block - descriptor based on the specified data.

The packet buffer (103) is a storage block in which the entire packet is stored during the processing of the packet descriptor in the HASH calculation block (105) and the duplicate search block (106).

Descriptor buffer (104) - temporary storage where descriptors are stored while waiting for processing.

HASH calculation block (105) - a block in which the following fields are extracted from the descriptor: recipient IP address, sender IP address, L4 protocol code, recipient TCP / UDP port, sender TCP / UDP port and based on these fields a sequence is formed bytes (vector of bytes) over which the HASH sum is calculated according to the counter algorithm with Galois authentication (HASH sum calculation method). This is necessary in order to reduce the 13-byte descriptor to 4 bytes and facilitate the search for the descriptor in RAM-memory (107) by the key, which is the HASH sum.

Duplicate search block (106) - a block in which information about packets is compared: stored in RAM-memory (107) and the received packet. The descriptor stored there is retrieved from the RAM memory (107) by the address value, which is equal to the value of the HASH sum obtained in the HASH calculation block (105). In the duplicate search block (106), the packet parameters are compared and the time stamp difference value is checked in the TIME field of the descriptor stored in RAM memory (107) with the descriptor of the received packet.

RAM-memory block (107) - a block that is random access memory. The memory address is equal to the value of the HASH sum. Each memory cell stores information about packets, including: destination IP address, source IP address, IP DSCP, packet ID IP ID, IP OFFSET fragment offset, protocol code L4, destination TCP/UDP port, TCP/UDP -port of the sender, TCP sequence number of the packet (TCP SEQ) and packet reception time (TIME). Protocol code L4 – transport layer protocol code that defines TCP or UDP protocol.

Packet removal block (108) - a block that skips or does not skip (remove) a packet.

Output interface (109) - performs the transmission of a network packet to the network.

The above blocks: packet analysis block (102), descriptor buffer (104), HASH calculation block (105), duplicate search block (106) can be implemented or implemented on the basis of a field-programmable logic integrated circuit (FPGA) or a very large integrated circuit (VLSI) . Also, these blocks can be implemented by a state machine and several registers with appropriate internal logic and are implemented automatically, for example, from the presented network protocol, they are controlled by microcode stored in the built-in memory.

LIMIT_T is the maximum value of time, loaded by the operator, during which packages with the same descriptor value are considered duplicates. LIMIT_T is in the range from 10ms to 30s.

The above blocks, processing circuits/algorithms can also be implemented using a memory element that records processed data, and a processing processor, which can be a general purpose processor, state machine or other hardware-programmable logic device, discrete logic element or transistor logic , a discrete hardware component, or any combination thereof, to perform the functions described above. An FPGA or VLSI can be programmed to perform the proposed method and placed anywhere on the data center computer network or network packet brokers (NPBs).

The claimed method for determining duplicate packets in a network traffic flow is as follows.

The operator, through a configuration interface not connected to the network, sets the maximum value of time during which packets with the same descriptor value are considered duplicates (LIMIT_T). LIMIT_T remains the same for all transmitted packets until it is changed by the operator.

The input packet stream (FIG. 1, FIG. 2) is received sequentially through the input interface (101), which includes the functions of converting from an electrical signal to a set of serial digital values and performing a check for compliance with the requirement of the Ethernet protocol.

The packets are then passed to the packet parsing unit (102), which extracts the Ethernet, IP, and TCP/UDP headers from each packet and records the time the packet was received. Then the descriptor is formed.

In this case, the descriptor is stored in the descriptor buffer (104), and the packet is stored in the packet buffer (103). At the same time, in the packet buffer (103) (Fig. 3), the data packet is loaded in full, and since its size is much larger than the size of the descriptor, the time spent on loading it into the packet buffer is usually greater than or equal to the time spent to handle the handle.

The descriptor is passed to the HASH calculation block (105) (Fig. 1), in which, based on the packet header fields (IP addresses, IP identifiers, L4 protocol code, TCP / UDP ports of the recipient and sender), the HASH sum is calculated using the counter algorithm with Galois authentication. The protocol code L4 indicates the protocol number in the "Protocol" field of the packet header (what protocol data does the packet contain, for example, TCP, UDP).

The counter with Galois authentication is a variation of the mode of operation of symmetric block ciphers. In the counter algorithm with Galois authentication, the input blocks of the byte vector are numbered sequentially, the block number is encoded by the block algorithm. The output of the encryption function is used in XOR operation (exclusive or) with the plaintext to get the HASH sum. The scheme is a stream cipher, so using a unique byte vector guarantees a unique HASH sum.

Next, the descriptor with the calculated HASH sum goes to the duplicate search block (106).

The duplicate search block (106) loads the descriptor from RAM memory (107) at the address equal to the value of the HASH sum of the incoming packet descriptor. After that, in the duplicate search block (106), the parameters of the packets from the descriptors are compared, the time difference (ΔТ) stored in the descriptors of packets with the same HASH sum is determined, where

ΔT is the time difference between the packet descriptor stored in the descriptor of the analyzed packet and the packet descriptor stored in the RAM memory (107).

If the parameters of the packets from the descriptors are the same, then the difference between the time values (ΔT) stored in the descriptor of the analyzed packet and stored in the RAM memory (107) of the packet descriptor is calculated. The resulting value is compared with the maximum time that packets with the same descriptor value are considered duplicates (LIMIT_T).

If it is determined in the duplicate search block (106) that the parameters of the analyzed packet from the descriptor do not match the parameters of the packet from the descriptor stored in the RAM memory (107), regardless of the value of the time difference ΔT, then the information about the packet is overwritten in the RAM memory , the packet removal unit (108) receives a command to transfer a packet from the packet buffer (103) to the output interface (109), and the packet analysis unit (102) receives a command to receive a new packet from the input interface (101).

If in the duplicate search block (106) it is determined that the parameters of the analyzed packet from the descriptor match the parameters of the packet from the descriptor stored in RAM memory (107), and the time difference ΔT is greater than or equal to the specified time LIMIT_T (ΔT ≥ LIMIT_T), then overwriting the packet parameters in RAM memory (107), the packet deletion unit (108) receives a command to transfer the packet from the packet buffer (103) to the output interface (109), and the packet analysis unit (102) receives a command to receive a new packet from the input interface (101).

If in the duplicate search block (106) it is determined that the parameters of the analyzed packet from the descriptor match the parameters of the packet from the descriptor stored in RAM memory (107) and the time difference ΔT is less than the specified time LIMIT_T (ΔT < LIMIT_T), then the packet removal block (108) a command is sent to remove the packet from the packet buffer (103), and the packet analysis unit (102) is given a command to receive a new packet from the input interface (101). This determines that the package is a duplicate and removes the package.

The input and output network interfaces of the switch can be, for example, the Ethernet standard.

As a configuration interface, depending on the element base, any digital interface can be used, for example, JTAG.

To ensure maximum throughput and minimum processing delay, the data filtering unit must be implemented in hardware based on programmable logic integrated circuits (FPGAs) or specialized custom or semi-custom integrated circuits.

The proposed method significantly reduces the time spent on processing a data packet by analyzing not the entire packet, but only its descriptor, i.e. pipeline work, which consists in accepting (loading) a data packet, simultaneously extracting a descriptor from it and making decisions about the presence / absence of a duplicate only by its HASH sum.

Example 1 The parameters of the analyzed packet from the descriptor of the received packet do not match the parameters of the packet from the descriptor stored in RAM-memory, regardless of the value of the time difference ΔТ

The packets are passed to the packet analysis block (102), which extracts the Ethernet, IP and TCP/UDP headers from each packet.

Then a descriptor is formed that includes the value of the packet header fields (recipient IP address, source IP address, IP DSCP, IP ID packet identifier, IP OFFSET fragment offset, L4 protocol code, recipient TCP / UDP port, TCP / UDP -port of the sender, TCP SEQ) and the time of receipt of the packet TIME, while the descriptor is stored in the descriptor buffer (104), and the packet is stored in the packet buffer (103).

The descriptor is passed to the HASH calculation block (105), in which the calculation of the HASH sum over the packet fields is performed.

In the duplicate search block (106), it is determined that the parameters of the analyzed packet from the descriptor do not match the parameters of the packet from the descriptor stored in RAM memory (107). The information about the packet is overwritten in the RAM memory (107), the packet deletion unit (108) receives a command to transfer the packet from the packet buffer (103) to the output interface (109), and the packet analysis unit (102) receives a command to receive a new packet from input interface (101).

Example 2. The parameters of the analyzed packet from the descriptor of the received packet match the parameters of the packet from the descriptor stored in RAM-memory, set LIMIT_T=10ms, time difference Δ T=11ms

The input stream of packets is received sequentially through the input interface (101). The operator loaded the value LIMIT_T=10 ms.

The packets are passed to the packet analysis block (102), which extracts the Ethernet, IP and TCP/UDP headers from each packet.

Then a descriptor is formed that includes the value of the packet header fields (recipient IP address, source IP address, IP DSCP, IP ID packet identifier, IP OFFSET fragment offset, L4 protocol code, recipient TCP / UDP port, TCP / UDP -port of the sender, TCP SEQ) and the time of receipt of the packet TIME, while the descriptor is stored in the descriptor buffer (104), and the packet is stored in the packet buffer (103).

The descriptor is passed to the HASH calculation block (105), in which the calculation of the HASH sum over the packet fields is performed. The duplicate search block (106) loads the packet from the descriptor stored in RAM memory (107) at the address equal to the value of the HASH sum of the incoming packet descriptor. After that, in the duplicate search block (106), the parameters of packets from descriptors are compared, the time difference ΔT is calculated for packets from descriptors with the same HASH sum.

IN In the duplicate search block (106), it is determined that the packet parameters from the descriptor match the packet parameters from the descriptor in RAM memory (107) and, since ΔT=11ms > LIMIT_T=10 ms, the packet parameters are overwritten in RAM memory (107 ), the packet removal unit (108) receives a command to transfer a packet from the packet buffer (103) to the output interface (109), and the packet analysis unit (102) receives a command to receive a new packet from the input interface (101).

Example 3. The parameters of the analyzed packet from the descriptor of the received packet match the parameters of the packet from the descriptor stored in RAM-memory, set LIMIT_T=10 ms, time difference Δ T=7ms

The input stream of packets is received sequentially through the input interface (101). The operator loaded the value LIMIT_T=10 ms.

The packets are passed to the packet analysis block (102), which extracts the Ethernet, IP and TCP/UDP headers from each packet.

Then a descriptor is formed that includes the value of the packet header fields (recipient IP address, source IP address, IP DSCP, IP ID packet identifier, IP OFFSET fragment offset, L4 protocol code, recipient TCP / UDP port, TCP / UDP -port of the sender, TCP SEQ) and the time of receipt of the packet TIME, while the descriptor is stored in the descriptor buffer (104), and the packet is stored in the packet buffer (103).

The descriptor is passed to the HASH calculation block (105), in which the calculation of the HASH sum over the packet fields is performed.

In the duplicate search block (106), it is determined that the parameters of the analyzed packet from the descriptor match the parameters of the packet from the descriptor stored in RAM memory (107) and the time difference ΔT=7ms < LIMIT_T=10 ms.

The packet remover (108) is given a command to delete the packet from the packet buffer (103), and the packet analyzer (102) is given a command to receive a new packet from the input interface (101). This determines that the package is a duplicate and removes it.

Claims (9)

A method for detecting duplicate packets in a network traffic stream, including receiving an input packet stream through an input interface, passing it to a packet analysis unit, in which Ethernet, IP and TCP / UDP headers are extracted from the packet and a descriptor is formed on their basis, packets are stored in packet buffer, and descriptors - in the descriptor buffer, characterized in that LIMIT_T is set via a configuration interface not connected to the network in the duplicate search block and remains unchanged until it is changed by the operator, in the packet analysis block, the time of receipt of the packet is recorded, descriptors are transferred from the descriptor buffer to the HASH calculation block, in the HASH calculation block based on the packet header fields, including the destination IP address, source IP address, IP DSCP, IP ID packet identifier, IP OFFSET fragment offset, L4 protocol code, destination TCP/UDP port, TCP/UDP port the sender, the sequence number of the packet in TCP, the HASH sum is calculated using the counter algorithm with Galois authentication; then the descriptor with the calculated HASH sum enters the duplicate search block, which loads the descriptor from RAM memory at the address equal to the value of the HASH sum of the descriptor of the incoming packet, after which the packet parameters from the descriptors are compared, the time difference (ΔТ) is determined from the descriptor of the analyzed packet and a descriptor loaded from RAM memory, if it is determined in the duplicate search block that the parameters of the analyzed packet from the descriptor do not match the parameters of the packet from the descriptor stored in RAM memory, regardless of the value of the time difference ΔТ, then the information about the packet is rewritten to the RAM memory, the command is sent to the packet deletion block to transfer a packet from the packet buffer to the output interface, and the packet analysis unit is given a command to receive a new packet from the input interface, if it is determined in the duplicate search block that the parameters of the analyzed packet from the descriptor match the parameters of the packet from the descriptor stored in RAM memory, and the time difference ΔT is greater than or equal to the specified time LIMIT_T, then the packet parameters in RAM memory are overwritten by the packet deletion block a command is sent to transfer a packet from the packet buffer to the output interface, and the packet analysis unit receives a command to receive a new packet from the input interface, if the duplicate search block determines that the parameters of the packet being analyzed from the descriptor match the parameters of the packet from the descriptor stored in RAM memory, and the time difference ΔT is less than the specified time LIMIT_T, then the command to delete the packet from the packet buffer is sent to the packet removal block, and the analysis block packet – command to receive a new packet from the input interface.

Images