CN111857731A

CN111857731A - Flux storage method based on linux platform

Info

Publication number: CN111857731A
Application number: CN202010757584.2A
Authority: CN
Inventors: 吴建亮; 胡鹏; 魏欣怡
Original assignee: Guangzhou Jeeseen Network Technologies Co Ltd
Current assignee: Guangzhou Jeeseen Network Technologies Co Ltd
Priority date: 2020-07-31
Filing date: 2020-07-31
Publication date: 2020-10-30

Abstract

A flux storage method based on a linux platform is based on a netfilter framework of a linux system, and data delivery is carried out between a kernel layer and a user layer through a ring cache region. The annular buffer area is provided with two pointers: head and tail, the head points to the position of the next reading, and the tail points to the position of the next writing; the annular cache region stores by adopting an array, and the memory addresses of elements in the array are continuous; and circular access of data in the array is realized through moving the head pointer and the tail pointer. The invention can store the flow with high performance and high quality. Compared with the libpcap, the method has higher performance and lower packet loss rate. Compared with pfring and dpdk, the device is simple and convenient to install and good in compatibility.

Description

Flux storage method based on linux platform

Technical Field

The invention relates to the technical field of flow grabbing, in particular to a flow storage method based on a linux platform.

Background

The conventional common packet grabbing technologies include libpcap, pfring, dpdk and the like. The packet capture mechanism of the libpcap is to add a bypass process on a data link layer, not to interfere the process of the network protocol stack of the system, filter and buffer the sent and received data packets through a Linux kernel, and finally directly transmit the data packets to an upper application program.

The PF _ RING is invented by the Luca Deri to improve the efficiency of processing the data packet by the kernel, and also considers patches of application programs, such as Libpcap, TCPDUMP and the like, and auxiliary programs (such as ntop checking and analyzing network flow and the like). PF _ RING is a new type of network socket that can greatly improve the speed of packet capture. The method realizes complete zero copy, maps the memory space of the user to the memory space of the driver, and enables the application of the user to directly access the register and the data of the network card.

Similar to pfring, Intel DPDK allows processes in user space to directly access the network card using the library provided by the DPDK without passing through the kernel, which is higher in performance on packet processing than pfring, DPDK.

The libpcap is simple to install and has no limitation of a network card, but the packet loss rate is high. pfring is cumbersome to install, requiring that the original driver be uninstalled first, and then recompiled and install the drivers in pfring into the system. And not all network cards support. The dpdk also needs to be compiled and installed, and is generally used as a protocol stack of an application layer, so that the network card is also limited.

Therefore, it is necessary to provide a linux platform-based traffic preservation method with good compatibility, simple installation, good stability and low packet loss rate, aiming at the defects of the prior art.

Disclosure of Invention

The invention aims to avoid the defects of the prior art and provides a flux storage method based on a linux platform, which has the characteristics of good compatibility, simplicity in installation, good stability and low packet loss rate.

The object of the invention is achieved by the following technical measures.

The method is based on a netfilter framework of a linux system, and data delivery is carried out between a kernel layer and a user layer through a ring cache region.

Preferably, IN the method for saving the flow based on the linux platform, an nf _ register _ hook function is called IN a netfilter frame of the linux system, and a flow grabbing function is registered at a LOCAL _ IN hook point so as to drive the flow grabbing function to grab the flow data when the flow enters.

Preferably, in the traffic saving method based on the linux platform, the annular cache area is provided with two pointers: head and tail, the head points to the position of the next reading, and the tail points to the position of the next writing;

the annular cache region stores by adopting an array, and the memory addresses of elements in the array are continuous;

and circular access of data in the array is realized through moving the head pointer and the tail pointer.

Preferably, in the traffic saving method based on the linux platform, when traffic enters, the traffic grabbing function is driven to grab traffic data, and when the annular cache area has a space, the grabbed traffic data is written into the annular cache area; otherwise, the ring buffer is not written.

Preferably, in the traffic saving method based on the linux platform, for the ring cache area, when the head is tail, the ring cache area is empty; when head is (tail + 1)% bufferSize, the annular buffer zone is in a full state, wherein the bufferSize is the maximum storage capacity of the annular buffer zone.

Preferably, in the traffic saving method based on the linux platform, the data format stored in the ring cache area is consistent with the pcap file format.

Preferably, in the traffic saving method based on the linux platform, the application layer detects whether the traffic data exists in the ring buffer area according to the same time interval, and if the traffic data exists, the application layer reads the traffic data and stores the traffic data in the disk.

Preferably, in the traffic saving method based on the linux platform, after the application layer reads the traffic data, the application layer obtains the time information according to the stream data, and stores the time information into the disk by taking the hour as the file name.

Preferably, in the traffic saving method based on the linux platform, after the disk stores the traffic data, the application layer compiles the data in the disk.

Preferably, in the traffic saving method based on the linux platform, the application layer detects whether traffic data exists in the ring buffer according to a time interval of 1 second.

The flow storage method based on the linux platform is based on a netfilter framework of a linux system, and data delivery is carried out between a kernel layer and a user layer through a ring cache region. The invention is based on the combination of netfilter and ring buffer, and can store high-performance and high-quality flow. Compared with the libpcap, the method has higher performance and lower packet loss rate. Compared with pfring and dpdk, the device is simple and convenient to install and good in compatibility.

Drawings

The invention is further illustrated by means of the attached drawings, the content of which is not in any way limiting.

Fig. 1 is a schematic diagram of a system architecture of a flux saving method based on a linux platform.

FIG. 2 is a flow chart of a flux saving method based on a linux platform according to the present invention.

Fig. 3 is a schematic diagram of the netfilter framework of the linux platform.

Detailed Description

The invention is further illustrated by the following examples.

Example 1.

A flux storage method based on a linux platform is based on a netfilter framework of a linux system, and data delivery is carried out between a kernel layer and a user layer through a ring cache region.

Fig. 1 is a system architecture diagram according to the method, and fig. 2 is a flow chart of the flux saving method based on the linux platform according to the present invention.

According to the method for saving the flow based on the linux platform, an nf _ register _ hook function is called IN a netfilter frame of a linux system, and a flow grabbing function is registered at a LOCAL _ IN hook point so that when flow enters, the flow grabbing function is driven to grab flow data.

netfilter is a subsystem introduced by Linux 2.4.x as a generic, abstract framework that provides a complete set of management mechanisms for hook functions, enabling connection tracing such as packet filtering, Network Address Translation (NAT) and protocol type based.

The netfilter architecture is that several detection points (HOOKs) are placed at several positions of the whole network flow, and processing functions are registered at each detection point for processing, and the positions of five HOOK points of the IP layer are shown in fig. 3. The method selects and calls the nf _ register _ hook function, and registers the flow capture function on the LOCAL _ IN hook point.

The ring buffer typically has a read pointer and a write pointer. The read pointer points to data readable in the ring buffer and the write pointer points to a writable buffer in the ring buffer. Data reading and writing of the buffer can be realized by moving the read pointer and the write pointer.

Specifically, the ring buffer in this embodiment is provided with two pointers: head and tail, head points to the next read location and tail points to the next write location. The ring buffer is stored in an array with consecutive memory addresses of the elements in the array, which is CPU cache friendly-that is, at the hardware level, the elements in the array are preloaded, so in RingBuffer the CPU does not need to occasionally load the next element in the array from main memory. And circular access of data in the array is realized through moving the head pointer and the tail pointer. When head is tail, the annular buffer area is empty; when head is (tail + 1)% bufferSize, the annular buffer zone is in a full state, wherein the bufferSize is the maximum storage capacity of the annular buffer zone.

The flow storage method based on the linux platform drives a flow capture function to capture flow data when flow enters, and writes the captured flow data into an annular cache area when the annular cache area has a space; otherwise, the ring buffer is not written.

The data format stored in the annular buffer area is consistent with the format of the pcap file. The application layer detects whether the flow data exist in the annular buffer area or not according to the same time interval, if so, the flow data are read and stored in the disk, and the time interval can be 1 second or 5 seconds or other time periods. Preferably, after the application layer reads the flow data, the application layer acquires time information according to the flow data, the time information is stored in the disk according to the hour as the file name, only one time check is performed in the process, the reading speed in the ring buffer is increased, and the packet loss rate is greatly reduced. And after the magnetic disk stores the flow data, the application layer compiles the data in the magnetic disk.

The specific flow of the invention for flow storage is as follows:

1. when the flow enters, the captured flow data is driven to be stored in the ring buffer, and the data format stored in the ring buffer is consistent with the format of the pcap file.

2. The application layer reads data from the ring buffer and stores the data into the disk

And after data is acquired from the ring buffer, according to the structure body in the step 1, acquiring time, and storing the time into a disk by taking the hour as a file name. In the process, only 1 time is checked, so that the reading speed from the ring buffer is increased, and the packet loss rate is greatly reduced.

3. And (4) compiling and installing the driver, and selecting the driver code not to be associated with the bottom-layer network card. Can be compiled on Linux 2.4.x and above operating systems. After compiling, directly loading the driver by insmod.

The invention is based on the combination of netfilter and ring buffer, and can store high-performance and high-quality flow. Compared with the libpcap, the method has higher performance and lower packet loss rate. Compared with pfring and dpdk, the device is simple and convenient to install and good in compatibility.

Based on the combination of netfilter and ring buffer, the flow data is rapidly stored according to time, the stored flow data can be used as a stable and reliable data source for abnormal flow analysis, sample extraction and web attack, and the time cost for learning other complex packet capturing technologies is reduced.

Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting the protection scope of the present invention, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A flux storage method based on a linux platform is characterized in that data delivery is carried out between a kernel layer and a user layer through a ring cache region based on a netfilter framework of a linux system.

2. The linux platform-based traffic preservation method according to claim 1, wherein IN a netfilter framework of a linux system, an nf _ register _ hook function is called, and a traffic grab function is registered at a LOCAL _ IN hook point so as to be driven to grab traffic data when there is traffic entering.

3. The linux platform-based traffic preservation method of claim 2, wherein the ring buffer is provided with two pointers: head and tail, the head points to the position of the next reading, and the tail points to the position of the next writing;

4. The linux platform-based traffic preservation method of claim 3,

when the flow enters, the flow grabbing function is driven to grab the flow data, and when the annular cache area has a space, the grabbed flow data are written into the annular cache area; otherwise, the ring buffer is not written.

5. The linux platform-based traffic preservation method of claim 4, wherein for a ring buffer, when head ═ tail, the ring buffer is empty; when head is (tail + 1)% bufferSize, the annular buffer zone is in a full state, wherein the bufferSize is the maximum storage capacity of the annular buffer zone.

6. The linux platform-based traffic preservation method according to any one of claims 1 to 5, wherein a data format stored in the ring cache is consistent with a pcap file format.

7. The linux platform-based traffic preservation method of claim 6, wherein the application layer detects whether the traffic data exists in the ring buffer at the same time interval, and if so, reads the traffic data and stores the traffic data in the disk.

8. The linux platform-based traffic preservation method according to claim 7, wherein after the application layer reads the traffic data, the application layer obtains time information according to the stream data, and stores the time information into the disk with a filename in an hour basis.

9. The linux platform-based traffic preservation method of claim 8, wherein the application layer compiles data in the disk after the disk stores the traffic data.

10. The linux platform-based traffic preservation method of claim 7, wherein the application layer detects whether traffic data exists in the ring buffer at time intervals of 1 second.