CN211293923U - Network data platform based on X86 board card - Google Patents

Abstract

The utility model relates to a network data platform based on X86 board, it is based on the network data platform of X86 board and carries out the high-speed collection of network data package on the calculation blade of X86 board, and carry out the rule filtration of the character string or regular expression of the collection data package, hit or gather the data simultaneously and carry out the packet structure analysis and processing, this platform has realized the multiple functions of the access collection/rule hit/analysis and processing of network data package, can handle to the 10G Ethernet signal that optic fibre inserts or the 40G Ethernet signal that inserts from the chassis backplate, insert PTN network signal as special analysis and processing platform in the actual production application, handle and output and hit data set and analysis and processing result set, this processing platform disposes and can be regarded as the processing unit of ATCA machine case on the calculation blade of X86, form a whole set of access filtration and analysis and processing equipment to PTN Ethernet signal with other access processing boards, the flexibility is strong; and the requirements of users are met.

Description

Network data platform based on X86 board card

Technical Field

The utility model relates to a network data gathers and filters and analysis platform, in particular to network data platform based on X86 integrated circuit board.

Background

The current network data processing platform is mostly collection analysis or independent collection function in function, less include after gathering directly carry out character string or regular expression filter matching function, and current data processing equipment does not support PTN signal this kind of data analysis who takes the unusual Ethernet structure of MPLS label, does not have filtering capability yet, and the user filters the analysis to the collection of class Ethernet structure's PTN signal to and the user is higher and higher to 60G signal processing ability's requirement. In terms of designing an implementation scheme, users have more and more requirements on deep analysis of data, currently, DPDK is used for data routing forwarding and simple data hierarchical analysis, DPDK and Hyperscan are used for butt joint of processing flows, secondary development is achieved, few cases exist, most of data processing equipment is used for analyzing and processing standard ethernet at present, and nDPI deep packet inspection is only used for analyzing and processing standard ethernet.

Disclosure of Invention

In view of integration and comprehensive demand to gathering filtration analysis function among the network data processing, the utility model provides a network data platform based on X86 integrated circuit board, secondary development and the integration of code level have been carried out DPDK and Hyperscan's development frame on X86 integrated circuit board calculation blade, let the unified filtration of gathering of access data, carry out the structural analysis of data simultaneously and ndpi's package in business identification and statistics, concrete technical scheme is, a network data platform based on X86 integrated circuit board, the external interface of integrated circuit board includes four 10 GE's optical ports, two RG 45's net gape, a console debugging interface, HDMI inserts the interface, two USB gapes, the backplate possesses the 40G net gape, a giga net gape, but X86 integrated circuit board system start-up inserts display and mouse keyboard, look over system start-up state, its characterized in that: the backplane 40GE of the x86 board card is connected with the backplane 40GE of the exchange board to provide a data flow path, the 10GE optical port of the front panel of the x86 board card is interconnected with the 10GE optical port of the rear panel of the line card for bottom polling acquisition function, the control interface of the equipment is used for realizing interconnection of the backplane and the Ethernet, and the backplane gigabit network ports of all the board cards are communicated to realize control instruction interaction; the system is divided into a data acquisition module, a rule processing module and a data analysis module according to functional modules, the data acquisition module is in one-way connection with the data analysis module, the data acquisition module, the rule hit processing module and the data analysis module are sequentially connected, and a user data rule configuration and screening unit, a rule management unit and a rule filtering hit unit are sequentially connected in the rule processing module.

The technical effect of the utility model is that the platform possesses processing unit's control interface, can fuse fast to complete machine equipment and carry out control function as the product and realize. The DPDK data development kit library Hyperscan regular matching engine library nDPI deep packet inspection library is flexibly applied, an API (application program interface) provided by the DPDK data development kit library nDPI deep packet inspection library is developed and linked for the second time, the seamless butt joint of collected and filtered data is realized, the processing platform has triple functions of collecting, filtering and analyzing, can be used simultaneously, can also be used independently as one of the functions, and is high in flexibility. The processing platform realizes the functions of acquiring, filtering and analyzing the PTN signals, meets the flexible processing of MPLS label data by users, outputs a filtering result set and an analysis and statistics result set for the users, and meets the requirements of the users on the filtering, acquisition, control and deep mining of the data.

Drawings

Fig. 1 is a block diagram of the hardware structure of the present invention;

fig. 2 is a functional block diagram of the present invention.

Detailed Description

The present invention will be further explained with reference to the accompanying drawings.

As shown in fig. 1 and 2, an external interface of an X86 board card includes four 10GE optical ports, two RG45 ports, a console debug interface, an HDMI access interface, two USB ports, a backplane with a 40G port and a gigabit port, an X86 board card system starts to be accessible to a display and a mouse keyboard, and checks the system start state, the backplane 40GE of the X86 board card is connected to the backplane 40GE of the switch board to provide a data flow path, the 10GE optical port of the front panel of the X86 board card is interconnected with the rear panel 10GE optical port of the line card for bottom polling acquisition function, and a control interface of the device is implemented by backplane ethernet interconnection to interconnect the backplane gigabit ports of all board cards to implement control instruction interaction; the system is divided into a data acquisition module, a rule processing module and a data analysis module according to functional modules, the data acquisition module is in one-way connection with the data analysis module, the data acquisition module, the rule hit processing module and the data analysis module are sequentially connected, and a user data rule configuration and screening unit, a rule management unit and a rule filtering hit unit are sequentially connected in the rule processing module.

A data acquisition, filtration and analysis method adopts an operating system of an X86 board card as a centros 7.4 linux platform, utilizes a DPDK high-speed acquisition Hyperscan high-performance filter engine for calculating the multi-core large-memory high main frequency processing performance of a blade by utilizing an X86 and a data acquisition, filtration and analysis method for nDPI deep packet inspection and analysis, comprises data acquisition, rule hit processing and data analysis,

firstly, data acquisition, 1, initializing a DPDK frame, taking a core id and a network card port number which need to be used as input starting parameters, configuring which core the data of one network card port is enqueued for receiving, dequeuing the data processing by which core to perform branch judgment processing, configuring an analysis processing core id, configuring a bottom polling enable bit of the network card port, configuring a mixed mode packet,

2. analyzing the kernel parameters, initializing a DPDK frame and analyzing the distribution condition of kernel ids by an application program, respectively starting each kernel thread, distributing rx queues and lock-free ring spaces for each kernel thread, applying for enough large pages for caching,

3. the five data receiving core threads are started, and respectively receive the data from the back panel 40GE and the front panel 10GE, carry out enqueue operation,

4. starting five data processing core threads, preparing data dequeue operation, walking different processing flows according to the roles of the core threads when dequeue reading data to be processed, entering a data falling collection module for processing if the data are the acquisition core threads, and entering a rule hit processing module for processing if the data are not the acquisition core threads;

second, rule hit processing, which is composed of new rule, delete rule, rule filtering hit,

the new rules are set up in bulk and in new,

1. the rules from the master control create the issue control,

2. the user issues a new rule, the user issues a corresponding keyword or regular expression rule through the BS control network management, the rule is defined as an L4 layer rule according to the OSI 7-layer protocol stack, the user can set L2 layer and L3 layer rules aiming at the target data to be filtered, namely, the L2 layer is a link layer rule, the L3 layer is an IP layer rule, the user sets the rule according to the application scene and issues to the master control, the master control issues to each board card according to the rule level,

3. rule analysis, dividing a complete rule into three parts according to the hit principle, configuring the three parts into a hyperscan rule database,

4. the first part writes the L2 level and L3 level rule IDs combined into fixed-position string rules into the hyperscan rules database, obtains a rule ID number,

5. the second part writes the keywords or regular expression rules of the specific L4 layer into a hyperscan rule database and obtains the second rule id, 6, the third part needs to do an AND operation on the rule ids of the first two rules, and sets hit return, the new rule is completed,

the rules of the waste rock and the deletion are set,

1. the user performs corresponding rule deletion operation on a BS network management control interface of the device,

2. after the control command is issued to the main control, the main control is issued to the x86 board card, the application program of the x86 board card analyzes and deletes the rule content, compares the rule content with the rule list in the local memory,

3. scanning the rule ID to be deleted, deleting the corresponding rule,

4. meanwhile, the deleted rule base is recompiled into the hyperscan rule database to achieve the effect of deleting the rule;

the first rule is a filtering hit,

1. the data flow is sent to a matching search interface hs _ scan of hyperscan to carry out rule filtering, hit matching, if the data packet accords with the rule in the rule database, the callback function is hit and called, the content of the hit packet is returned,

2. a hit rule ID for inquiring the drop processing mode of the packet according to the hit rule ID and the L2 layer and L3 layer rule IDs carried in the packet,

3. storing the data dump into a specified file directory and a specified file;

thirdly, analyzing the data, namely analyzing the data,

1. based on a core thread mechanism under a DPDK framework, a core thread id is specified and analyzed,

2. starting an analysis thread to scan an analysis processing data directory, wherein the directory scan supports multi-level nested directory scan, acquiring a currently unanalyzed pcap packet file under the directory scan, calling a packet reading interface to perform content reading operation of each Ethernet packet,

3. a callback function entering the packet processing, firstly analyzing the packet structure, analyzing the outer Mac header and the type according to the structure characteristics of the ptn signal, identifying whether the next layer contains a Vlan structure according to the type, if so, analyzing 4 bytes of the Vlan, obtaining the outer Vlan id, if not, directly obtaining an mpls label structure, wherein 4 bytes of the mpls label structure are a unit and a multi-layer 4-byte structure is possible, therefore, the last mpls label structure is identified by the stack top and stack bottom bit of the mpls label, each mpls label analyzes the corresponding label value, if the carried label value is 14, the last mpls label structure is the control frame of the Mac OAM, analyzing the field values corresponding to the mpls and lspid according to the OAM control frame structure, if not, the field value carried behind the mpls label is the inner Mac header, and determining whether there is a layer of Vlan destination after Mac source Mac and dstpe, if yes, analyzing the inner layer vlan ID, judging whether the carried Ethernet data is an ARP packet or a CDP packet or an IP packet according to the type value, after distinguishing the specific packet types, the ARP packet can analyze the IP and Mac of the sender and receiver carried in the Ethernet data according to the ARP related protocol standard, the CDP packet can analyze the TLV variable-length format according to the CDP related protocol standard to obtain the value of each type, such as information of equipment ID, address, port ID, platform and the like, the IP packet can carry IP quintuple information, namely source IP, destination IP, source port and destination port, and the information can be analyzed according to the structural characteristics of the IP packet,

4. if the Ethernet data packet is one of OAM, CDP, ARP and IP packet, then corresponding analysis operation is carried out, and the analysis content is put into a relational database for storage and convenient query and display,

5. and performing service protocol identification and statistics in data by using nDPI deep packet inspection, acquiring application layer protocol information and key load information to acquire user behavior, and inserting an analysis result into a database for storage.

Claims (1)

1. The utility model provides a network data platform based on X86 integrated circuit board, the external interface of integrated circuit board includes four 10 GE's optical port, two RG 45's net gape, a console debugging interface, HDMI access interface, two USB mouths, backplate possess 40G net gape, a giga net gape, but X86 integrated circuit board system start access display and mouse keyboard, look over system start-up state, its characterized in that: the backplane 40GE of the x86 board card is connected with the backplane 40GE of the exchange board to provide a data flow path, the 10GE optical port of the front panel of the x86 board card is interconnected with the 10GE optical port of the rear panel of the line card for bottom polling acquisition function, the control interface of the equipment is used for realizing interconnection of the backplane and the Ethernet, and the backplane gigabit network ports of all the board cards are communicated to realize control instruction interaction; the system is divided into a data acquisition module, a rule processing module and a data analysis module according to functional modules, the data acquisition module is in one-way connection with the data analysis module, the data acquisition module, the rule hit processing module and the data analysis module are sequentially connected, and a user data rule configuration and screening unit, a rule management unit and a rule filtering hit unit are sequentially connected in the rule processing module.

Images