CN102201948B

CN102201948B - Quick matching method for network intrusion detection system

Info

Publication number: CN102201948B
Application number: CN 201110139546
Authority: CN
Inventors: 张萌萌
Original assignee: North China University of Technology
Current assignee: Beijing University of Technology; North China University of Technology
Priority date: 2011-05-27
Filing date: 2011-05-27
Publication date: 2013-09-18
Anticipated expiration: 2031-05-27
Also published as: CN102201948A

Abstract

The invention provides a novel and different mode matching method, which is characterized in that useful loads of packets are divided into multiple blocks with a fixed length and each block is subjected to mode matching one by one; the blocks are compared for less times based on a fact: normal data streams in the network intrusion detection system are hardly matched with any virus signature, and most of keys required to compare are short; parallel detection for the multiple keys accelerates processing speed and reduces data dependency among instructions; and an algorithm is used as a Hash table for regular expression keys with a non-fixed length so as to avoid further detecting the most packets as far as possible. The algorithm requires less memory to store commonly-used data which are stored in a high-speed cache, thus a CPU (central processing unit) is not required to access a main memory at most of the time. The delayed time of the memory access is hundreds of times than the clock period of the CPU commonly; and due to the less memory is required for accessing, the algorithm accelerates the speed of the mode matching.

Description

The fast matching method that is used for Network Intrusion Detection System

Joint study

The application is by North China University of Tech and the joint study of information institute of Beijing Jiaotong University, and obtains following fund assistance: Beijing talent of institution of higher education directly under the jurisdiction of a municipal government teaches in-depth planning item (PHR201008187) by force; State natural sciences fund (No.60903066, No.60972085); Beijing's natural science fund (No.4102049), the beginning teacher of Ministry of Education fund (No.20090009120006).

Technical field

The present invention relates to a kind of Network Intrusion Detection System, more specifically, relate to a kind of Fast Match Algorithm for Network Intrusion Detection System.

Background technology

In 1998, Martin Mr. Roesch used the C language development intruding detection system Snort of open source code (" SNORT Network Intrusion Detection System ", http://www.snort.org).Until today, Snort has developed into the powerful network invasion monitoring/system of defense (NetworkIntrusion Detection/Prevention System) of characteristics such as multi-platform, a real-time traffic analysis, network IP packet record, i.e. NIDS/NIPS.Snort meets general public permission (GPL-GUN General Pubic License), can obtain Snort by free download on the net, and only needs a few minutes just can install and bring into use its .snort based on libpcap.

Network Intrusion Detection System (NIDS) is a kind of important fail safe instrument for network manager's protecting network; it can allow the network manager monitor network by checking monitoring real time data bag and the malicious attack that detects such as refusal is attacked in unwarranted visit, TCP and denial of service (DoS).NIDS service regeulations (or signature) whether classify to packet, be malice with the specified data bag by database.A kind of common methods of searching for matched rule effectively is to be the given chart of rule set foundation such as rule tree or finite automata, and to use packet to come it is traveled through as input of character string.Because flow and Cyberthreat is growing, the intrusion detection consumes resources more that becomes, now express network and large-scale rule set in addition again, a Network Intrusion Detection System can take CPU plenty of time and internal memory.Therefore, the pattern matching engine of high-throughput and be the key of whole intrusion detection performance to the reduction of internal memory access frequency.

Except the preliminary treatment of data packet header, all be based on payload inspection in the many important service of current network.NIDS, traffic monitoring and the 7th layer filter the Accurate Analysis that requires packet content, with comprise application program, virus, concrete one such as protocol definition is predefined data packet matched.Traditionally, these data sets are made of a series of digital signature, these digital signature can be searched by string matching, security signature has been designated as the accurate coupling based on character string, but definite string matching is not enough to find the malice pattern, so more expressive regular expression (regulation expressoin) is used to describe the digital signature of various payload now.

For example, in Linux application protocol grader, all protocol identifiers all are represented as regular expression.Similarly, the Network Intrusion Detection System Snort of open source code (" SNORT NetworkIntrusion Detection System ", http://www.snort.org) do not have the state of regular expression rule set from April, 2003, develop in 8786 kinds of rules November in 2010 5549 kinds and contain and have the compatible regular expression (PCRE) of a Perl at least.Another intruding detection system Bro that increases income also uses regular expressions and is its pattern language.These all are applied in fire compartment wall and the equipment by different manufacturers, for example Cisco System Co..

Packet content scanning is vital for network security and network monitoring.Modern network equipment need be carried out the service that the depth data bag detects to realize fail safe and special purpose at high speed.The Boyer-Moore string is searched algorithm---and this is a kind of especially effectively string search algorithm, and be used for actual character string search document benchmark---the packet that is widely used in the degree of depth detects, but there are two important deficiencies in it: first, owing to need carry out preliminary treatment and can not search for a plurality of keys simultaneously each target string (key), its speed be fast inadequately really; The second, it only carries out accurate string matching.Recently, finite automata (FA) is most popular for the method that realizes the regular expression coupling, but they need complicated pretreatment to constitute FA, and needs a large amount of internal memories.

Non deterministic finite automaton (NFA) needs more state conversion for each character, and therefore the time complexity of searching is O (m), and wherein m represents the number of states among the NFA.On the other hand, NFA also is extremely effectively structure of aspect, space.Yet deterministic finite automaton (DFA) only needs a state traversal to each character, but they need a great amount of ram for present regular expression set.When checking the payload of a particular data packet, only need to consider a very little subclass of regular collection, so DFA can waste the internal memory of the overwhelming majority.Based on above reason, these solutions are in the deep-packet detection of reality and inapplicable, because exigent speed during their online handle packet.

In order to accelerate the speed of pattern matching, having proposed some hardware solutions (for example is suggested, C.H.Lin, C.T.Huang, C.P.Jiang and S.C.Chang, " Optimization of PatternMatching Circuits for Regular Expression on FPGA ", IEEE Transactions onVery Large Scale Integration Systems, Vol.15, Iss.12, pp.1303-1310, in October, 2007; W.Zhang, Y.Xue, D.S.Wang, T.Song, " A multiple simple regularexpression matching architecture and coprocessor for deep packet inspection ", Asia-Pacific Conference on Computer Systems Architecture, pp.1-8,2008; P.Marco, B.Ivano, S.Marco D., " ReCPU:A parallel and pipelined architecturefor regular expression matching ", IFIP International Conference on Very LargeScale Integration, pp.19-24.2007 October; I.Bonesana, M.Paolieri, M.D.Santambrogio, " An adaptable FPGA-based System for Regular ExpressionMatching ", Design, Automation and Test in Europe, DATE ' 08, pp.1262-1267, in March, 2008; N.Yamagaki, R.Sidhu, S.Kamiya, " High-speedregular expression matching engine using multi-character NFA ", InternationalConference on Field Programmable Logic and Applications, pp.131-136, in September, 2008), they are based on finite automata (FA) mostly.

Therefore, the current existence all the time for the demand that can improve the Network Intrusion Detection System of intrusion detection speed by minimizing cpu clock cycle holding time and access to content number of times.

Summary of the invention

According to present know-how, this paper proposes a method for mode matching innovation, different: we are divided into the piece of a plurality of regular lengths to the payload of packet, one by one to each piece execution pattern coupling.Only need the comparison of lesser amt for each piece, this is based on such fact: normal data flow is complementary with any virus signature hardly in the Network Intrusion Detection System, and most of key that we need compare is all very short.Processing speed has been accelerated in parallel inspection to a plurality of keys, reduces data dependence between instruction.For the regular expression key of on-fixed length, our algorithm is used as a Hash table and avoids further inspection to most data packets as far as possible.Our algorithm only needs a spot of internal memory to store the data of frequent use, and these data are stored in the high-speed cache (Cache), so the CPU overwhelming majority time does not need to visit main memory.The time that internal storage access postpones normally hundreds of times to the cpu clock cycle owing to need seldom internal storage access, our algorithm has been accelerated the speed of pattern matching.

According to an embodiment, a kind of quick character string matching method based on the Snort intruding detection system wherein, is divided into the Payload of packet the piece of a plurality of identical sizes, the length of each piece is the L byte, and the window that 3 length of needs are the L-bit records temporary transient result.The window that just is used for recording checked result is called as CW, the window that is used for recording the previous result who produces is called PW, be used for recording by current block and produce and the window that is used for the result of next piece coupling is known as NW, this method may further comprise the steps: (1) is for first piece, all set the bit in these three windows for 0, and a counter " CNT " is initialized as L; (2) if CNT is L, L copy of first character of current block and key is carried out NXOR (XNOR) computing, and the result of this L-bit is stored among the CW; If not, L copy of the character late of current block and key carried out the NXOR computing, carry out the AND computing with CW subsequently, and the result is stored to CW; (3) if this is last character in the key, and new CW is not 0, then just found coupling; If not, then entered for the 4th step; (4) if PW and CW are 0, then will { CW, NW} CNT the bit that move right be copied to NW PW then, and CNT is set is L and move to next piece and proceed to step 2 further to check; (5) if PW all is 0, and CW also is 0 entirely except minimum effective bit, then will { CW, NW} CNT the bit that move right is copied to NW PW then, CNT is arranged to L and moves to next piece and proceeds to step 2 further to check; (6) if PW is not 0, perhaps CW is not 0 or 1, then will { PW, CW, NW} 1 bit that moves right, and the CNT number subtracted 1; (7) get back to step 2.

Description of drawings

Fig. 1 is a kind of simple tree structure that uses in Snort;

Fig. 2 is the quick structure of string matching according to an embodiment of the invention; And

Fig. 3 is the quick schematic diagram of the example of character string matching method according to an embodiment of the invention.

Embodiment

NIDS and Snort

Snort (" SNORT Network Intrusion Detection System ", http://www.snort.org) uses a kind of simple language to come definition rule, in order to describe network behavior.Every kind of rule is made of five necessary fields and a large amount of Optional Fields.Necessary field comprises that (TCP for example, UDP), source/purpose IP address and port numbers, all these is the part of a data packet head to protocol type.Snort is interpreted as " Optional Field " to the keyword in the bracket.Optional Field commonly used is: " content " (payload of search data bag is to search designated mode), and " msg " (message that will send when packet produces an event is set), etc.Be used for checking that whether a packet is with rough power mode search rule storehouse with a kind of direct method that arbitrary rule is complementary: one by one test each rule at this packet.This method is easy to operation, but relatively expends time in.In order to reduce the regular quantity that will check, the tree structure that Snort has set up " rule tree " by name stores and organizes all rules, sees Fig. 1.For each rule, necessary field store is at rule tree node (RTN), and Optional Field is stored in optional tree node (OTN).OTN is associated with corresponding RTN.If a plurality of rules have identical necessary field, then only create single RTN, a plurality of OTN can share this single RTN.The detection engine of Snort has made up the index that is used for source port field and destination interface field, so that the rule of fast access TCP and UDP.Use source port number and the destination port number of each input packet to search for its index then, so that the identification matched rule.If found matched rule in index structure, then Snort execution character string pattern between the Payload of the rule of mating and input packet is mated.If character string pattern matching success then checks all remaining necessary fields of storing (protocol type, TCP for example, UDP, source/purpose IP address) and optional condition in rule tree.Yet, can spend more time and resource at the character string pattern matching of payload.

Therefore, in order to reduce the quantity at the character string pattern matching of payload, the present invention will check all protocol fields, and be not only port numbers, and like this Snort just can refuse the more data bag and not enter the character string pattern matching stage.Since current index be based on in the data packet head be necessary what field checked, detect engine only to the packet execution character string pattern coupling of less data in the entire packet, this is vital for the NIDS that detects packet with linear speed.

The related work of regular expression coupling

Quick regular expression matching algorithm of the present invention is absorbed in the Payload execution pattern coupling to packet.When the whole necessary field in a kind of rule and the packet header was all mated, pattern matching engine of the present invention mated at Payload execution character string pattern, and key is stored in the corresponding OTN set.

Traditional DFA based on NIDS has three main restrictions: at first, a fact is arranged, and they do not take full advantage of, that be exactly normal data flow almost with the neither coupling of any virus signature; The second, DFA is extremely inadequate for a lot of part signature match; The 3rd, when rule set upgraded, DFA need go to rebuild for a long time.The mechanism that the present invention proposes can solve these drawbacks, can carry out NIDS more efficiently.

When need be to the character string pattern matching of Payload, only need to check a spot of password.Most typical is on microprocessor, and by carry out the password coupling to get off: at first will convert given key to corresponding NFA or DFA, NFA or DFA can be used to search for the input text character then.When a DFA can be in a constant time handles each character (, its time that needs is O (1)), for a n character key, the DFA amount of state can reach O (2n), and this can reduce the performance of computer in some cases significantly.Some new methods are come the big DFA of construction with all keys, in order to check whole keys concurrently; These new methods can check a character at every turn, but all need accessed content ninety-nine times out of a hundred, and this will increase the stand-by period of search.Unfortunately, most packets hardly with any cipher key match, so we wish to get rid of those unmatched packets as soon as possible, and can check the Payload of a packet concurrently, rather than check a character at every turn.

Quick regular expression matching process

Provide according to an embodiment of the invention regular expression matching process fast below with reference to Fig. 2-3.

At first, consider accurate string matching.We are divided into the piece of identical size to the Payload of packet, and the length of supposing each piece is the L byte, and the window that to need 3 length be the L-bit records temporary transient result.The window that just is used for recording checked result is called as CW (Current windows), the window that is used for recording the previous result who produces is called as PW (Previous windows), is used for recording by current block producing and the window that is used for the result of next piece coupling is known as NW (Next windows).Figure 2 illustrates the structure of string matching engine according to an embodiment of the invention.

According to an embodiment, engine shown in Figure 2 comes each piece is checked by carrying out following steps step by step:

(1) for first piece, all sets the bit in these three windows for 0, and a counter " CNT " is initialized as L.

(2) if CNT is L, L copy of first character of current block and key is carried out NXOR (XNOR) computing, and the result of this L-bit is stored among the CW; If not, L copy of the character late of current block and key carried out the NXOR computing, carry out the AND computing with CW subsequently, and the result is stored to CW.

(3) if this is last character in the key, and new CW is not 0, then just found coupling; If not, then entered for the 4th step.

(4) if PW and CW are 0, will make up then that { CW, NW} CNT the bit that move right is copied to PW with NW then, and CNT is set is L and move to next piece and proceed to step 2 further to check.

(5) if PW all is 0, and CW also is 0 except minimum effective bit entirely, will make up then that { CW, NW} CNT the bit that move right is copied to PW with NW then, CNT is arranged to L and moves to next piece and proceeds to step 2 further to check.

(6) if PW is not 0, perhaps CW is not 0 or 1, then will make up PW, CW, NW} 1 bit that moves right, and the CNT number subtracted 1.

(7) get back to step 2.

In Fig. 3, provided an instantiation of each step more than using.In example, L=8, key=" key ", piece 1 and piece 2 comprise character string " Tmnorrkeyinokruk ".To (c), move to a new piece from (b), and { CW, NW} 7 bits (CNT=7) that move right.Some windows in Fig. 3, have been omitted.If the content of window is " 00000000 ", then in (e), found coupling.

Above steps only is subclass of method of the present invention, because we do not consider the asterisk wildcard in the regular expression, and we suppose that the length of key is less than or equal to the length of piece.These two problems will further be discussed in following content.From the basis of theory and practice, we can draw following results:

If a. piece does not comprise first character of key, then only need to carry out once checking for this piece, and experimental result shows that in the time of L=8, about 83% piece only need be carried out 1 to 3 inspection.

B. the length of piece is more long, will obtain more good performance, because the number of comparisons of a piece only depends on the longest coupling in this piece and previous.Therefore, the number of comparisons in a bigger combination block roughly is equal to the number of comparisons in the longest number of times at those smaller pieces.

C. this method does not need preliminary treatment, and owing to does not need to preserve or recover too many data and access memory seldom.

D. we can check a plurality of keys simultaneously.One by one use piece of cipher key match, and the interim result that only need preserve each key, these interim results can be stored in register or the high-speed cache to avoid internal storage access consuming time.This also can reduce the data dependence in the execution process instruction of single key.

We need consider the worst situation, although it seems and can not take place.In the worst case, each piece all needs to carry out L time relatively with a key, but has only a specific key energy and the worst this situation to be complementary.Even need more times relatively, method of the present invention still can reach very high packet throughput, because compare (it often needs tens cpu clock cycles) with internal storage access, these simple instructions have the shorter stand-by period (maximum tens cpu clock cycles).We after will inquire into the worst this situation.

Special circumstances

Above method is based on certain hypothesis, but needs taking everything into consideration in practice, so we discuss these special circumstances.

1. long key problem

If the length of a key, then can be used double window (or many windows) greater than the length of piece, handle and computing with similar method.To those skilled in the art, be easy to the embodiment of above-mentioned single CW is extended to the design of double window or many windows.In fact, we are divided into length with a long key is several sections of the L byte, and respectively they and a piece is compared, and then resulting a plurality of results is passed to the comparison of next piece.

2. inaccuracy character

In recent rule set, some characters are not " accurately ", but the length of these characters is accurate, for example "! A " and " a|b ", it represents " not being character a " and " character a or character b " respectively.For these characters, we only need to be modified in the computing of using in the comparison.For example, we can use XOR (XOR) computing replace the NXOR computing represent "! A ", use (NXORa) | (NXOR b) represents " a | b ".

3. elongated key

Have length variable in some regular expressions, for example crin star (Kleene Star) (*), it is an asterisk wildcard, usually is used for representing to mate zero in the regular expression or more times appearance.Therefore, when the character of random amount was arranged in the key, we had two kinds of methods to handle.

First method, we can seek the prefix of regular length of key as a cryptographic Hash, and as a new key, some packets will be excluded with this cryptographic Hash.This is because if the part of the Payload of a packet and key does not match, and then certain and whole key does not match.And we also can prove the accurate prefix that can find at least one character length in any regular expression, because necessarily can not come mark with the crin star at first character of each regular expression.For example, regular expression " a*bc+d " is equivalent to " bc+d ", because " a ^*" be insignificant, and we only need inspection " bc " (it is maximum accurate prefix) as cryptographic Hash, to carry out the string matching to the Payload of packet.After finding first coupling, just need carry out the matching check in a nearlyer step.But, we do not need to check whole regular expressions, only need to check its remainder, have so just reduced the state of DFA.Therefore we are the simply constructed DFA of remainder in the regular expression, and use the result who is produced by this cryptographic Hash further to check.When being effective, the only further character string that checks after the cryptographic Hash of mating of need, and the state in DFA finishes this possible matching check.Therefore when the Payload of packet needs further to check, have only its sub-fraction to enter DFA.

We can not set up several little DFA, merge all these elongated keys but can select to set up a bigger DFA engine.But experimental result shows, this method is not as the former, this be because: it is faster than setting up a big combination DFA that (1) sets up several little DFA; (2) after having checked prefix, only need further to check the sub-fraction of packet Payload, and only have the sub-fraction of packet Payload to enter DFA; (3) some little DFA can be reused by other packets, also are easy to be deposited in the buffer memory.

Second method, usage counter rather than DFA.But this method has been avoided use DFA need have been carried out preliminary treatment to regular expression.Preprocessor is divided into several sections with regular expression, is used for distinguishing part and the adjustable length part that length is determined.For example, " abc+d " can be divided into three parts: " abc ", " c ^*" and " d ".The method of using when checking " abc " with " d " is identical with discussion before, and unique difference is how to check " c ^*".After having checked character string " abc ", our matching engine inspection " c ^*" coupling, difference is that the result of NXOR operation is stored in the interim window (TW), rather than upgrades when front window (CW), and three windows (PW, CW and NW) remain unchanged in this matching process.Subsequently, matching engine uses the NXOR computing to come compare string " d ", and its result is designated as T.I byte among the CW can utilize following equation to upgrade:

CW(i)＝T(i)&(CW(i)|(TW(i+1)&CW(i+1))|(TW(i+2)&TW(i+1)&CW(i+2))|...|(TW(L-1)&...&TW(i+1)&CW(L-1)))(1)

If regular expression has requirement to the length of character " c ", then we need usage counter to be recorded in the length of " c " between " abc " and " d ".For example to character " c " coupling N time or more times coupling.

Comparison with Boyer-Moore character string search method

Boyer-Moore string searching method is a kind of especially effectively string search algorithm, and it is a standard basis at actual characters string searching method.Usually, Snort uses Boyer-Moore character string search method to finish pattern matching, but this algorithm need carry out preliminary treatment and only check a character at every turn each key, even can skip some characters in the Payload.In addition, this algorithm becomes faster along with the elongated of the length of the key of searching for, but in the rule of Snort, most key is all very short.

Compare with the Boyer-Moore string method of searching, our algorithm has following advantage:

1. this algorithm does not need each key preliminary treatment.

2. the length that moves of packet Payload is (piece) fixed, but in Boyer Moore algorithm, the check result before the length that the packet Payload moves depends on..

3. above two " static state " characteristics make easier hardware development, for example FPGA of being applied to of method of the present invention.

4. the packet movable length is fixed but not the characteristics of assay before depending on, makes that method of the present invention is easier to check a plurality of keys concurrently.

5. this algorithm uses parallel mode to check a plurality of characters, rather than one by one inspection.

6. this algorithm is easier to expansion.

For Boyer Moore algorithm, the worst case of seeking all couplings in the Payload needs about 3*N time to compare, and wherein N is the quantity of character in the Payload.In the method for the invention, the worst situation needs about N time to compare.Under worst case, the length of key and the equal in length of piece.

Although aforementioned open file has been discussed exemplary arrangement and/or embodiment, it should be noted that under situation about not deviating from by the scope of the scheme of the description of claims definition and/or embodiment, can make many variations and modification at this.And, although describe or the described scheme of requirement and/or the key element of embodiment with singulative, also it is contemplated that the situation of plural number, unless clearly represented to be limited to odd number.In addition, all or part of of scheme and/or embodiment can be used in combination with any other scheme and/or all or part of of embodiment arbitrarily, unless shown different.

Claims

1. quick character string matching method based on the Snort Network Intrusion Detection System, wherein, the Payload of packet is divided into the piece of a plurality of identical sizes, the length of each piece is the L byte, and the window that 3 length of needs are the L-bit records temporary transient result, the window that just is used for recording checked result is called as CW, the window that is used for recording the previous result who produces is called PW, be used for recording by current block and produce and the window that is used for the result of next piece coupling is known as NW, this method may further comprise the steps:

(1) for first piece, all sets the bit in these three windows for 0, and a counter " CNT " is initialized as L;

(2) if CNT is L, L copy of first character of current block and key is carried out XNOR NXOR computing, and the result of this L-bit is stored among the CW; If not, L copy of the character late of current block and key carried out the NXOR computing, carry out the AND computing with CW subsequently, and the result is stored to CW;

(3) if this is last character in the key, and new CW is not 0, then just found coupling; If not, then entered for the 4th step;

(4) if PW and CW are 0, then will { CW, NW} CNT the bit that move right be copied to NW PW then, and CNT is set is L and move to next piece and proceed to step 2 further to check;

(5) if PW all is 0, and CW also is 0 entirely except minimum effective bit, then will { CW, NW} CNT the bit that move right is copied to NW PW then, CNT is arranged to L and moves to next piece and proceeds to step 2 further to check;

(6) if PW is not 0, perhaps CW is not 0 or 1, then will { PW, CW, NW} 1 bit that moves right, and the CNT number subtracted 1;

(7) get back to step 2.

2. the method for claim 1, wherein use double window or many windows to substitute described CW, PW and NW.

3. the method for claim 1, wherein for the key that comprises the inaccuracy character, can use XOR XOR computing to replace the NXOR computing.

4. the method for claim 1, wherein for elongated key, seek the prefix of regular length of key as a cryptographic Hash,, as a new key remainder of this key is further checked with this cryptographic Hash.