EP2215565B1 - Deterministic finite automata (dfa) graph compression - Google Patents

Deterministic finite automata (dfa) graph compression Download PDF

Info

Publication number
EP2215565B1
EP2215565B1 EP08854609.8A EP08854609A EP2215565B1 EP 2215565 B1 EP2215565 B1 EP 2215565B1 EP 08854609 A EP08854609 A EP 08854609A EP 2215565 B1 EP2215565 B1 EP 2215565B1
Authority
EP
European Patent Office
Prior art keywords
node
graph
nodes
character
arcs
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP08854609.8A
Other languages
German (de)
French (fr)
Other versions
EP2215565A1 (en
Inventor
Rajan Goyal
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cavium LLC
Original Assignee
Cavium LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US11/986,970 priority Critical patent/US8180803B2/en
Application filed by Cavium LLC filed Critical Cavium LLC
Priority to PCT/US2008/011543 priority patent/WO2009070191A1/en
Publication of EP2215565A1 publication Critical patent/EP2215565A1/en
Application granted granted Critical
Publication of EP2215565B1 publication Critical patent/EP2215565B1/en
Application status is Active legal-status Critical
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/903Querying
    • G06F16/90335Query processing
    • G06F16/90344Query processing by using string matching techniques

Description

    BACKGROUND
  • The Open Systems Interconnection (OSI) Reference Model defines seven network protocol layers (L1-L7) used to communicate over a transmission medium. The upper layers (L4-L7) represent end-to-end communications and the lower layers (L1-L3) represent local communications.
  • Networking application aware systems need to process, filter and switch a range of L3 to L7 network protocol layers, for example, L7 network protocol layers such as, HyperText Transfer Protocol (HTTP) and Simple Mail Transfer Protocol (SMTP), and L4 network protocol layers such as Transmission Control Protocol (TCP). In addition to processing the network protocol layers, the networking application aware systems need to simultaneously secure these protocols with access and content based security through L4-L7 network protocol layers including Firewall, Virtual Private Network (VPN), Secure Sockets Layer (SSL), Intrusion Detection System (IDS), Internet Protocol Security (IPSec), Anti-Virus (AV) and Anti-Spam functionality at wire-speed.
  • Network processors are available for high-throughput L2 and L3 network protocol processing, that is, performing packet processing to forward packets at wire-speed. Typically, a general purpose processor is used to process L4-L7 network protocols that require more intelligent processing. Although a general purpose processor can perform the compute intensive tasks, it does not provide sufficient performance to process the data so that it can be forwarded at wire-speed.
  • Content aware networking requires inspection of the contents of packets at "wire speed." The content may be analyzed to determine whether there has been a security breach or an intrusion. A large number of patterns and rules in the form of regular expressions are applied to ensure that all security breaches or intrusions are detected. A regular expression is a compact method for describing a pattern in a string of characters. The simplest pattern matched by a regular expression is a single character or string of characters, for example, IcI or /cat/. The regular expression also includes operators and meta-characters that have a special meaning.
  • Through the use of meta-characters, the regular expression can be used for more complicated searches such as, "abc*xyz". That is, find the string "abc", followed by the string "xyz", with an unlimited number of characters in-between "abc" and "xyz". Another example is the regular expression "abc??abc*xyz;" that is, find the string "abc," followed two characters later by the string "abc" and an unlimited number of characters later by the string "xyz." An Intrusion Detection System (IDS) application inspects the contents of all individual packets flowing through a network, and identifies suspicious patterns that may indicate an attempt to break into or compromise a system. One example of a suspicious pattern may be a particular text string in a packet followed 100 characters later by another particular text string. Content searching is typically performed using a search algorithm such as, Deterministic Finite Automata (DFA) to process the regular expression. The DFA processes an input stream of characters sequentially using a DFA graph and makes a state transition based on the current character and state.
  • WO 2004/013777 relates to systems and methods for creating a finite state automata that matches patterns in parallel including the steps of creating states of the automata from a set of patterns to be matched and passing over the patterns a second time adding transitions to the states to match all the possible patterns that can start within the pattern.
  • US 2004/0176945 relates to a finite state transducer generator that includes a recursive transition network creating part that creates a recursive transition network, an arc replacement part that recursively repeats an operation where an arc in a finite state transducer is replaced by a network corresponding to an input label of the arc by a network in the recursive transition network, and a priority calculating part that calculates an arc replacement priority based on statistical information regarding frequency of applying grammar rules.
  • DACIUK J: "Experiments with automata compression", XP002508980, provides an overview of DFA graph compression compression.
  • SUMMARY
  • Aspects of the present invention are set forth in the accompanying independent claims.
  • Typically, a design trade-off must be made in the implementation of DFA graph search systems. The trade-off is between creating a compressed DFA graph which may decrease complexity of the graph and may comprise a lower memory requirement for storage, but may require multiple (more than one) external memory accesses for each input character processed. Alternatively, a larger and more complex DFA graph may be utilized that may require only a single memory access per character, but at the cost of a large memory requirement for storage. The tradeoff is between graph size (memory) requirement and processing time.
  • In an embodiment of the present invention, a compressed DFA graph is presented which achieves the best of both worlds (i.e., provides a reduced graph size and a lesser processing time). The compressed DFA graph will not only comprises a low memory requirement, but may also require only one memory access for the processing of each input character.
  • In an embodiment of the present invention a processor and corresponding method for traversing a compressed DFA is presented. The processor may comprise a memory unit configured to store a graph having a plurality of nodes interconnected through valid arcs. The processor may also comprise a walker configured to walk the plurality of nodes in the graph to search for an expression in an input stream by utilizing a hash value to manage a retrieval function of the walker. The graph may be a compressed graph including no redundant arcs.
  • The walker may be configured to travel node to node through a valid arc, where a current valid arc of a current node leads to a next node, the current valid arc representing a character match in the expression. The walker may be further configured to employ a hash function to generate a hash value associated with a next input character, the walker process may be configured to index into the next node with the generated hash value, to read a next valid arc, the next valid arc including a copy of a next character. The hash function may be a common hash function for the entire graph and stored in graph metadata. Each valid arc may also comprise a hash function associated with the next node it points to.
  • The walker may also be configured to compare the next input character with the copy of the next character, where the next input character matches a next character of the expression when a comparison match is detected. The walker may be configured to traverse the graph to a designated node when a comparison match is not detected. The walker may also be configured to traverse the graph to a designated node associated with a current sub-graph when a comparison match is not detected.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The foregoing will be apparent from the following more particular description of example embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention.
    • FIGS. 1A and 1B are block diagrams of a security appliance including a network services processor and a protocol processor, respectively;
    • FIG. 2 is a block diagram of the network services processor shown in FIG. 1A;
    • FIG. 3 is a block diagram illustrating content search elements used by the processor of FIGS. 1A and 1B;
    • FIG. 4 is a block diagram of an example data structure that is used by the Content Search Mechanism (CSM) to traverse a graph;
    • FIGS. 5A and 6 are examples of DFA graphs;
    • FIGS. 5B(1) - 5B(3) are block diagrams of example data structures of the nodes of FIG. 5A;
    • FIGS. 5C is an example of pseudo code used to control a walker process traversing the DFA graph of FIGS. 5B(1) - 5B(3);
    • FIG. 7 is block diagram of an example data structure of the nodes of FIG. 5A upon compression;
    • FIG. 8 is an illustrative example of DFA graph traversing; and
    • FIG. 9 is an example of pseudo code used to control a walker process of the example of FIG. 8.
    DETAILED DESCRIPTION
  • Fig. 1 A is a block diagram of an example security appliance 102 including a network services processor 100. The security appliance 102 may be a standalone system that may switch packets received at one Ethernet port (Gig E) to another Ethernet port (Gig E) and perform a plurality of security functions on received packets prior to forwarding the packets. For example, the security appliance 102 may be used to perform security processing on packets received on a Wide Area Network prior to forwarding the processed packets to a Local Area Network.
  • The network services processor 100 processes Open System Interconnection network L2-L7 layer protocols encapsulated in received packets. As is well-known to those skilled in the art, the Open System Interconnection (OSI) reference model defines seven network protocol layers (L1-7). The physical layer (L1) represents the actual interface, electrical and physical that connects a device to a transmission medium. The data link layer (L2) performs data framing. The network layer (L3) formats the data into packets. The transport layer (L4) handles end to end transport. The session layer (L5) manages communications between devices, for example, whether communication is half-duplex or full-duplex. The presentation layer (L6) manages data formatting and presentation, for example, syntax, control codes, special graphics and character sets. The application layer (L7) permits communication between users, for example, file transfer and electronic mail.
  • The network services processor 100 may schedule and queue work (packet processing operations) for upper level network protocols, for example L4-L7, and allow processing of upper level network protocols in received packets to be performed to forward packets at wire-speed. Wire-speed is the rate of data transfer of the network over which data is transmitted and received. By processing the protocols to forward the packets at wire-speed, the network services processor does not slow down the network data transfer rate.
  • The network services processor 100 may include a plurality of Ethernet Media Access Control interfaces with standard Reduced Gigabyte Media Independent Interface (RGMII) connections to the off-chip PHYs 104a, 104b.
  • The network services processor 100 may also receive packets from the Ethernet ports (Gig E) through the physical interfaces PHY 104a, 104b, and perform L7-L2 network protocol processing on the received packets and forwards processed packets through the physical interfaces 104a, 104b to another hop in the network or the final destination or through the PCI bus 106 for further processing by a host processor. The network protocol processing may include processing of network security protocols such as Firewall, Application Firewall, Virtual Private Network (VPN) including IP Security (IPSec) and/or Secure Sockets Layer (SSL), Intrusion Detection System (IDS) and Anti-virus (AV).
  • The network services processor 100 may also include a low latency memory controller for controlling low latency Dynamic Random Access Memory (DRAM) 118. The low latency DRAM 118 may be used for Internet Services and Security applications allowing fast lookups, including the string-matching that may be required for Intrusion Detection System (IDS) or Anti Virus (AV) applications and other applications that require string matching.
  • The network services processor 100 may perform pattern search, regular expression processing, content validation, transformation and security accelerate packet processing according to an embodiment of the present invention. The regular expression processing and pattern search may be used to perform string matching for AV and IDS applications and other applications that require string matching.
  • A DRAM controller in the network services processor 100 may control access to an external Dynamic Random Access Memory (DRAM) 108 that is coupled to the network services processor 100. The DRAM 108 may store data packets received from the PHYs interfaces 104a, 104b or the Peripheral Component Interconnect Extended (PCI-X) interface 106 for processing by the network services processor 100. In one embodiment, the DRAM interface supports 64 or 128 bit Double Data Rate II Synchronous Dynamic Random Access Memory (DDR II SDRAM) operating up to 800MHz. The DRAM may also store rules data required for lookup and pattern matching in DFA graph expression searches.
  • A boot bus 110 may provide the necessary boot code which may be stored in flash memory 112 and may be executed by the network services processor 100 when the network services processor 100 is powered-on or reset. Application code may also be loaded into the network services processor 100 over the boot bus 110, from a device 114 implementing the Compact Flash standard, or from another high-volume device, which can be a disk, attached via the PCI bus.
  • The miscellaneous I/O interface 116 offers auxiliary interfaces such as General Purpose Input/Output (GPIO), Flash, IEEE 802 two-wire Management Interface (MDIO), Universal Asynchronous Receiver-Transmitters (UARTs) and serial interfaces.
  • It should be appreciated that the example security appliance 102 may alternatively include a protocol processor 101 (Fig. 1B). The protocol processor 101 may include the element of the network services processor 100 with the addition of a content processing accelerator 107, connected to the processor 101 via the PCI/PCI-X connection 106, and an external DRAM 111 connected to the accelerator 107. The accelerator 107 and DRAM 111 may be employed in content search applications, therefore making all content searching operations external to the processor 101.
  • Fig. 2 is a block diagram of the network services processor 100, or the protocol processor 101 shown in Figs. 1A and 1B, respectively. The network services processor 100, and/or the protocol processor 101, delivers high application performance using a plurality of processors (cores) 202 located on a L1 network protocol. Network applications may be categorized into data plane and control plane operations. Each of the cores 202 may be dedicated to performing data plane or control plane operations. A data plane operation may include packet operations for forwarding packets. A control plane operation may include processing of portions of complex higher level protocols such as Internet Protocol Security (IPSec), Transmission Control Protocol (TCP) and Secure Sockets Layer (SSL). A data plane operation may include processing of other portions of these complex higher level protocols.
  • A packet may be received by any one of the interface units 210a, 210b through a SPI-4.2 or RGM II interface. A packet may also be received by the PCI interface 224. The interface unit 210a, 210b handles L2 network protocol pre-processing of the received packet by checking various fields in the L2 network protocol header included in the received packet. After the interface unit 210a, 210b has performed L2 network protocol processing, the packet is forwarded to the packet input unit 214. The packet input unit 214 may perform pre-processing of L3 and L4 network protocol headers included in the received packet. The pre-processing includes checksum checks for Transmission Control Protocol (TCP)/User Datagram Protocol (UDP) (L3 network protocols).
  • The packet input unit 214 may write packet data into buffers in Level 2 cache 212 or DRAM 108 in a format that is convenient to higher-layer software executed in at least one processor 202 for further processing of higher level network protocols. The packet input unit 214 may also support a programmable buffer size and can distribute packet data across multiple buffers to support large packet input sizes.
  • The Packet order/work (POW) module (unit) 228 may queue and schedule work (packet processing operations) for the processor 202. Work is defined to be any task to be performed by a processor that is identified by an entry on a work queue. The task can include packet processing operations, for example, packet processing operations for L4-L7 layers to be performed on a received packet identified by a work queue entry on a work queue. Each separate packet processing operation is a piece of the work to be performed by a processor on the received packet stored in memory (L2 cache memory 212 or DRAM 108). For example, the work may be the processing of a received Firewall/ Virtual Private Network (VPN) packet. The processing of a Firewall/VPN packet may include the following separate packet processing operations (pieces of work): (1) defragmentation to reorder fragments in the received packet; (2) IPSec decryption (3) IPSec encryption; and (4) Network Address Translation (NAT) or TCP sequence number adjustment prior to forwarding the packet.
  • The network services processor 100, and/or the protocol processor 101, may also include a memory subsystem. The memory subsystem may include level 1 data cache memory 204 in each processor 202, instruction cache in each processor 202, level 2 cache memory 212, a DRAM controller 216 for external DRAM memory and the interface 230 to external low latency memory 118. The memory subsystem is architected for multi-processor support and tuned to deliver both high-throughput and low-latency required by memory intensive content networking applications. Level 2 cache memory 212 and external DRAM memory 108 (Figs. 1A and 1B) may be shared by all of the processors 202 and I/O co-processor devices.
  • The network services processor 100, and/or the protocol processor 101, may also include application specific co-processors that offload the processors 202 so that the network services processor achieves high-throughput. The application specific co-processors include a DFA co-processor 244 that performs Deterministic Finite Automata (DFA) and a compression/decompression co-processor 208 that performs compression and decompression.
  • Each processor 202 may be a dual-issue, superscalar processor with instruction cache 206, Level 1 data cache 204, built-in hardware acceleration (crypto acceleration module) 200 for cryptography algorithms with direct access to low latency memory over the low latency memory bus 230. The low-latency direct-access path to low latency memory 118 bypasses the L2 cache memory 212 and can be directly accessed from both the processors (cores) 202 and a DFA co-processor 244. In one embodiment, the latency to access the low-latency memory is less than 40 milliseconds.
  • Prior to describing the operation of the content search macros used for regular expression processing and pattern search in further detail, the other modules in the network services processor 100 will be described. In an example, after the packet has been processed by the processors 202, a packet output unit (PKO) 218 reads the packet data from L2 cache or DRAM, performs L4 network protocol post-processing (e.g., generates a TCP/UDP checksum), forwards the packet through the interface unit 210a, 210b and frees the L2 cache 212 or DRAM 108 locations used to store the packet.
  • Each processor 202 is coupled to the L2 cache by a coherent memory bus 234. The coherent memory bus 234 is the communication channel for all memory and I/O transactions between the processors 202, an I/O Bridge (IOB) 232 and the Level 2 cache and controller 212.
  • A Free Pool Allocator (FPA) 236 maintains pools of pointers to free memory in level 2 cache memory 212 and DRAM 108. A bandwidth efficient (Last In First Out (LIFO)) stack is implemented for each free pointer pool. If a pool of pointers is too large to fit in the Free Pool Allocator (FPA) 236, the Free Pool Allocator (FPA) 236 builds a tree/list structure in level 2 cache 212 or DRAM 108 using freed memory in the pool of pointers to store additional pointers.
  • The I/O Bridge (IOB) 232 manages the overall protocol and arbitration and provides coherent I/O partitioning. The IOB 232 includes a bridge 238 and a Fetch and Add Unit (FAU) 240. The bridge 238 includes buffer queues for storing information to be transferred between the I/O bus, coherent memory bus, the packet input unit 214 and the packet output unit 218.
  • The Fetch and Add Unit (FAU) 240 is a 2KB register file supporting read, write, atomic fetch-and-add, and atomic update operations. The Fetch and Add Unit (FAU) 240 can be accessed from both the processors 202 and the packet output unit 218. The registers store highly-used values and thus reduce traffic to access these values. Registers in the FAU 240 are used to maintain lengths of the output queues that are used for forwarding processed packets through the packet output unit 218.
  • The PCI interface controller 224 has a DMA engine that allows the processors 202 to move data asynchronously between local memory in the network services processor and remote (PCI) memory in both directions.
  • Typically, content aware application processing utilizes a deterministic finite Automata (DFA) to recognize a pattern in the content of a received packet. The DFA is a finite state machine, that is, a model of computation including a set of states, a start state, an input alphabet (set of all possible symbols) and a transition function that maps input symbols and current states to a next state. Computation begins in the start state and changes to new states dependent on the transition function. The DFA is deterministic, that is, the behavior can be completely predicted from the input. The pattern is a finite number of strings of characters (symbols) to search for in the input stream (string of characters).
  • The pattern is commonly expressed using a regular expression that includes atomic elements, for example, normal text characters such as, A-Z, 0-9 and meta-characters such as, *, ^ and |. The atomic elements of a regular expression are the symbols (single characters) to be matched. These are combined with meta-characters that allow concatenation (+) alternation (1), and Kleene-star (*). The meta-character for concatenation is used to create multiple character matching patterns from a single character (or sub-strings) while the meta-character for alternation (|) is used to create a regular expression that can match any of two or more sub-strings. The meta-character Kleene-star (*) allows a pattern to match any number, including no occurrences of the preceding character or string of characters. Combining different operators and single characters allows complex expressions to be constructed. For example, the expression (th(is|at)*) will match the following character strings: th, this, that, thisis, thisat, thatis, or thatat.
  • The character class construct [...] allows listing of a list of characters to search for, e.g. gr[ea]y looks for both grey and gray. A dash indicates a range of characters, for example, [A-Z]. The meta-character "." matches any one character.
  • The input to the DFA state machine is typically a string of (8-bit) bytes, that is, the alphabet is a single byte (one character or symbol). Each byte in the input stream results in a transition from one state to another state.
  • The states and the transition functions can be represented by a graph, where each
    node in the graph represents a state and arcs in the graph represent state transitions. The current state of the state machine is represented by a node identifier that selects a particular graph node. The graph may be stored in low latency memory 118, or the main DRAM 108, and accessed by the processors 202 over the low latency bus. The processors 202 may access a DFA-based graph stored in the low latency memory, or the main DRAM 108, directly. The graph will be described later in conjunction with Fig. 5A.
  • Fig. 3 is a block diagram illustrating content search macros that may be used by a processor 202 in the network services processor 100 shown in Fig. 2. Content search macros 300 may include a walker software component (process) 302 for searching the DFA-based content search graph that may be generated via a compiler software component 304. The content search macros 300 may be stored in L2/DRAM (212, 108) and may be executed by a processor 202. The DFA-based content search graph may be stored in low latency memory 118 which is accessible directly by the processor 202 through the low latency bus and low-latency memory controller shown in Fig. 2. The compiler 304 translates expressions into a DFA-based content search graph with a plurality of nodes.
  • After the compiler 304 has generated the content search graph and the graph may be stored in low latency memory 118, the walker process 302 executed by one of the processors 202 walks input data (e.g., a string of characters) in the received data packet one character at a time and outputs a set of matches based on a search for a pattern in the input data using the content search graph.
  • Fig. 4 is a block diagram of an example of a typical data structure 400 that may be stored in Low Latency Memory Random Access Memory 118, or the main DRAM 108, and used by the Content Search Mechanism (CSM) executing in a processor 202 to traverse a graph. The data structure 400 may be generated by the compiler component 304 based on the expressions to be searched for in the input stream.
  • The data structure may include a plurality of nodes, for example nodes 402 and 404, that may be used in a content search graph according to an embodiment of the present invention. Each node in the graph may include an array of 256 next node pointers, one for each unique input byte value; that is, 28 (256 possible values, or 256 addresses) representing an ASCII value of the input. Each next node pointer contains a next node ID that directly specifies the next node/state for the input byte value.
  • As shown in Fig. 4, a current node 402 comprises 256 arcs. Each arc represents an input ASCII value. For example, in node 404, the arc addressed as '97' includes a next node pointer for the character 'a.' Similarly, a next node 404 also comprises 256 arcs, each arc comprising a unique address and including a next node pointer for a corresponding ASCII value.
  • The arcs of a node may be forward arcs (e.g., arcs which point to next nodes in the DFA graph), backward arcs (e.g., arcs which point back to a root node or a prior node), or repeating arcs (e.g., arcs which point back to the node to which they are associated with). Arc 408 of node 404 comprises a node pointer to node 404, and is therefore an example of a repeating arc. Arc 410 of node 404 comprises a next node pointer to node 402, which in this context is considered to be a prior node, and therefore arc 410 is an example of a backward arc. In the example provided by Fig. 4, the arc addressed as '66' of current node 402 comprises a forward next node pointer 406 pointing to next node 404, representing a character match of 'B' with the input stream. It should be appreciated that although Fig. 4 only shows 2 nodes, any number of nodes may be included in a DFA based content search graph.
  • Fig. 5A provides an example of a DFA graph 500 complied via the compiler 304. For simplicity, only forward arcs have been illustrated in the example graph 500. A table 504 illustrates all of the possible expression matches, and the corresponding nodal paths, for the example DFA graph 500. Figs. 5B(1)-5B(3) illustrate an example of the memory representation of the nodes in the uncompressed DFA graph 500, of Fig. 5A. Table 551 illustrates the relationship between the node characters and their corresponding ASCII values. Fig. 5C illustrates an example of pseudo code that may be employed by the walker process in traversing the uncompressed DFA graph 500.
  • In Figs. 5B(1)-5B(3), the node marked as '0' is the root node and is a starting position for traversing the graph with the walker process 302. Each of the nodes are interconnected through arcs represented by the lines connecting each node. Each of the nodes '0'- '16' includes 256 arcs, stored in memory, indexed from 0 to 255, as shown to the left of each node. For the sake of clarity, the arcs in all the nodes which point to Node 0 (i.e. root node) are shown empty in Fig 5B(1)-5B(3). For example, out of 256 arcs in Node 1, arc at index 79 point to next node 2 (as shown in Fig sub(1)) and rest of 255 arcs contain next node pointer pointing to Node 0. As shown in the pseudo code of Fig. 5C, the node structure may be defined 505 as a vector including 256 sections, each section corresponding to a unique arc in the node. An arc structure 507 may also be defined. In the present example provided by Fig. 5C, the arc structure is defined to be 32 bits in size including flags, arc pointers, and reserved space. In Figs. 5B(1)- 5B(3) the index values are displayed on the left-hand side of each node for illustrative purposes, these index values are not stored in memory. Each arc comprised in the nodes may be accessed using the ASCII value, corresponding to the index values shown in Figs. 5B(1)-5B(3), of a character under review, where each access requires an access into memory.
  • The arcs shown in Fig. 5A are forward arcs representing a character match between the expression being searched and an input character. Of the nodes shown in Figs. 5B(1)-5B(3), nodes'0,' '1,' '2,' '3,' '5,' '7,' '8,' '9,' '11,' and '14' each include only one forward valid arc (e.g., an arc that provides a forward next node pointer to a node other than the root node). Nodes '4,' '6,' and '12' each include two forward valid arcs.
  • The nodes comprising a double line, as shown in Fig. 5A, (e.g., nodes 3, 9, 10, 13, 15, and 16) are referred to as mark nodes and represent a string match in the input stream. For example, the double line around node 3 represents a string match of 'CON'; node 9 represents a string match of 'CONTENT'; node 10 represents a string match of 'CONTEXT'; node 13 represents a string match of 'CONTINUE'; node 15 represents a string match of 'CONTINUUM'; and node 16 representing a string match of 'CONTENTS.' Nodes '10,' '13,' '15,' and '16' are leaf nodes. A leaf node is a node that is located at the end of a DFA branch and has each of its arcs pointing back to the root node. A leaf node is also typically a mark node representing the end of an expression match. The walker process may identify a leaf or mark node by a flag setting included in the node data structure. As an example, the flag in an arc may indicate that a next node is marked node. Therefore, the arc pointing to Node 3 (from Node 2 for character 'N') may have a flag setting indicating that the next node (which is Node 3) is marked node representing an expression match. If the flag setting indicates that a mark or leaf node has been traversed, the walker process may report that a match in the expression has been found.
  • Referring to Figs. 5B(1)-5B(3), in operation, the walker process 302 may evaluate the input stream one byte at a time. As an example, consider the input stream 502. Initially, the walker process 302 may assign the root node pointer to a current node pointer variable 'node_ptr,' which will be evaluated (Fig. 5C, line 1). The walker 302 evaluates the first character of the input stream 502 which is 'B,' according to the pseudo code, the ASCII value of the first character may be assigned to the character variable 'ch' (Fig. 5C, line 2). The walker then proceeds to the root node to access the next node pointer associated with the character 'B.' The walker process may use the variable 'ch' to index in to the current node, which in the present example is the root node '0' (Fig. 5C, line 4). In the example provided by the DFA graph 500, as shown in Fig. 5B(1), the root node only includes a forward match for the character 'C.' In the case that a forward valid arc pointing to a mark node is detected, a flag may be set 507 indicating that an expression match has been found. In example embodiments of the present invention, the walker process may monitor flag settings to determine if a mark node has been traversed and if an expression match has been found (Fig. 5C, line 5).
  • In the current example, the arc associated with the character 'B' is a repeating arc (not shown). The presence of the repeating arc will not cause the flag 507 to be set and therefore, the walker process 302 may assigned the next node pointer variable 'node_ptr' to the arc associated with the character 'B' resulting in the walker process 302 to continue evaluating the root node '0' (Fig. 5C, line 6).
  • The walker process 302 then proceeds to the next character in the input stream 502 which is 'C,' and reassigns the character variable 'ch' to the ASCII value of the current character 'C' (Fig. 5C, lines 7 and 2). Upon locating the arc associated with the ASCII value of the character 'C,' the walker 302 finds a next node pointer providing a forward match and pointing to node '1' (Fig. 5C, line 4). Since, node '1' is not a mark node, the flag status will remain unchanged (Fig. 5C, line 5). Thereafter, the walker process 302 intakes the next input stream character 'O,' and proceeds to find the associated arc and next node pointer providing a forward match and leading to node '3' (Fig. 5C, lines 6, 7, and 2-4). Since node '3' is a mark node, a flag 507 may be set causing the walker process 302 to register that an expression match for the string 'CON' in the input stream has been found (Fig. 5C, line 5).
  • Depending on the specific application, the walker process 302 may proceed to evaluate the next character in the input stream 502, which is the character 'W' (Fig. 5C, lines 6, 7, and 2-5). The arc in node '3' associated with the character 'W' comprises a backward next node pointer to the root node '0,' the only forward match associated with node '3' is for the character 'T.' Thus, the walker process 302 follows the next node pointer associated with the character 'W' and proceeds to the root node '0' (Fig. 5C, lines 6 and 7). Thereafter, the walker process 302 proceeds to evaluate the next character in the input stream 502, which is 'X' and reassigns the character variable 'ch' equal to the ASCII value associated with the character 'X' (Fig. 5C, line 2).
  • Upon evaluating the next input character 'X,' it is determined that the arc associated with the character 'X' in the root node '0' includes a repeating next node pointer pointing to the root node '0' (Fig. 5C, line 4). Thus, the walker process 302 will proceed to evaluate the next character in the input stream. Following the same logic discussed above, the walker process 302 may proceed to find an expression match for the string 'CONTENT' in mark node '9.' Upon reading the next character 'J,' the walker process traverses back to the root node '0' and the arc and next node pointer associated with the character 'J' are read from the root node '0.' Upon detecting a repeating arc and reaching the end of the input stream 502, the walker process 302 completes its walking of the DFA graph 500 (Fig. 5C, lines 3 and 8).
  • The examples provided by Figs. 5A and 5B(1)-5B(3) illustrate a simplified DFA graph. Fig. 6 provides an illustration of a more realistic view of only a subportion of a DFA graph. Again, the backward arcs to the root node of each node have not been shown for simplicity. The DFA graph 600 comprises a number of branches which are interconnected. A DFA graph such as the one shown in Fig. 6 requires a substantial amount of memory to store, as well as a significant amount of time and system resources to traverse the graph.
  • Typically, a design trade-off must be made in the implementation of DFA graph search systems. The trade-off is between creating a compressed DFA graph which may decrease complexity of the graph and may comprise a lower memory requirement for storage, but may require two to three memory accesses for each input character. Alternatively, a larger and more complex DFA graph may be utilized that may require only a single memory access per input character, but at the cost of a large memory requirement for storage.
  • In an embodiment of the present invention, a compressed DFA graph is presented. The compressed DFA graph that has the advantages of low memory utilization and single memory access per the reading of each input character. In an embodiment of the present invention, all arcs which point back to the root node, or a designated node, are deleted from all other nodes and are not stored in memory as part of graph. Fig. 7 provides an illustrative example of the nodes of a compressed version of the DFA graph using this embodiment for uncompressed graphs shown in Figs. 5A and 5B(1)-5B(3). In contrast to the nodes of Figs. 5B(1)-5B(3), the nodes of Fig. 7 include only valid arcs (e.g., arcs which represent a character match in an expression). Furthermore, the valid arcs included in the compressed nodes comprises a copy of the character the arc is associated with as well as a next node pointer, as is shown in Fig. 7.
  • In a compressed node, all arcs which are not associated with a expression match are pruned or deleted. Nodes'0,' '1,' '2,' '3,' '5,' '7,' '8,' '9,' '11,' and 14' of Fig. 5B each included 256 arcs with only one valid arc. The same nodes, as compressed in Fig. 7, include only one arc, which is indexed as '0' as shown to the left of each node. Similarly, the uncompressed nodes '4,' '6,' and '12' of Fig. 5B included 256 arcs with only two arc being valid. The compressed versions of nodes '4,' '6,' and '12' of Fig. 7 include only two arcs, which are valid arcs indexed as '0' and '1.' The leaf nodes '10,' '13,' '15,' and '16' have been pruned or deleted in the compressed DFA graph. Arcs in compressed nodes which originally pointed to leaf nodes in the uncompressed DFA graph (e.g., nodes '8,' '9,' and '14') may include a form of identification (e.g., a special pointer 'Spl ptr') that may be used by the walker process as an indication the searching for a particular expression has ended.
  • In another embodiment of present invention, a set of designated nodes (including at least 1 node) may be identified. All the arcs of all nodes, which point to same next node for corresponding character as (one of) the designated node does, are pruned and not stored in Nodes. Taking the DFA graph of Fig. 6 as an example, if node '13' was assigned as a designated node, all of the arcs pointing to node '8' for the character 'y' are pruned (e.g., the arcs associated with the character 'y' from nodes '3,' '6,' '7,' '15,' '18,' '19,' '17,' and '21'; with the exception of the designated node '13'), all the arcs pointing to node '11' for the character '3' are deleted (e.g., the arcs associated with the character '3' from nodes '4,' '10,' '12,' '16,' '17,' '18,' '20,' and '21'; with the exception of the designated nod '13'), all arcs pointing to node '17' for the character 'a' are deleted (e.g., the arcs associated with the character 'a' from nodes '18,' and '21'; with the exception of the designated node '13'), and all the arcs pointing to node '18' for the character 'b' (e.g., the arc associated with the character 'b' from node '21'; with the exception of the designated node '13') are deleted as well. The deleted arcs in each node leave holes in the node data structure. These holes may then be compressed in order to save memory and required storage space.
  • As mentioned before, there may be multiple designated nodes which may allow further arc pruning. There may be different designated nodes for different sub-sections of the DFA graph. Pruned nodes may contain information about the designated node used for pruning. This information may also be stored in arc pointing to pruned node.
  • Graph compression using example embodiments of the present invention may greatly reduce memory requirements and processing speeds associated with the DFA graph. For example, in Intrusion Detections Systems (IDS) applications, it is possible to achieve a reduction in size of the DFA graph of at least 20X or greater. In Anti-Virus systems (AV), even greater DFA graph reduction may be obtained. The amount of graph reduction is highly application and signatures (i.e. patterns used to compile graph) dependent, but the insights used for this invention are fundamental to any DFA graph.
  • Due to the compression of memory to account for the holes, it is no longer possible to read the address of an arc via indexing an associated character's ASCII value. Therefore, in an embodiment of the present invention, a method for traversing a graph featuring compressed nodes and arcs is presented.
  • The walker process may be configured such that if a valid arc has not been found, the walker may default and traverse to a designated node. The address of the designated node may be included in metadata which may be stored in memory. It should be appreciated that the designated node need not be the root node and may instead be any node in the DFA graph.
  • Fig. 8 provides an illustrative example of how a walker process 302 may traverse a compressed graph (e.g., a graph with redundant arcs removed). Fig. 9 provides an example of pseudo code that may be used to manage the walker process 302 in a compressed DFA graph. In the example provided by Fig. 8, the root node '0' has been assigned by the compiler as the designated node. It should be appreciated that any node in the compressed DFA graph may be assigned as the designated node, and it should be further appreciated, as mentioned above, that the DFA graph may include any number of designated nodes.
  • In the DFA graph 800, each valid arc of a prior node '#-1' includes the following three items: (1) a copy of a character associated with an expression character match to a current node '#'; (2) a hash function 'H#' associated with a current node '#' which generates a hash value used to index into the current node '#' in order to find a next arc associated with a next node '#+1' and a next character; and (3) a node pointer pointing to the current node '#.' In contrast, the arcs in the DFA graphs of Figs. 5A, 5C(1)-5C(3), and 6 only include item (3). The arc structure may also be defined to include a flag setting or status, as well as reserved memory as shown in 905. Depending on the implementation, an arc size may typically be 64 or 32 bits. It should be appreciated that any other arc size may be employed.
  • In operation, the walker process 302 may initially assign the next node pointer variable 'node_ptr' to point to the root node '0' and may also assign the hash function variable 'hash_fn' to the hash function associated with the root node '0' (Fig. 9, lines 1 and 2). Consider the input stream 802, the walker process 302 analyzes the first byte of the input stream 802 which is the character 'a' and assigns the ASCII value associated with the character 'a' to the character variable 'ch' (Fig. 9, line 3). In the case that the root node is compressed, the walker process 302 may utilize the hash function, using the variable 'ch,' associated with the root node in order to find the valid arc associated with the character 'a.' The hash function may produce a hash value (i0a) that may be used to index in the root node '0' in order to find the arc associated with the character 'a' (Fig. 9, line 5).
  • A hash function is deterministic, therefore if two hash values generated from the same hash function are different, the two input values to the hash function must also be different. Based on this logic, a hash function never provides a false negative (e.g., the hash value of a character associated with a valid arc in a current node always correctly indexes to the arc), but however the hash function may provide a false positive (e.g., in the occurrence that input character is not associated with any valid arc, the hash function still provides an index to one of the arcs in the current node). Therefore, the walker process 302 may be configured to test hash values to ensure a false positive has not occurred. Another property of hash function is that there is no collision (i.e. no two input values generate same hash value).
  • Once the associated arc is obtained, the character copy included in the arc may be extracted and assigned to the variable 'arc_ch' (Fig. 9, line 6). The walker process 302 compares the ASCII values of the character copy 'arc_ch' with the current input character 'ch.' If the two ASCII values are not equal, the walker process 302 recognizes the current arc as a false positive. If there is no valid arc for this particular character (i.e., the arc associated with the character has been pruned), the walker process may traverse to the designated node and walk to the node where the designated node points to for this particular character.
  • In the current example, since the root, or designated, node '0' does include a valid arc for the character 'a,' the walker process will recognize the arc as being valid. Thereafter, the walker process 302 may evaluate flag settings in order to determine if the current node is a mark node (Fig. 9, line 9). The walker process 302 may proceed to assign the node pointer variable 'node_ptr' to the valid arc associated with the character 'a' and may also assign the hash function variable 'hash_fn' to the hash function associated with the next node to which the valid arc points (Fig. 9, lines 10-12). It should be appreciated that the root node may not be compressed, therefore the walker process 302 may index into node '0' in the usual manner explained in relation to Figs. 5A, 5B(1)-5B(3), and 5C. It should also be appreciated that the hash function associated with the root node may be stored as part of the graph metadata.
  • The walker process 302 proceeds to evaluate the next input stream 802 character, which is 'b,' and assigns the input variable 'ch' to the corresponding ASCII value (Fig. 9, line 3). Using the hash function included in the valid arc associated with the character 'a,' a hash value (i1b) is generated and used to index into the current node '1' (Fig. 9, line 5).
  • Since the character 'b' is associated with a valid arc in the current node '1,' the hash value i1b will index into node '1' and obtain the valid arc associated with the character 'b' as shown in Fig. 8 (Fig. 9, line 5). Upon obtaining the valid arc, the walker process 302 then utilizes a compare function to verify that the ASCII value of the character 'b' associated with the valid arc is the same as the ASCII value of the input stream character being evaluated 'b' to account for false positives (Fig. 9, lines 6 and 7). The comparison function then returns a positive match since the arc character and input character are equal.
  • Upon obtaining the positive match and checking the flag status, the walker process 302 proceeds to reassign the next node pointer and hash function variables and evaluates the next input stream byte, which is the character 'c'(Fig. 9, lines 8-11 and 3). The hash function associated with the 'b' arc, H2, then generates a hash value using the character 'c.' The generated hash value (i2c) is used to index into node '2' in order to find the valid arc associated with the character 'c' (Fig. 9, line 5). Upon obtaining the valid arc associated with the character 'c,' the walker process 302 again utilizes the comparison function to determine if the current input stream character 'c' is the same as the character associated with the valid arc which was just indexed. Since the two characters are indeed the same, the walker process 302 will receive a positive match notification from the comparison function, and therefore proceeds to generate a hash value with the next input character, 'f' (Fig. 9, lines 6-12 and 3-5).
  • Upon obtaining a hash value (i3f) from the valid arc associated with the character 'c,' the walker process 302 indexes into node '3.' In the example provided by Fig. 8, node '3' comprises two valid arcs, one associated with the character 'd' and the other associated with the character 'e.' In this situation, the hash function (H3) provides an index to either one of the valid arcs of node '3.' Upon using the comparison function, a negative match result is reported. The walker process 302 then proceeds to the designated node (root node '0') and searches for a valid arc associated with the character 'f' (Fig. 9, 6-8 and 1-2). Alternatively, if the designated node (in this case the root node) is not compressed, an associated hash function may be used to find the valid arc associated with the character 'f.' Since such an arc does not exist, the walker process will remain at the designated node, or root node '0,' and evaluate the next input character 'a' (Fig. 9, line 3). It should be appreciated that a single hash function may be utilized for the entire DFA graph, instead of a hash function being associated with each node. The single, or general, hash function may be stored in the graph metadata. It should also be appreciated that the comparison function may be a string character comparison function configured to compare character values rather than ASCII values. It should also be appreciated that any form of comparison may be employed.
  • The hash walker process described above only makes an access to memory while indexing into a node in order to obtain a valid arc. Therefore, at most, only one memory access is needed for each character in the input stream. Conventional walking processes typically require two or three memory accesses. Thus, the compressed arc and node graph, and the hash walking process may be utilized to reduce the number of memory accesses as well as reduce the amount of memory needed to store the DFA graph.
  • It should be appreciated that other methods of managing the walker process 302 may be utilized to traverse the compressed node and arc DFA graph. For example, the walker process 302 may be configured to read each arc in a node one by one until a valid arc for the character in question has been found.
  • Alternatively, in a single memory access, the walker processor 302 may be configured to retrieve a number of arcs (e.g., the walker process may retrieve five arcs in a single memory access) and utilize the comparison function on all of the retrieved arcs. If a positive string match is not obtained, the walker process may make another memory access to retrieve a second grouping of arcs, and so on until a positive match has been found or until all of the arcs in the node have been evaluated.
  • In another embodiment, each node may comprise a directory, or location table, generated by the compiler such that upon retrieving the location table from memory, the walker process 302 need only find the entry of the character in question in order to find the address of the associated arc. If the associated arc does not exist, the walker process may default to a root or designated node.
  • Those of ordinary skill in the art should recognize that methods involved in providing the example embodiments described above may be embodied in a computer program product that includes a computer usable medium. For example, such a computer usable medium can include a readable memory device, such as a solid state memory device, a hard drive device, a CD-ROM, a DVD-ROM, or a computer diskette, having stored computer-readable program code segments. The computer readable medium can also include a communications or transmission medium, such as electromagnetic signals propagating on a computer network, a bus or a communications link, either optical, wired, or wireless, carrying program code segments as digital or analog data signals. The program code enables and supports computer implementation of the operations described in FIGS. 1A-9 or any other described embodiments.
  • While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claims (12)

  1. A computer implemented method for generating a deterministic finite automata-based graph comprising:
    storing a graph of expressions in memory, the graph including a plurality of nodes interconnected by arcs;
    assigning at least one node of the plurality of nodes as a designated node;
    pruning arcs associated with respective nodes of the plurality of nodes that interconnect to a same next node as its designated node for a same character, wherein pruned arcs in each of the respective nodes leaves holes in a corresponding data structure; and
    compressing the holes, wherein pruned nodes are the respective nodes for which arcs have been pruned and the pruned nodes contain information about the designated node used for the pruning to enable a walker process to traverse to the designated node in an event an address of an arc cannot be read via indexing in the corresponding node data structure due to the compressing of the holes.
  2. The method of Claim 1 further comprising:
    assigning the at least one designated node to be associated with at least one sub-section of the graph.
  3. The method of Claim 1 further comprising:
    traversing the compressed graph with a walker process managed by a hash function.
  4. The method of Claim 1 further comprising:
    traversing the compressed graph with a walker process configured to obtain at least two arcs from memory during a single access.
  5. The method of Claim 1 further comprising:
    traversing the compressed graph with a walker process configured to obtain an arc based on a location table.
  6. The method of Claim 1 wherein a particular valid arc of the current node comprises a copy of the character associated with the particular node.
  7. An apparatus comprising:
    a memory unit configured to store a graph of expressions including a plurality of nodes interconnected by arcs; and
    a processor configured to assign at least one node of the plurality of nodes as a designated node and to prune arcs associated with respective nodes of the plurality of nodes that interconnect to a same next node as its designated node for a same character, wherein pruned arcs in each of the respective nodes leave holes in a corresponding node data structure, the processor further configured to compress the holes, wherein pruned nodes are the respective nodes for which arcs have been pruned and the pruned nodes contain information about the designated node used for pruning to enable a walker process to traverse to the designated node in an event an address of an arc cannot be read via indexing in the corresponding node data structure due to the compressing of the holes.
  8. The apparatus of Claim 7 wherein the processor is further configured to assign the at least one designated node to be associated with at least one sub-section of the graph.
  9. The apparatus of Claim 7 further comprising:
    a walking unit configured to traverse the graph, the walking unit managed by a hash function.
  10. The apparatus of Claim 7 further comprising:
    a walking unit configured to traverse the graph by obtaining at least two arcs from memory during a single access.
  11. The apparatus of Claim 7 further comprising:
    a walking unit configured to traverse the graph by obtaining an arc based on a location table.
  12. The apparatus of Claim 7 wherein a particular valid arc of the current node comprises a copy of the character associated with the particular node.
EP08854609.8A 2007-11-27 2008-10-07 Deterministic finite automata (dfa) graph compression Active EP2215565B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US11/986,970 US8180803B2 (en) 2007-11-27 2007-11-27 Deterministic finite automata (DFA) graph compression
PCT/US2008/011543 WO2009070191A1 (en) 2007-11-27 2008-10-07 Deterministic finite automata (dfa) graph compression

Publications (2)

Publication Number Publication Date
EP2215565A1 EP2215565A1 (en) 2010-08-11
EP2215565B1 true EP2215565B1 (en) 2016-05-04

Family

ID=40052880

Family Applications (1)

Application Number Title Priority Date Filing Date
EP08854609.8A Active EP2215565B1 (en) 2007-11-27 2008-10-07 Deterministic finite automata (dfa) graph compression

Country Status (3)

Country Link
US (1) US8180803B2 (en)
EP (1) EP2215565B1 (en)
WO (1) WO2009070191A1 (en)

Families Citing this family (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8301788B2 (en) * 2004-09-10 2012-10-30 Cavium, Inc. Deterministic finite automata (DFA) instruction
US8560475B2 (en) * 2004-09-10 2013-10-15 Cavium, Inc. Content search mechanism that uses a deterministic finite automata (DFA) graph, a DFA state machine, and a walker process
US8392590B2 (en) * 2004-09-10 2013-03-05 Cavium, Inc. Deterministic finite automata (DFA) processing
US8819217B2 (en) * 2007-11-01 2014-08-26 Cavium, Inc. Intelligent graph walking
US8086609B2 (en) 2007-11-01 2011-12-27 Cavium, Inc. Graph caching
US8180803B2 (en) 2007-11-27 2012-05-15 Cavium, Inc. Deterministic finite automata (DFA) graph compression
US7949683B2 (en) * 2007-11-27 2011-05-24 Cavium Networks, Inc. Method and apparatus for traversing a compressed deterministic finite automata (DFA) graph
US8473523B2 (en) 2008-10-31 2013-06-25 Cavium, Inc. Deterministic finite automata graph traversal with nodal bit mapping
US8442931B2 (en) * 2008-12-01 2013-05-14 The Boeing Company Graph-based data search
US20110016154A1 (en) * 2009-07-17 2011-01-20 Rajan Goyal Profile-based and dictionary based graph caching
US8659604B2 (en) 2009-09-29 2014-02-25 International Business Machines Corporation Dimensional reduction mechanisms for representing massive communication network graphs for structural queries
US8566344B2 (en) * 2009-10-17 2013-10-22 Polytechnic Institute Of New York University Determining whether an input string matches at least one regular expression using lookahead finite automata based regular expression detection
US9398033B2 (en) 2011-02-25 2016-07-19 Cavium, Inc. Regular expression processing automaton
US8990259B2 (en) 2011-06-24 2015-03-24 Cavium, Inc. Anchored patterns
US9858051B2 (en) * 2011-06-24 2018-01-02 Cavium, Inc. Regex compiler
US8954700B2 (en) 2011-08-02 2015-02-10 Cavium, Inc. Method and apparatus for managing processing thread migration between clusters within a processor
US9203805B2 (en) * 2011-11-23 2015-12-01 Cavium, Inc. Reverse NFA generation and processing
US9064032B2 (en) * 2012-10-05 2015-06-23 Intel Corporation Blended match mode DFA scanning
US9426165B2 (en) 2013-08-30 2016-08-23 Cavium, Inc. Method and apparatus for compilation of finite automata
US9507563B2 (en) 2013-08-30 2016-11-29 Cavium, Inc. System and method to traverse a non-deterministic finite automata (NFA) graph generated for regular expression patterns with advanced features
US9426166B2 (en) 2013-08-30 2016-08-23 Cavium, Inc. Method and apparatus for processing finite automata
US9419943B2 (en) 2013-12-30 2016-08-16 Cavium, Inc. Method and apparatus for processing of finite automata
US9275336B2 (en) 2013-12-31 2016-03-01 Cavium, Inc. Method and system for skipping over group(s) of rules based on skip group rule
US9544402B2 (en) 2013-12-31 2017-01-10 Cavium, Inc. Multi-rule approach to encoding a group of rules
US9667446B2 (en) 2014-01-08 2017-05-30 Cavium, Inc. Condition code approach for comparing rule and packet data that are provided in portions
US9602532B2 (en) 2014-01-31 2017-03-21 Cavium, Inc. Method and apparatus for optimizing finite automata processing
US9904630B2 (en) 2014-01-31 2018-02-27 Cavium, Inc. Finite automata processing based on a top of stack (TOS) memory
US10110558B2 (en) 2014-04-14 2018-10-23 Cavium, Inc. Processing of finite automata based on memory hierarchy
US9438561B2 (en) 2014-04-14 2016-09-06 Cavium, Inc. Processing of finite automata based on a node cache
US10002326B2 (en) 2014-04-14 2018-06-19 Cavium, Inc. Compilation of finite automata based on memory hierarchy
CN103957012B (en) * 2014-04-18 2017-06-06 华为技术有限公司 A kind of compression method and device of DFA matrixes
US9916532B2 (en) 2014-06-09 2018-03-13 Cognitive Scale, Inc. Method for performing graph query operations within a cognitive environment
US10445317B2 (en) * 2014-06-09 2019-10-15 Cognitive Scale, Inc. Graph query engine for use within a cognitive environment
US10262077B2 (en) * 2014-06-27 2019-04-16 Intel Corporation Systems and methods for pattern matching and relationship discovery
US20170193376A1 (en) * 2016-01-06 2017-07-06 Amit Agarwal Area/energy complex regular expression pattern matching hardware filter based on truncated deterministic finite automata (dfa)
CN106980653B (en) * 2017-03-03 2019-07-12 清华大学 DFA compression method and device, regular expression matching method and system

Family Cites Families (60)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5327544A (en) * 1991-08-29 1994-07-05 At&T Bell Laboratories Method and apparatus for designing gateways for computer networks
US5230061A (en) * 1992-01-02 1993-07-20 The University Of Akron Clause counter map inference engine
US7301541B2 (en) * 1995-08-16 2007-11-27 Microunity Systems Engineering, Inc. Programmable processor and method with wide operations
US6192282B1 (en) * 1996-10-01 2001-02-20 Intelihome, Inc. Method and apparatus for improved building automation
US7975305B2 (en) * 1997-11-06 2011-07-05 Finjan, Inc. Method and system for adaptive rule-based content scanners for desktop computers
US6760833B1 (en) * 1997-08-01 2004-07-06 Micron Technology, Inc. Split embedded DRAM processor
US6076087A (en) * 1997-11-26 2000-06-13 At&T Corp Query evaluation on distributed semi-structured data
US6560680B2 (en) * 1998-01-21 2003-05-06 Micron Technology, Inc. System controller with Integrated low latency memory using non-cacheable memory physically distinct from main memory
US6047283A (en) * 1998-02-26 2000-04-04 Sap Aktiengesellschaft Fast string searching and indexing using a search tree having a plurality of linked nodes
US6578110B1 (en) * 1999-01-21 2003-06-10 Sony Computer Entertainment, Inc. High-speed processor system and cache memories with processing capabilities
US7188168B1 (en) * 1999-04-30 2007-03-06 Pmc-Sierra, Inc. Method and apparatus for grammatical packet classifier
US7185081B1 (en) * 1999-04-30 2007-02-27 Pmc-Sierra, Inc. Method and apparatus for programmable lexical packet classifier
US6493698B1 (en) * 1999-07-26 2002-12-10 Intel Corporation String search scheme in a distributed architecture
US6775757B1 (en) * 1999-12-14 2004-08-10 Genesis Microchip Inc. Multi-component processor
US6661794B1 (en) * 1999-12-29 2003-12-09 Intel Corporation Method and apparatus for gigabit packet assignment for multithreaded packet processing
US6925641B1 (en) * 2000-02-04 2005-08-02 Xronix Communications, Inc. Real time DSP load management system
US7046848B1 (en) * 2001-08-22 2006-05-16 Olcott Peter L Method and system for recognizing machine generated character glyphs and icons in graphic images
US20030110208A1 (en) * 2001-09-12 2003-06-12 Raqia Networks, Inc. Processing data across packet boundaries
US7240040B2 (en) * 2001-09-12 2007-07-03 Safenet, Inc. Method of generating of DFA state machine that groups transitions into classes in order to conserve memory
US6856981B2 (en) * 2001-09-12 2005-02-15 Safenet, Inc. High speed data stream pattern recognition
US7225188B1 (en) * 2002-02-13 2007-05-29 Cisco Technology, Inc. System and method for performing regular expression matching with high parallelism
JP4047053B2 (en) * 2002-04-16 2008-02-13 富士通株式会社 Retrieval apparatus and method using sequence pattern including repetition
US7093023B2 (en) * 2002-05-21 2006-08-15 Washington University Methods, systems, and devices using reprogrammable hardware for high-speed processing of streaming data to find a redefinable pattern and respond thereto
US20050108518A1 (en) * 2003-06-10 2005-05-19 Pandya Ashish A. Runtime adaptable security processor
US6952694B2 (en) * 2002-06-13 2005-10-04 Intel Corporation Full regular expression search of network traffic
WO2004013777A1 (en) 2002-08-05 2004-02-12 Fish Robert System and method of parallel pattern matching
US7711844B2 (en) * 2002-08-15 2010-05-04 Washington University Of St. Louis TCP-splitter: reliable packet monitoring methods and apparatus for high speed networks
US7119577B2 (en) * 2002-08-28 2006-10-10 Cisco Systems, Inc. Method and apparatus for efficient implementation and evaluation of state machines and programmable finite state automata
US7146643B2 (en) * 2002-10-29 2006-12-05 Lockheed Martin Corporation Intrusion detection accelerator
KR100558765B1 (en) * 2002-11-14 2006-03-10 한국과학기술원 Method for executing xml query using adaptive path index
US7085918B2 (en) * 2003-01-09 2006-08-01 Cisco Systems, Inc. Methods and apparatuses for evaluation of regular expressions of arbitrary size
US7464254B2 (en) * 2003-01-09 2008-12-09 Cisco Technology, Inc. Programmable processor apparatus integrating dedicated search registers and dedicated state machine registers with associated execution hardware to support rapid application of rulesets to data
US7689530B1 (en) * 2003-01-10 2010-03-30 Cisco Technology, Inc. DFA sequential matching of regular expression with divergent states
WO2004072797A2 (en) * 2003-02-07 2004-08-26 Safenet, Inc. System and method for determining the start of a match of a regular expression
EP1604277A2 (en) * 2003-02-28 2005-12-14 Lockheed Martin Corporation Hardware accelerator personality compiler
US7305372B2 (en) * 2003-03-04 2007-12-04 Kurzweil Technologies, Inc. Enhanced artificial intelligence language
JP2004271764A (en) * 2003-03-06 2004-09-30 Nagoya Industrial Science Research Inst Finite state transducer generator, program, recording medium, generation method, and gradual syntax analysis system
US7706378B2 (en) * 2003-03-13 2010-04-27 Sri International Method and apparatus for processing network packets
US7496892B2 (en) * 2003-05-06 2009-02-24 Andrew Nuss Polymorphic regular expressions
US7093231B2 (en) * 2003-05-06 2006-08-15 David H. Alderson Grammer for regular expressions
GB0315191D0 (en) * 2003-06-28 2003-08-06 Ibm Methods, apparatus and computer programs for visualization and management of data organisation within a data processing system
US20050138276A1 (en) * 2003-12-17 2005-06-23 Intel Corporation Methods and apparatus for high bandwidth random access using dynamic random access memory
US7586851B2 (en) * 2004-04-26 2009-09-08 Cisco Technology, Inc. Programmable packet parsing processor
US20050273450A1 (en) * 2004-05-21 2005-12-08 Mcmillen Robert J Regular expression acceleration engine and processing model
EP1607823A3 (en) 2004-06-14 2006-01-25 Lionic Corporation Method and system for virus detection based on finite automata
US8560475B2 (en) * 2004-09-10 2013-10-15 Cavium, Inc. Content search mechanism that uses a deterministic finite automata (DFA) graph, a DFA state machine, and a walker process
DK1794979T3 (en) * 2004-09-10 2017-07-24 Cavium Inc Selective copying of data structure
US8392590B2 (en) 2004-09-10 2013-03-05 Cavium, Inc. Deterministic finite automata (DFA) processing
US8301788B2 (en) * 2004-09-10 2012-10-30 Cavium, Inc. Deterministic finite automata (DFA) instruction
WO2006029508A1 (en) * 2004-09-13 2006-03-23 Solace Systems Inc. Highly scalable subscription matching for a content routing network
US7765183B2 (en) * 2005-04-23 2010-07-27 Cisco Technology, Inc Hierarchical tree of deterministic finite automata
US20070133593A1 (en) * 2005-11-21 2007-06-14 Udaya Shankara Searching Strings Representing a Regular Expression
US7685193B2 (en) * 2006-04-28 2010-03-23 Kognitio Limited Database system with multiple processing nodes
US7725510B2 (en) * 2006-08-01 2010-05-25 Alcatel-Lucent Usa Inc. Method and system for multi-character multi-pattern pattern matching
US7904961B2 (en) * 2007-04-20 2011-03-08 Juniper Networks, Inc. Network attack detection using partial deterministic finite automaton pattern matching
US7668802B2 (en) * 2007-07-30 2010-02-23 Alcatel Lucent Method and appliance for XML policy matching
US8819217B2 (en) * 2007-11-01 2014-08-26 Cavium, Inc. Intelligent graph walking
US7949683B2 (en) * 2007-11-27 2011-05-24 Cavium Networks, Inc. Method and apparatus for traversing a compressed deterministic finite automata (DFA) graph
US8180803B2 (en) 2007-11-27 2012-05-15 Cavium, Inc. Deterministic finite automata (DFA) graph compression
US8473523B2 (en) * 2008-10-31 2013-06-25 Cavium, Inc. Deterministic finite automata graph traversal with nodal bit mapping

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
GUILLAUME LAZZARA: "Automata and performances", TECHNICAL REPORT, no. 606, 1 June 2006 (2006-06-01), pages 1 - 130, XP007921508 *

Also Published As

Publication number Publication date
WO2009070191A1 (en) 2009-06-04
EP2215565A1 (en) 2010-08-11
US20090138494A1 (en) 2009-05-28
US8180803B2 (en) 2012-05-15

Similar Documents

Publication Publication Date Title
US7831606B2 (en) Signature search architecture for programmable intelligent search memory
US7512634B2 (en) Systems and methods for processing regular expressions
CN1249963C (en) Equipment and method for fast and self adapting processing block by using block digest information
EP1875390B1 (en) Hierarchical tree of deterministic finite automata
US7831822B2 (en) Real-time stateful packet inspection method and apparatus
Vasiliadis et al. Regular expression matching on graphics hardware for intrusion detection
US8861524B2 (en) Method for TCAM lookup using a key in multi-threaded packet processors
CN1561625B (en) Tagging packets with a lookup key to facilitate usage of a unified packet forwarding cache
Dharmapurikar et al. Fast and scalable pattern matching for content filtering
US6954789B2 (en) Method and apparatus for monitoring traffic in a network
US7240040B2 (en) Method of generating of DFA state machine that groups transitions into classes in order to conserve memory
RU2419986C2 (en) Combining multiline protocol accesses
CN100472480C (en) Packet processing method and device
US20070230445A1 (en) Integrated Circuit Apparatus And Method For High Throughput Signature Based Network Applications
US20100293345A1 (en) Network communications processor architecture with memory load balancing
EP1832037B1 (en) Template access control lists
US6742107B2 (en) Dynamically configured processing of composite stream input data using next conversion determining state transition table searched by converted input data
Mitra et al. Compiling pcre to fpga for accelerating snort ids
Bosshart et al. P4: Programming protocol-independent packet processors
Tan et al. Bit-split string-matching engines for intrusion detection and prevention
US20030051043A1 (en) High speed data stream pattern recognition
US9092471B2 (en) Rule parser
EP1392035A2 (en) Packet filtering method
JP3935880B2 (en) Hybrid search memory for network processors and computer systems
CN1886705B (en) Method and apparatus for efficient implementation and evaluation of state machines and programmable finite state automata

Legal Events

Date Code Title Description
AK Designated contracting states:

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

17P Request for examination filed

Effective date: 20100506

AX Request for extension of the european patent to

Extension state: AL BA MK RS

RIN1 Inventor (correction)

Inventor name: GOYAL, RAJAN

DAX Request for extension of the european patent (to any country) deleted
RAP1 Transfer of rights of an ep published application

Owner name: CAVIUM, INC.

RAP1 Transfer of rights of an ep published application

Owner name: CAVIUM, INC.

17Q First examination report

Effective date: 20130214

INTG Announcement of intention to grant

Effective date: 20151111

AK Designated contracting states:

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 797465

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160515

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602008044100

Country of ref document: DE

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20160504

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160804

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 797465

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160504

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160805

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160905

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602008044100

Country of ref document: DE

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

26N No opposition filed

Effective date: 20170207

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20161007

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161031

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161007

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161031

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161007

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161007

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20081007

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161031

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

PG25 Lapsed in a contracting state announced via postgrant inform. from nat. office to epo

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160504

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 11

PGFP Postgrant: annual fees paid to national office

Ref country code: FR

Payment date: 20180913

Year of fee payment: 11

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602008044100

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06F0017300000

Ipc: G06F0016000000

PGFP Postgrant: annual fees paid to national office

Ref country code: SE

Payment date: 20181011

Year of fee payment: 11

Ref country code: FI

Payment date: 20181009

Year of fee payment: 11

Ref country code: DE

Payment date: 20180925

Year of fee payment: 11