EP1579320A2 - Accelerateur d'analyseur materiel - Google Patents

Accelerateur d'analyseur materiel

Info

Publication number
EP1579320A2
EP1579320A2 EP03809942A EP03809942A EP1579320A2 EP 1579320 A2 EP1579320 A2 EP 1579320A2 EP 03809942 A EP03809942 A EP 03809942A EP 03809942 A EP03809942 A EP 03809942A EP 1579320 A2 EP1579320 A2 EP 1579320A2
Authority
EP
European Patent Office
Prior art keywords
state
character
state table
recited
pallette
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP03809942A
Other languages
German (de)
English (en)
Inventor
Michael C. Dapp
Eric C. Lett
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lockheed Martin Corp
Original Assignee
Lockheed Corp
Lockheed Martin Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US10/331,315 external-priority patent/US20040083466A1/en
Application filed by Lockheed Corp, Lockheed Martin Corp filed Critical Lockheed Corp
Publication of EP1579320A2 publication Critical patent/EP1579320A2/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/205—Parsing

Definitions

  • the present invention generally relates to processing of applications for controlling the operations of general purpose computers and, more particularly, to performing parsing operations on applications programs, documents and/or other logical sequences of network data packets.
  • certain character strings correspond to certain commands or identifications, including special characters and other important data (collectively referred to as control words) which allow data or operations to, in effect, identify themselves so that they may be thereafter treated as "objects" such that associated data and commands can be translated into the appropriate formats and commands of different applications in different languages in order to engender a degree of compatibility of respective connected platforms sufficient to support the desired processing at a given machine.
  • the detection of these character strings is performed by an operation known as parsing, similar to the more conventional usage of resolving the syntax of an expression, such as a sentence, into its component parts and describing them grammatically.
  • the conventional approach is to implement a table-based finite state machine (FSM) in software to search for these strings of interest .
  • the state table resides in memory and is designed to search for the specific patterns in the document.
  • the current state is used as the base address into the state table and the ASCII representation of the input character is an index into the table. For example, assume the state machine is in state 0
  • the absolute address for the state entry would be the sum/concatenation of the base address (state 0) and the index/ASCII character (02) .
  • the FSM begins with the CPU fetching the first character of the input document from memory.
  • the CPU then constructs the absolute address into the state table in memory corresponding ' to the initialized/current state and the input character and then fetches the state data from the state table.
  • the CPU updates the current state to the new value, if different (indicating that the character corresponds to the first character of a string of interest) and performs any other action indicated in the state data (e.g. issuing a token or an interrupt if the single character is a special character or if the current character is found, upon a further repetition of the foregoing, to be the last character of a string of interest) .
  • the above process is repeated and the state is changed as successive characters of a string of interest are found. That is, if the initial character is of interest as being the initial character of a string of interest, the state of the FSM can be advanced to a new state (e.g. from initial state 0 to state 1) . If the character is not of interest, the state machine would (generally) remain the same by specifying the same state (e.g. state 0) or not commanding a state update) in the state table entry that is returned from the state table address. Possible actions include, but are not limited to, setting interrupts, storing tokens and updating pointers. The process is then repeated with the following character.
  • the entire XMLTM or other language document is parsed character-by-character in the above-described manner.
  • the FSM steps through various states character-by- character until a string of interest is fully identified or a character inconsistent with a possible string of interest is encountered (e.g. when the string is completed/fully matched or a character deviates from a target string) .
  • no action is generally taken other than returning to the initial state or a state corresponding to the detection of an initial character of another target string.
  • the token is stored into memory along with the starting address in the input document and the length of the token.
  • the state table can provide multiple transitions from any given state. This approach allows the current character to be analyzed for multiple target strings at the same time while conveniently accommodating nested strings .
  • any process which requires an extreme amount of processing time at relatively high priority is, in some ways, similar to some characteristics of a denial-of-service (DOS) attack on the system or a node thereof or can be a tool that can be used in such an attack.
  • DOS denial-of-service
  • DOS attacks frequently present frivolous or malformed requests for service to a system for the purpose of maliciously consuming and eventually overloading available resources.
  • Proper configuration of hardware accelerators can greatly reduce or eliminate the potential for overloading of available resources.
  • systems often fail or expose security weaknesses when overloaded. Thus, eliminating overloads is an important security consideration.
  • a hardware parser accelerator including a document memory, a character pallette containing addresses corresponding to characters in the document, a state table containing a plurality of entries corresponding to a character, a next state pallette including a state address or offset, and a token buffer, wherein entries in said state table include at least one of an address into said next state pallette and a token.
  • Figure 1 is a representation of a portion of a state table used in parsing a document
  • FIG. 2 is a high level schematic diagram of the parser accelerator in accordance with the invention.
  • Figure 3 illustrates a preferred character palette format as depicted in Figure 2 .
  • FIGS 4A and 4B illustrate a state table format and a state table control register used in conjunction therewith in a preferred form of the invention as depicted in Figure 2,
  • Figure 5 illustrates a preferred next state palette format as depicted in Figure 2
  • Figure 6 is a preferred token format as depicted in Figure 5.
  • FIG. 1 a representation of a portion of a state table useful in understanding the invention. It should be understood that the state table shown in Figure 1 is potentially only a very small portion of a state table useful for parsing an XMLTM document and is intended to be exemplary in nature. It should be noted that an XMLTM document is used herein as an example of one type of logical data sequence which can be processed using an accelerator in accordance with the invention. Other logical data sequences can also be constructed from network data packet contents such as user terminal command strings intended for execution by shared server computers.
  • the state table is divided into an arbitrary number of rows, each having a base address corresponding to a state .
  • the rows of the base address are divided into a number of columns corresponding to the number of codes which may be used to represent characters in the document to be parsed; in this example, two hundred fifty-six (256) columns corresponding to a basic eight bit byte for a character which is used as an index into the state table.
  • an entry of "go to state 0" signifies detection of a character which distinguishes the string from any string of interest, regardless of how many matching characters have previously been detected and returns the parsing process to the initial/default state to begin searching for another string of interest . (For this reason, the "go to state 0" entry will generally be, by far, the most frequent or numerous entry in the state table.) Returning to state 0 may require the parsing operation to return to a character in the document subsequent to the character which began the string being followed at the time the distinguishing character was detected. 4. An entry including a command with "go to state 0 indicates completion of detection of a complete string of interest.
  • the command will be to store a token (with an address and length of the token) which thereafter allows the string to be treated as an object.
  • a command with "go to state n" provides for launching of an operation at an intermediate point while continuing to follow a string which could potentially match a string of interest. 5.
  • the character must be fetched from CPU memory
  • the state table entry must be fetched (again from CPU memory) and various pointers (e.g. to a character of the document and base address in the state table) and registers (e.g. to the initial matched character address and an accumulated length of the string) must be updated in sequential operations. Therefore, it can be readily appreciated that the parsing operation can consume large amounts of processing time.
  • Figure 2 A high-level schematic block diagram of the parser accelerator 100 in accordance with the invention is illustrated in Figure 2. As will be appreciated by those skilled in the art, Figure 2 can also be understood as a flow diagram illustrating the steps performed in accordance with the invention to perform parsing. As will be discussed in greater detail below in connection with
  • Figures 3, 4A, 4B, 5 and 6, the invention exploits some hardware economies in representing the state table such that a plurality of hardware pipelines are developed which operate essentially in parallel although slightly skewed in time.
  • the updating of pointers and registers can be performed substantially in parallel and concurrently with other operations while the time required for memory accesses is much reduced through both faster access hardware operated in parallel and prefetching from
  • the document such as an XMLTM document is stored externally in DRAM 120 which is indexed by registers 112, 114 and transferred by, preferably, thirty-two bit words to and input buffer 130 which serves as a multiplexer for the pipelines.
  • Each pipeline includes a copy of a character palette 140, state table 160 and a next state palette 170; each accommodating a compressed form of part of the state table.
  • the output of the next state palette 170 contains both the next state address portion of the address into entries in the state table 160 and the token value to be stored, if any.
  • Operations in the character palette 140 and the next state palette 170 are simple memory accesses into high speed internal SRAM which may be performed in parallel with each other as well as in parallel with simple memory accesses into the high speed external DRAM forming the state table 160 (which may also be implemented as a cache) . Therefore, only a relatively few clock cycles of the CPU initially controlling these hardware elements (but which, once started, can function autonomously with only occasional CPU memory operation calls to refresh the document data and to store tokens) are required for an evaluation of each character in the document .
  • the basic acceleration gain is the reduction of the sum of all memory operation durations per character in the CPU plus the CPU overhead to the duration of a single autonomously performed memory operation in high-speed SRAM or DRAM.
  • memory structures referred to herein as "external” is intended to connote a configuration of memories 120, 140, which is preferred by the inventors at the present time in view of the amount of storage required and access from the hardware parser accelerator and/or the host CPU.
  • No other connotation intended and a wide variety of hardware alternatives such as synchronous DRAM (SDRAM) will be recognized as suitable by those skilled in the art in view of this discussion.
  • SDRAM synchronous DRAM
  • Figure 3 illustrates the preferred form of a character palette which corresponds to the characters which are or may be included in the strings of interest .
  • This format preferably provides entries numbered 0 - 255, corresponding to the number of columns in the state table of Figure 1. (The term "palette” is used in much the same sense as in the term "color palette” containing data for each color supported and collectively referred to as a gamut .
  • a character referred to as a "null character" which does not result in a change of state can be expressed in one column of the state table rather than many such columns. It is desirable to test for a null character output at 144 which can substantially accelerate processing for parsing since it allows immediate processing of the next character without a further memory operation for state table access.
  • the format can be accommodated by a single register or memory locations configured as such by, for example, data in base address register 142 which points to a particular character palette (schematically illustrated by overlapping memory planes in Figure 2) .
  • the current eight bit character from the document e.g.
  • XMLTM document one of four provided from the input buffer 130 as received as a four byte word from the external DRAM 120, addresses an entry in the character palette which then outputs an address as an index or partial pointer into the state memory.
  • a palette in such a format a portion of the functionality of Figure 1 can be provided in the form of a single register of relatively limited capacity; thus allowing a plurality thereof to be formed and operated in parallel while maintaining substantial hardware economy and supporting others in the state table 160.
  • Figure 4A shows the preferred state table format which is constituted or -configured similarly to the character palette (e.g. substantially as a register) .
  • the principal difference from the character palette of Figure 3 is that the length of the register is dependent on the number of responses to characters desired and the number and length of strings of interest. Therefore, it is considered desirable to provide for the possibility of implementing this memory in CPU or other external DRAM (possibly with an internal or external cache) if the amount of internal memory which can be economically provided is insufficient in particular instances. Nevertheless, it is clear that a substantial hardware economy is provided since highly duplicative entries in the state table of Figure 1 can be reduced to a single entry; the address of which is accommodated by the data provided as described above in accordance with the character palette of Figure 3.
  • the output of the state table 160 is preferably one, two or four bits but provision for as much as thirty-two bits may provide increased flexibility, as will be discussed below in connection with Figure 4B. In any case, the output of the state table provides an address or pointer into the next state palette 170.
  • a preferred implementation feature of the invention includes a state table control register 162 which allows a further substantial hardware economy, particularly if a thirty-two bit output of state table 160 is to be provided.
  • the state table control register provides for compression of the state table information by allowing a variable length word to be stored in and read out of the state table.
  • the state table control register 162 stores and provides the length of each entry in the state table 160 of Figure 4A. Since some state table entries in Figure 1 are highly duplicative (e.g. "go to state 0", "stay in state n” ) , these entries not only can be represented by a single entry in state table 160 or at least much fewer than in Figure 1 but may also be represented by fewer bits, possibly as few as one which will yield substantial hardware economies even if most or all duplicative entries are included in the state table, as may be found convenient in some state tables. The principle of this reduction will be recognized by those skilled in the art as similar to so-called entropy coding.
  • next state pallette 170 is preferably implemented in much the same manner as the character palette 140 discussed above. However, as with the state memory 160, the number of entries that may be required is not, a priori , known and the length of individual entries is preferably much longer (e.g. two thirty-two bit words) . On the other hand, the next state palette 170 can be operated as a cache (e.g. using next state palette base address register 172) since only relatively small and predictable ranges of addresses need be contained at any given time.
  • the lower address thirty- two bit word output from the next state palette 170 is the token to be saved.
  • This token preferably is formed as a token value of sixteen bits, eight bits of token flags, both of which are stored in token buffer 190 at an address provided by pointer 192 to the beginning of the string and together with the length accumulated by counting successful character comparisons, and eight bits of control flags.
  • the control flags set interrupts to the host CPU or control processing in the parser accelerator.
  • One of these latter control flags is preferably used to set a skip enable function for characters which do not cause a change of state at a state other than state 0 such as a string of the same or related characters of arbitrary length which may occur in a string of interest, as alluded to above.
  • next state table entry can be reused without fetching it from SRAM/SDRAM.
  • the input buffer address 112 is incremented without additional processing; allowing substantial addition acceleration of parsing for certain strings of characters.
  • the second thirty-two bit word is an address offset fed back to register 180 and adder 150 to be concatenated with the index output from the character palette to form a pointer into the state table for the next character.
  • the initial address corresponding to state 0 is supplied by register 182.
  • the accelerator has access to the program memory of the host CPU where the character data (sometimes referred to as packet data connoting transmission of a network) and state table are located.
  • the accelerator 100 is under control of the main CPU via memory-mapped registers.
  • the accelerator can interrupt the main CPU to indicate exceptions, alarms and terminations.
  • pointers (112, 114) are set to the beginning an end of the input buffer 130 data to be analyzed, the state table to be used (as indicated by base address 182 and other control information (e.g. 142) is set up within the accelerator.
  • the CPU issues a command to the accelerator which, in response, fetches a first thirty-two bit word of data from the CPU program memory (e.g. 120 or a cache) and places it into the input buffer 130 from which the first byte/ASCII character is selected.
  • the accelerator fetches the state information corresponding to the input character (i.e. Figure 4A corresponds to a single character or a single column of the full state table of Figure 1) and the current state.
  • the state information includes the next state address and any special actions to be performed such as interrupting the CPU or terminating the processing.
  • the advancing of the state information thus supports detection not only of single strings of interest but also nested strings, alluded to above, and sequences of strings or corresponding tokens such as words or phrases of text in a document.
  • the interrupts and or exceptions which can be issued in response thereto are not limited to internal control of the parser and the issuance of tokens but may generate alerts or other initiate other processing to provide other functions such as intercepting unwanted electronic mail or blocking objectionable subject matter or content-based routing, possibly through issuance of special tokens for.
  • the accelerator next selects the next byte to be analyzed from input buffer 130 and repeats the process with the new state information which will already be available to adder 150.
  • the operation or token information storage can be performed concurrently. This continues until all four characters of the input word have been analyzed. Then (or concurrently with the analysis of the fourth character by prefetching) buffers 112, 114 are compared to determine if the end of the document buffer 120 is reached and, if so, an interrupt is sent back to the CPU. If not, a new word is fetched, the buffer 112 is updated and the processing is repeated.
  • pointers and counters are implemented in dedicated hardware they can be updated in parallel rather than serially as would be required if implemented in software. This reduces the time to analyze a byte of data to the time required to fetch the character from a local input buffer, generate the state table address from high speed local character palette memory, fetch the corresponding state table entry from memory and to fetch the next state information, again from local high speed memory. Some of these operations can be performed concurrently in separate parallel pipelines and other operations specified in the state table information (partially or entirely provided through the next state palette) may be carried out while analysis of further characters continues .
  • the invention provides substantial acceleration of the parsing process through a small and economical amount of dedicated hardware. While the parser accelerator can interrupt the CPU, the processing operation is entirely removed therefrom after the initial command to the parser accelerator.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Document Processing Apparatus (AREA)
  • Devices For Executing Special Programs (AREA)
  • Image Generation (AREA)

Abstract

Selon l'invention, un matériel spécialisé est utilisé pour procéder à l'analyse de documents de type documents XML<TM> en un temps très réduit tout en éliminant une partie sensible des frais de traitement d'une CPU hôte. L'utilisation conventionnelle d'une table d'état est répartie en une palette de caractères, une table d'état sous forme abrégée, et une palette d'état suivant. Les palettes peuvent être mises en oeuvre dans une mémoire spécialisée rapide et un système cache peut être utilisé afin d'accélérer les accès à la table d'état abrégée. Le traitement est effectué dans des pipelines parallèles qui peuvent être en partie concurrentes. Des registres spécialisés peuvent être mis à jour en parallèle ainsi que des chaînes de caractères spéciaux de longueur arbitraire adaptées par une caractéristique de saut de palette de caractères sous contrôle d'un bit indicateur afin d'accélérer davantage l'analyse de documents.
EP03809942A 2002-10-29 2003-10-03 Accelerateur d'analyseur materiel Withdrawn EP1579320A2 (fr)

Applications Claiming Priority (9)

Application Number Priority Date Filing Date Title
US331315 1994-10-28
US42177402P 2002-10-29 2002-10-29
US42177302P 2002-10-29 2002-10-29
US42177502P 2002-10-29 2002-10-29
US421773P 2002-10-29
US421774P 2002-10-29
US421775P 2002-10-29
US10/331,315 US20040083466A1 (en) 2002-10-29 2002-12-31 Hardware parser accelerator
PCT/US2003/031314 WO2004040446A2 (fr) 2002-10-29 2003-10-03 Accelerateur d'analyseur materiel

Publications (1)

Publication Number Publication Date
EP1579320A2 true EP1579320A2 (fr) 2005-09-28

Family

ID=32234359

Family Applications (1)

Application Number Title Priority Date Filing Date
EP03809942A Withdrawn EP1579320A2 (fr) 2002-10-29 2003-10-03 Accelerateur d'analyseur materiel

Country Status (6)

Country Link
EP (1) EP1579320A2 (fr)
JP (1) JP2006505043A (fr)
KR (1) KR20050072128A (fr)
AU (1) AU2003277249A1 (fr)
CA (1) CA2504652A1 (fr)
WO (1) WO2004040446A2 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7665015B2 (en) 2005-11-14 2010-02-16 Sun Microsystems, Inc. Hardware unit for parsing an XML document
US7716577B2 (en) 2005-11-14 2010-05-11 Oracle America, Inc. Method and apparatus for hardware XML acceleration
US7665016B2 (en) * 2005-11-14 2010-02-16 Sun Microsystems, Inc. Method and apparatus for virtualized XML parsing
US7596745B2 (en) 2005-11-14 2009-09-29 Sun Microsystems, Inc. Programmable hardware finite state machine for facilitating tokenization of an XML document
KR101231054B1 (ko) * 2008-12-19 2013-02-07 한국전자통신연구원 전용 하드웨어 엔진이 구비된 하드웨어 가속장치 및 그 방법

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS60105039A (ja) * 1983-11-12 1985-06-10 Nippon Telegr & Teleph Corp <Ntt> 文字列照合方式
JPH0746362B2 (ja) * 1986-02-03 1995-05-17 日本電信電話株式会社 文字列照合方法
JP2713962B2 (ja) * 1988-03-16 1998-02-16 株式会社日立製作所 文字列検索方法および装置
US5193192A (en) * 1989-12-29 1993-03-09 Supercomputer Systems Limited Partnership Vectorized LR parsing of computer programs
JPH04348469A (ja) * 1990-07-23 1992-12-03 Hitachi Ltd 文字列検索装置およびその方法
US5511213A (en) * 1992-05-08 1996-04-23 Correa; Nelson Associative memory processor architecture for the efficient execution of parsing algorithms for natural language processing and pattern recognition
US5414833A (en) * 1993-10-27 1995-05-09 International Business Machines Corporation Network security system and method using a parallel finite state machine adaptive active monitor and responder
US5995963A (en) * 1996-06-27 1999-11-30 Fujitsu Limited Apparatus and method of multi-string matching based on sparse state transition list
JP3062119B2 (ja) * 1997-06-20 2000-07-10 日本電気アイシーマイコンシステム株式会社 文字列探索用テーブル、その作成方法及び文字列探索方法
JP3368883B2 (ja) * 2000-02-04 2003-01-20 インターナショナル・ビジネス・マシーンズ・コーポレーション データ圧縮装置、データベースシステム、データ通信システム、データ圧縮方法、記憶媒体及びプログラム伝送装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2004040446A2 *

Also Published As

Publication number Publication date
CA2504652A1 (fr) 2004-05-13
WO2004040446A3 (fr) 2004-09-02
AU2003277249A1 (en) 2004-05-25
JP2006505043A (ja) 2006-02-09
KR20050072128A (ko) 2005-07-08
WO2004040446A2 (fr) 2004-05-13

Similar Documents

Publication Publication Date Title
US20040083466A1 (en) Hardware parser accelerator
US7146643B2 (en) Intrusion detection accelerator
US20070061884A1 (en) Intrusion detection accelerator
KR101648235B1 (ko) 정합-데이터 보고 모듈을 갖는 패턴 인식 프로세서
US7080094B2 (en) Hardware accelerated validating parser
US7254632B2 (en) Apparatus and method for pattern matching in text based protocol
US7734091B2 (en) Pattern-matching system
US7644080B2 (en) Method and apparatus for managing multiple data flows in a content search system
US20060259508A1 (en) Method and apparatus for detecting semantic elements using a push down automaton
US20040172234A1 (en) Hardware accelerator personality compiler
KR20120106978A (ko) 패턴 인식 프로세서에서 전력 소비를 줄이기 위한 방법들 및 장치들
JP2008507789A (ja) マルチパターン検索のための方法およびシステム
EP1579320A2 (fr) Accelerateur d&#39;analyseur materiel
US20020052902A1 (en) Method to convert unicode text to mixed codepages
CN100430896C (zh) 硬件解析器加速器
CA2504491A1 (fr) Analyseur de validation acceleree materiel
CN113868485A (zh) 数据加载方法及装置
WO2008036501A2 (fr) Recherche d&#39;expressions rationnelles dans des contenus de paquets au moyen de circuits de recherche spécialisés
JP2005275884A (ja) 字句をデータに変換する装置、方法及びプログラム
JPH06139279A (ja) 検索装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20050518

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20100521

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20101001