CN100461091C - Methods and systems for content detection in a reconfigurable hardware - Google Patents

Abstract

Methods and systems consistent with the present invention identify a repeating content in a data stream. A hash function is computed for at least one portion of a plurality of portions of the data stream. The at least one portion of the data stream has benign characters removed therefrom to prevent the identification of a benign string as the repeating content. At least one counter of a plurality of counters is incremented responsive to the computed hash function result. Each counter corresponds to a respective computed hash function result. The repeating content is identified when the at least one of the plurality of counters exceeds a count value. It is verified that the identified repeating content is not a benign string.

Description

Carry out the method and system of content detection with reconfigurable hardware

The cross reference of related application

The application has required in the U.S. Provisional Application sequence number 60/604 of submission on August 24th, 2004,372 exercise questions are the date of application and the right of priority of " METHODS AND SYSTEMS FOR CONTENTDETECTION IN A RECONFIGURABLE HARDWARE ", at this its content are quoted for your guidance according to the degree that law allowed.

Technical field

Present invention relates in general to network communication field, and relate in particular to the method and system that is used for detecting via the data content that network passed on.

Background technology

Internet Worm (worm) comes work by utilize the leak in operating system and other software of being moved in system.Attack can damage security and make network performance degradation.Their influence comprises because the coml tremendous economic loss that system downtime and workman can't work and be caused.Expectation can become the part of crucial internet basic arrangement in the system that will be used to make network to avoid malicious code harm future.Current, these systems that are known as intrusion detection and prevention system (Intrusion Detection and Prevention Systems IDPS) only filter the worm of previous sign usually, so the effect of these systems is very limited.

Summary of the invention

The method according to this invention and system detect the frequent content that occurs in the network traffic flow, such as worm condition code (worm signature).Realize content detection with hardware, this compares with the conventional method based on software higher handling capacity is provided.Scanning identifies the pattern (pattern) of similar content via the data that data stream sent in network.The frequent data pattern that occurs is identified and is reported as may be the condition code of worm condition code or other type.Can walk abreast scan-data so that high-throughput to be provided.By parallel interior (on-chip) block storage of chip that hashes to of several data byte windows (window ofbytes) is kept handling capacity, wherein can walk abreast and upgrade block storage in each chip.Can compare the known features sign indicating number of being stored in the content that is identified and (off-chip) storer outside chip and determine whether to exist false (false positive) certainly.Owing to the pattern that can identify frequent appearance according to method and system of the present invention, so they are not limited to the known condition code of sign.

According to the method according to this invention, provide a kind of method that in data handling system, is used for the duplicate contents of identification data stream.Described method comprises step: be at least one the part compute Hash functions in a plurality of parts of described data stream; In response to institute's computed hash function result the value of at least one counter in a plurality of counters is increased, each counter is corresponding to respective computed hash function result; When surpassing predetermined threshold, the value of at least one counter in described a plurality of counters identifies described duplicate contents; And the duplicate contents that checking is identified not is to be optimum string (benign string).

According to system according to the present invention, provide a kind of system that is used for the duplicate contents of identification data stream.Described system comprises: the hash function counting circuit is used at least one the part compute Hash functions in a plurality of parts of described data stream; A plurality of counters increase the value of at least one counter in a plurality of counters in response to institute's computed hash function result, and each counter is corresponding to respective computed hash function result; The duplicate contents concentrator marker is used for identifying described duplicate contents when the value of at least one counter of described a plurality of counters surpasses predetermined count value; And validator, be used to verify that the duplicate contents that is identified is not is optimum string.

According to system according to the present invention, provide a kind of system that is used for the duplicate contents of identification data stream.Described system comprises: the device that is used at least one the part compute Hash functions in a plurality of parts of described data stream; Be used for the device that the value of at least one counter of a plurality of counters increased in response to institute's computed hash function result, each counter is corresponding to respective computed hash function result; Be used for when the value of at least one counter of described a plurality of counters surpasses predetermined count value, identifying the device of described duplicate contents; Verify that with being used to the duplicate contents that is identified is not is the device of optimum string.

Those skilled in the art are when investigating the following drawings and embodiment, and further feature of the present invention becomes clearer.Be intended to all this spare systems, method, feature and advantage are comprised in this manual, in the scope of the present invention, and protect by claims.

Description of drawings

The accompanying drawing of incorporating into and constituting this instructions part illustrates implementation of the present invention, and is used for explaining advantage of the present invention and principle together with the description: in the accompanying drawings,

Figure 1A is the block diagram that is used to carry out the system of content detection according to the present invention;

Figure 1B illustrates the how functional block diagram of data streams of condition code checkout equipment according to the present invention;

Fig. 2 is the block diagram according to condition code checkout equipment of the present invention;

Fig. 3 is the block diagram according to counting processor of the present invention;

Fig. 4 is the block diagram according to character filters of the present invention;

Fig. 5 is the block diagram according to byte shift device of the present invention;

Fig. 6 is the block diagram that comprises the control grouping of optimum character string according to the present invention;

Fig. 7 is the block diagram according to big count vector of the present invention;

Fig. 8 is the block diagram of the big count vector of more detailed Fig. 7;

Fig. 9 is the block diagram according to streamline of the present invention;

Figure 10 is the functional block diagram that is used to describe the byte of parallel processing data stream;

How Figure 11 shows priority encoder at the example that does not have deal with data under the situation of conflicting;

How Figure 12 shows priority encoder at the example with deal with data under the situation of conflict;

Figure 13 is the block diagram according to analyzer of the present invention;

Figure 14 is the constitutional diagram according to analyzer state of the present invention; With

Figure 15 is the block diagram from the control grouping of being sent according to alarm generation device of the present invention.

Spread all over several accompanying drawings, corresponding accompanying drawing condition code refers to corresponding part.

Embodiment

Now according to come reference at length as illustrated the present invention in the accompanying drawing according to method, system and the implementation that manufactures a product.

Detect the frequent content that occurs in the data stream according to method and system of the present invention,, resist polymorphic technology simultaneously, such as by employed those technology of worm author such as the worm condition code.For with the realization of High Speed content detection, come the realization system with hardware.

Figure 1A is the block diagram that is suitable for the declarative data disposal system of using according to method and system of the present invention 100.As shown, a plurality of main frames are connected to a plurality of sub-networks.That is, main frame 102,104 and 106 is connected to sub-network 108; Main frame 110 and 112 is connected to sub-network 114; And main frame 116 and 118 is connected to sub-network 120.Between the sub-network separately and described sub-network and more the traffic flows between the macroreticular 128 (such as the Internet) through router one 26.Virtual Local Area Network concentrator 122 is concentrated the network traffic flow that enters router one 26.By condition code checkout equipment 124 being placed between router and the VLAN concentrator 122, can be the traffic flows of content scanning between sub-network.

In the illustrative example of Figure 1A, condition code checkout equipment 124 is port expander (field-programmable port extender, FPX) platforms of field-programmable.The FPX platform allows to pass through to use big field programmable gate array, and (field programmable gate array, FPGA) 130 (such as Xilinx XCV2000E FPGA) handle express network stream.Following characteristics sign indicating number testing circuit can be downloaded among the FPGA 130 so that by handling network flow up to the message volume speed of 2.5 kilomegabit per seconds.Use 32 bit wide data words to come network traffic timing (clock) in FPGA 130.It will be appreciated by those skilled in the art that to use and be different from hardware and software component as described herein and realize according to method and system of the present invention.For example, can adopt other equipment that are different from the FPX platform to realize the condition code checkout equipment.

In the described here illustrative example, illustrate with reference to detecting the worm condition code, but be not limited thereto according to method and system of the present invention.But according to the duplicate contents in the method and system identification data stream of the present invention.Duplicate contents can be but be not limited to: worm; Virus; A large amount of visit network address crowds' appearance; There are a large amount of similarly Emails that are sent to a plurality of addressees, such as spam; Via the content of peer-to-peer network institute repeated exchanged, such as music or video; And the duplicate contents of other pattern.

Figure 1B illustrates the how functional block diagram of data streams of condition code checkout equipment 124 according to the present invention.In illustrative example, field programmable gate array 130 comprises the functional module of character filters 150, Hash processor 152, count vector (count vector) 154, time average processor 156, analysis of threshold device 158, memory chip analyzer 160 and alarm generation device 162.These functional modules provide the illustrative Premium Features view of field programmable gate array 130.Below with reference to Fig. 3-15 field programmable gate array 130 and function thereof have been described in more detail.

As shown in the illustrative example, character filters 150 is sampled data from data stream 170, and to leach unlikely be the character of a binary data part, thereby the serial data 172 of N byte is provided.As below describing in more detail, worm is made of binary data in typical case.Like this, character filters 150 leaches some characters of unlikely performance worm condition code characteristic.152 pairs of N byte serials of Hash processor 172 calculate k position hash, and the condition code that is produced is hashed to count vector 154.As below describing in more detail, count vector 154 can comprise a plurality of count vector.When condition code hashes to count vector 154, be increased by the value of the specified counter of hash.By periodic intervals, be known as here and measure at interval, the count value in each count vector is reduced the amount that is equal to or greater than owing to proper communication average that service traffics arrive, as determined according to time average processor 156.When count vector 154 arrived predetermined threshold, determined as analysis of threshold device 158, memory chip analyzer 160 hashed to the table in the memory chip 212 to harmful string (offending string).When identical string occurring, the same place in memory chip 212 carries out hash so that these two strings of comparison next time.If two strings are identical, produce so and report to the police.If two strings are different, utilize new string to rewrite string in the memory chip 212 so.Therefore, memory chip analyzer 160 can be by reducing because the half frequent warning that string caused that occurs reduces the warning number.When receiving warning message, alarm generation device 162 sends the control that comprises harmful condition code (offending signature) to external mechanical and divides into groups to be used for further analysis.

Fig. 2 is the block diagram that is used to illustrate in greater detail condition code checkout equipment 124.In illustrative example, the circuit that is used for detection signal on network programmable gate array 130 at the scene is implemented as the application 202 that is known as worm_app.Worm_app202 is suitable in the framework of layered protocol packing (wrapper) 204.As below describing in more detail, counting processor 206 receives packaging signal from layered protocol packing 204, and described packaging signal analysis is converted to byte stream, and described byte stream is hashed to count vector, and the value of counter is increased.Counting processor 206 is further counted average to the worm condition code number that is detected and is handled optimum string.Counting processor 206 output signal count_match, this signal is asserted to height for the condition code of the worm offending_signature that surpasses threshold value and corresponding 10 byte longs.In addition, counting processor 206 can be packed 204 output signals to layered protocol.

Realize the worm_app circuit, make it that high-throughput and low the delay are provided.In order to reach this point performance, the worm_app circuit can have streamline.In illustrative example, streamline length is 27 clock period and can followingly decomposes:

-FIFO postpones: 3 clock period

-counting processor delay: 11 clock period

-analyzer postpones: 13 clock period

Analyzer 208 receives from the input signal of counting processor 206 and with the hash table 210 of being stored in the memory chip 212 such as static RAM (static random access memory SRAM) interface.If count_match is asserted to height, visit memory chip 212 by analyzer 208 so.If identified offending_signature in the hash table 210 of memory chip 212, analyzer 208 outputs are asserted to high signal analyzer_match so.Alarm generation device 214 receives from the analyzer_match signal of analyzer 208 and the packaging signal that it is received from counting processor 206 and is delivered to layered protocol packing 204.When the analyzer_match signal is asserted to when high, alarm generation device 214 sends the control grouping that includes offending_signature.

Figure 3 illustrates the component level view of illustrative counting processor 206.Counting processor 206 comprises packet buffer 302.As below describing, when the counter in the occupied and described block RAM of block RAM can't be increased, packet buffer 302 is buffered packet during counting average period.Beyond average period, packet buffer 302 is via the communication service transmission at counting.Character filters 304 judges which byte will be included in the worm condition code.The input string that byte shift device 306 collects and can count according to the output of character filters 304.Big count vector 308 hash increase and generation warning as required the value of corresponding counter from the character string that byte shift device 306 is received.Each functional module of counting processor will be described below in more detail.

In the block diagram of Fig. 4, illustrate in greater detail character filters 304.Character filters 304 can make selected character be excluded from hash is calculated.Because worm is made of binary data in typical case, so the condition code checkout equipment can be ignored some characters in the data stream, described character very can not be the part of binary data.These characters for example comprise blank, newline, line feed and the space in the data stream.Text document for example comprises a large amount of spaces and the null that is used to fill.Avoid another reasons of these characters to be that the character string in null or space needn't show the characteristic of the desirable features sign indicating number that can be used for identifying worm.Preferred use can not appear at the string in the document.Be not limited to the heuristic mode that is used to avoid harmful condition code according to method and system of the present invention.The alternate manner that can realize including, but not limited to: identify and ignore text in the email message, the whole string of pre-service, perhaps the stream editor replaces them with the expression formula of search rule and with going here and there.

Character filters 304 receives 32 bit data word data_in and signal data_en is used as input, and whether the data that described signal data_en is used for identifying data_in are effective.Character filters 304 is split as 4 single bytes (byte1 is to byte4) to 32 words and exports corresponding signal whether comprise valid data (byte1valid is to byte4valid) so that show described byte.If one of its character that to be character filters 304 just seeking, it is considered to invalid so.If for example 4 byte character string a, newline, b, null are received as input by character filters 304, and suppose that character filters 304 is configured to ignore newline and null character (NUL), the corresponding output signal of character filters 304 may be so:

Byte1:a, Byte1 is effective: height

Byte2:newline, Byte2 is effective: low

Byte3:b, Byte3 is effective: height

Byte4:null, Byte4 is effective: low

Fig. 5 is the block diagram of illustrative byte shift device 306.Byte shift device 306 may be by big count vector 308 hash from the byte shift version of character filters 304 values of reading in and output characteristic sign indicating number.Byte shift device 306 is also exported and need and be used to tell when big count vector 308 begins to count average signal by the byte number of hash (num_hash).Byte shift device 306 is accepted from the data of character filters 304 outputs.In illustrative example, the output characteristic sign indicating number is 13 byte longs, and each condition code all comprises the overlapping character strings of 4 10 bytes.

Following illustrative example is described the function of byte shift unit.If input is that the character string a from first preceding example is followed in " NIMDAADMIN123 " back, newline, b, null, the byte shift version of character string may be that " MDAADMIN123ab " and num_hash may be 2 so.As described below, the num_hash value is used by big count vector 308.

The operation of detected characteristics number of codes is average in order to safeguard, periodically reduces the counting of institute's detected characteristics sign indicating number.In illustrative example, this takes place in boundaries of packets afterwards in the treated byte of fixed number (such as 2.5 megabyte).Byte shift device 306 is followed the tracks of the byte number that has been hashed to big count vector 308.When handled total amount of byte surpassed threshold value, byte shift device 306 experienced the following steps so:

1. byte shift device 308 is waited for the last word of the current group that will read from packet buffer 302, stops reading from packet buffer 302 then.From this moment, enter count processor 206 message volume by temporary cache in packet buffer 302.Do like this is because can't hash and count these bytes when counting mean time.

2. when the last word of current group had been handled by big count vector 308, byte shift device 306 was asserted as height to the subtract_now signal.This signal is used for beginning counting on average by big count vector 308.

When start of payload (service load begins) signal from packing is asserted to when high, byte shift device 306 is asserted as height to the count_now signal.When end of frame (frame end) signal from packing is asserted to when high, count_now is asserted to low.In view of the above, can count the byte that only comprises service load.

Byte shift device 306 can also determine whether there is optimum string in data stream.Can discern by being programmed in the byte shift device 306 as one group of string such as the optimum string from the code segment of Microsoft Update, although this string appears on the network usually, yet be not to be worm.By receiving optimum string grouping at byte shift device 306 optimum serially adding is downloaded in the big count vector 308 via data stream.When for example being sent to destination address 192.168.200.2 on being grouped in port one 200, the described grouping of byte shift device 306 supposition comprises 13 hashed values of optimum string.Use high 5 of hashed value to quote one of 32 block RAMs and use least-significant byte to quote one of 256 counters in each block RAM.Figure 6 illustrates the figure of the illustrative control grouping 602 that comprises optimum string.Low 13 and benign_valid by first word of benign_string output service load are asserted to height.Because control grouping comprises the optimum string that needn't count, so that count_now is asserted to is low.Benign_valid and count_string signal are used for avoiding counting optimum string by big count vector 308, as following explanation.

Fig. 7 is the block diagram of the big count vector 308 of illustrative.The output of byte shift device 306 is inputs of big count vector 308.Big count vector 308 comprises logic, be used for the hash input character string, solve conflict between block RAM, read, make the value of counter to increase and write back to block RAM from block RAM.In illustrative example, big count vector 308 comprises 32 block RAMs, and each block RAM has 256 counters, and each counter all is 16 bit wides.Utilize this big or small illustrative counter, can support counting up to 64K.Below with reference to Fig. 8 the functional module of big count vector 308 will be described in more detail.

Illustrative 308 each clock period of big count vector are calculated four hashed values to four 10 byte character strings, and described four 10 byte character strings are included in the signal character string of 13 bytes.Each clock period is calculated an above hashed value with maintain throughput.Because the condition code of being followed the tracks of may appear at any point in the service load, and regardless of their skews in grouping how they hashed to identical place and, so use identical hash function in all cases.Each hash function produces 13 value.

In order to detect the content of frequent appearance, calculate k position hash on 10 bytes (80) window of big 308 pairs of streamed data of count vector.In order to calculate described hash, in configuration counting processor, produce one group of k * 80 random binary value.Each of described hash is calculated as the XOR of selecting subclass at random (XOR) to 80 input strings.By making the hash function randomization, the adversary can't determine to cause the byte mode of too much hash collision.A plurality of hash calculating to each service load guarantee that simple polymorphic method is difficult to prove effective.

In illustrative embodiment, use to be called H ₃Universal hash function.Hash function H ₃Be defined as:

$h\left(X\right)={\mathrm{<figure-callout\; id="1"\; label="d"\; filenames="C200580033049E00341.png,C200580033049E00401.png"\; state="\{\{state\}\}">d</figure-callout>}}_1\&CenterDot;x_1\&CirclePlus;{\mathrm{<figure-callout\; id="2"\; label="d"\; filenames="C200580033049E00371.png"\; state="\{\{state\}\}">d</figure-callout>}}_2\&CenterDot;x_2\&CirclePlus;{\mathrm{<figure-callout\; id="3"\; label="d"\; filenames="C200580033049E00331.png,C200580033049E00341.png"\; state="\{\{state\}\}">d</figure-callout>}}_3\&CenterDot;x_3\&CirclePlus;...{\&CirclePlus;d}_b\&CenterDot;x_b$

In above formula, b is the string length that step-by-step is measured.In illustrative example, the b=80 position.(d ₁, d ₂, d ₃... d _b) be k * 80 a random binary value group.The random binary value is at [0..2 ^M+n-1] in the scope (wherein n is the single counter size of step-by-step metering, and 2 ^mBe the number of the block RAM that uses).In other words, the d value has the scope identical with the hashed value that will produce.With respect to input the performed xor function of this sets of random values is generated hashed value, it has certain distribution concerning input value.

In order to calculate described hash, for each position in the character string, if this position equals ' 1 ', this random value that is associated and current result XOR mutually so is with the acquisition hashed value.For example, given d=(101; 100; 110; 011) and input string X=1010, so corresponding 3 hash functions are 101 XOR 110=011.

Big count vector 308 uses hashed values to index in the counter vector, and described counter is included in the count vector such as count vector 802.When certain condition code hashed to counter, this made the value of counter increase by one.By periodic intervals, can be known as here and measure at interval, the count value in each count vector is reduced and is equal to or greater than because the amount of proper communication average that service traffics arrive.When the value of counter arrives predetermined threshold, analyzer 208 access chip external storages 212, as below will describing, and counter is reset.For the illustrative implementation of the circuit on the Xilinx FPGA,, block RAM in the chip of dual-port realizes count vector by being configured to memory cell array.Each illustrative storer can be carried out a read operation and a write operation in each clock period.As shown in Figure 9, realize that three class pipeline reads in each clock period, increases and write store.Since this condition code each clock period all can change and since each appearance of each condition code all be counted, so need high-performance from memory sub-system.The storer of dual-port allows to write back the occurrence number of a condition code, the appearance of just reading another feature sign indicating number simultaneously.

Measure end at interval in order to mark, big count vector 308 is reset counter periodically.After having passed through fixing byte window, by value being written as zero all counters that reset.Yet the method has defective.If measure end at interval near the time corresponding to the Counter Value of malice condition code just below threshold value, this counter that resets so can make this condition code can not be detected.Therefore mode as an alternative, the big count vector 308 of illustrative periodically deducts mean value from all counters.Mean value is calculated as desired byte number, and described byte can be hashed to the value of each counter at interval.As described below, the method requires to use comparer and subtracter.

In order to realize high-throughput, can in each clock period, handle a plurality of strings.In order to allow a plurality of storage operations of executed in parallel, in content detection system 130, use a plurality of block RAMs that count vector is fragmented in a plurality of memory banks (bank) as shown in figure 10.Use the high bit of hashed value to determine to visit which block RAM.Use determines that than low level the value of which counter in given block RAM should increase.May hash to identical block RAM more than one character string.This situation is known as " bank conflict " here.Can use priority encoder to solve bank conflict.Because the operation of priority encoder, for the system of pressing the operation of OC-48 scanning frequency rate, each clock period may not can count between 1 and 3 character string.

The probability c of conflict can represent with following formula:

$c=1-\underset{i=N-B+1}{\overset{N-1}{\mathrm{\&Pi;}}}\frac{i}{N}$

In the equation above, N is that the number and the B of employed block RAM are the byte numbers that arrives each clock period.

Priority encoder such as priority encoder 804 solves contingent conflict when two or more higher 5 positions in four hashed values are identical.The address of the block RAM that priority encoder 804 outputs need be increased.As shown in Figure 8, use higher 5 positions of hashed value to identify the block RAM that will increase.Use low 8 counters that the position comes index its value in block RAM to increase.Bram_num1 quotes each block RAM to bram_num4.Ctr_addr1 to ctr_addr4 quote counter that its value in each block RAM will increase number.When corresponding block RAM sum counter address was effective, num1_valid was asserted to height to num4_valid.Can trigger described warning so which condition code is big count vector 308 follow the tracks of corresponding to four kinds of possible condition codes by any one generation in 32 block RAMs and described warning owing to report to the police.This realizes to sign4 by using the signal sign1 corresponding to bram_num and ctr_addr signal.In illustrative example, signal sign1 can have one of following five values to sign4: one, two, three and four corresponding to the first, second, third and the 4th condition code in the 13 byte signal character strings.The optimum string of value eight expressions.

Value num_hash wherein need to have determined the number of the block RAM that manages conflict.If for example this signal value is two, this means so byte shift device 306 by two byte shifts this condition code.Therefore, owing to two other condition code is counted, so only count two condition codes.

Figure 11 illustrates the illustrative example of the function of priority encoder under the situation that does not have conflict.In illustrative example, in first clock period, all four input bytes are thought effectively by character filters.Therefore, all four condition codes are by hash, and sign1 has effective value together with them corresponding bram_num and ctr_addr signal to sign4.In cycle, have only two by character filters to be thought effectively in four input bytes at second clock.Therefore, have only two condition codes by hash.Therefore have only sign1 and sign2 to have effective value about condition code 3 and 4.

Figure 12 illustrates the illustrative example of the function of priority encoder under the situation that has conflict.As shown in illustrative example, the block RAM that is increased conflicts in both cases.In both of these case, a preferential condition code solves conflict.Condition code with respect to the priority of another condition code in big count vector 308.

In illustrative embodiment, because not comprising, the inherent function of block RAM do not support to reset and count on average, so provide packing (wrapper) to realize this function around described block RAM.The function declaration ground of described packing is represented by the illustrative count vector shown in Fig. 8.In big count vector 308 illustration 32 copies of this count vector assembly, for each block RAM that is just using corresponding one.

As shown in the illustrative example of count vector, described count vector has reset (resetting) signal.When the reset signal is asserted to when low, the value of each counter is initialized to 0.Because parallel initialization block RAM in illustrative example, so these cost 256 clock period (the counter number in each block RAM).Hash has identified the address that will read in count_vector.Dout has identified in the counter data corresponding to hash.Addr has identified the address that will write back the counting of increase, and this will be described below.Ctr_data has identified the value that will write back to count_vector.Set_ctr provides to count_vector and has allowed to write.When subtract is asserted to when high, big count vector is carried out iteration and is therefrom deducted mean value by each counter.As previously mentioned, mean value is calculated as the desired byte number that may hash to the value of counter in each interval.If the value of given counter is less than mean value, it is initialized to zero so.If the value of given counter comprises the specific fields that is associated with optimum string, it is not reduced so.Just as the count initialized vector, concurrency was guaranteed to realize subtraction in 256 clock period.

In order to support optimum string, utilize the value that exceeds threshold value to fill counter corresponding to the hash of optimum string.When counter had this value, circuit was skipped described increase and is write back step.

For a limited number of common character string, can not count Hash bucket (bucket), thereby and can avoid sending and report to the police.But along with the number of optimum string number, because have only less counter to be used for the detected characteristics sign indicating number, so reduced validity near available counter.For the uncommon character string of greater number, can in the software of downstream, avoid false and produce certainly.Affirm that in order to reduce the vacation that is sent to downstream software but can be handled by main control system is the optimum so frequent character string that occurs.

Return with reference to Fig. 8, the input of fetch stage 806 is the output from priority encoder 804.Be connected to the address and the data bus (for example, count vector 802) of 32 block RAMs from the output of fetch stage 806.Yet, in Fig. 8, only show a count vector 802 for the sake of simplicity.Assert suitable address and data-signal according to the value of the bram_num of fetch stage 806 input.The signal sign1 that enters fetch stage 806 is assigned to signb1 any one (after this be known as " sign " signal and refer to any one block RAM simultaneously) in the sign b32 to sign4, and described signal is except when can flow out fetch stage 806 when handling the control grouping that comprises optimum string.In this case, the designated value 8 of output sign signal makes comparing component 808 and increase assembly 810 can suitably handle it.

The output of the count vector such as count vector 802 checked by its corresponding comparing component 808, and if it less than threshold value, the inc signal of so described comparing component is asserted to height.If it equals threshold value, so big count vector 308 count_match signals are set to height so that notify the condition code of potential frequent appearance to analyzer 208.The count_match signal makes that memory chip 212 is occupied to reach 13 clock period (this is because it is to begin to read 10 byte character strings, compare string character string and write back the time that this character string spends from memory chip 212), and count_match suppresses signal guarantees to exist at least 13 clock period between two count_match signals interval.

Increasing and writing back in the level, there are four illustrative functions, increase and write back level and can carry out by streamline.In all cases, ctr_data is the value that is written back to count vector.Four illustrative functions are as follows:

If-inc signal has been asserted to height, the value of ctr_data is set to Duo one than the output of count_vector so.

If-sign value is 8, the value that is associated with optimum string is designated as ctr_data so.In illustrative example, this value is 0 x FFFF.

If-count vector is output as 0 x FFFF, so identical value is assigned to ctr_data so that preserve optimum string.

The default value of-ctr_data is 0.If counter has surpassed threshold value, so described 0 value can not change.

Valid (effectively) signal (for example, bl_valid) when being inverted (flop) suitable number of times, be used as the input that writing of count vector enable (, set_ctr).

During placement and route, can place some block RAMs according to this mode of bearing the large propagation delay.This may make described circuit not satisfy timing constraint.In illustrative example by remedying this situation in the input and output that trigger are included in block RAM.In Fig. 8, additional trigger is not shown so that keep simple and clear.

When finding to be harmful to condition code, big count vector 308 output count_match and corresponding condition code (sign_num).The suitable number of times of counting processor 206 counter-rotating string reflects the delay of big count vector 308.When count_match is asserted to when high, select offending_signature according to the value of sign_num.

Figure 13 is the block diagram of illustrative analyzer 208.Analyzer 208 keeps suspicious condition code and how long the definite condition code of estimation occurs once.Thereby analyzer 208 can reduce the warning number that is sent by alarm generation device 214.For like this, the value that analyzer be sure of to cross the counter of predetermined threshold is actually the result who character string frequently occurs.When the value of counter strode across predetermined threshold, harmful string was hashed to the table in the memory chip 212.Use said method that harmful condition code is calculated 17 hashed values.Memory chip 212 data buss are 19 bit wides.Hashed value is mapped to the high 17 of address signal.Low two of address signal are changed to be used for representing three consecutive words (described storer is used for storing 10 byte character strings) in the storer.Hashed value is used for indexing in the hash table 210 of memory chip.When identical string occurring, analyzer hashes to same place and two strings of comparison in the memory chip 212 next time.If two strings are identical, produce so and report to the police.If two strings are different, analyzer 208 rewrites memory chip Unit 212 and stores another string so.In this case, because hash function hashes to identical value to the several half frequent character string that occurs, may overflow by the occurrence count device.Since the half frequent character string that occurs be not pay close attention to, so analyzer 208 can prevent the expense that produces the grouping of reporting to the police.

The illustrative signal of explained later analyzer 208:

Count_match: when being asserted as by big count vector 308 when high, certain condition code has made counter arrive threshold value.

Offending_signature: the condition code corresponding to count_match just is being asserted to height.

Analyzer_match; When being asserted to when high, analyzer has verified that the counter that arrives threshold value is not false definite results.

Mod1_req: when being asserted to when high this signal indicating request access chip external storage 212.This signal remains height during positive access chip external storage 212.

Mod1_gr: when being asserted to when high this signal indicating permits access memory chip 212.

Mod1_rw: when this signal is asserted to when high that analyzer 208 reads and analyzer 208 writes memory chip 212 when low when this signal is asserted to from memory chip 212.

Mod1_addr: show the memory chip address that therefrom to read or to write.

Mod1_d_in: comprise the data that just from memory chip 212, reading.

Mod1_d_out: comprise the data that just are written into memory chip 212.

Analyzer 208 is configured to comprise a plurality of finite states that are used for memory chip 212 visits.Figure 14 illustrates the illustrative finite state machine that is used for analyzer 208.Explained later is each illustrative state depicted in figure 14.

Idle: the default conditions that are analyzer 208.Analyzer 208 state transfer from then on swaps out when high when count_match is asserted to.

Prep_for_sram: the access chip that in this state, asks for permission external storage 212.Analyzer 208 state transfer from then on swaps out when permission is authorized to.

Send_read_request:, realize three kinds of send_read_request states as shown in the illustrative example of Figure 14.Read in all three kinds of states of request in transmission, mod1_rw is asserted to height and mod1_addr and is set to the value that hash derived according to offending_signature.

Wait1: the data that wait will be read from memory chip 212.

Read_data_from_sram: the data from memory chip 212 on mod1_d_in are read in the temporary register.

Check_match: temporary register is connected in series and is compared with offending_signature.If these two equate, analyzer_match is asserted to height and analyzer 208 changes back to idle so.If these two unequal, analyzer 208 writes back to storer to new character string so.

Send_write_request:mod1_rw is asserted to low, and as reading state, mod1_addr is set to the value that hash derived according to offending_signature.

In case mod1_gr raises, the each transformation in the analyzer 208 just takes place at the clock edge so.

Memory chip 212 is used for storing complete string (not hashed version), is that 10 bytes (80) are long at string described in the illustrative example.Though fast hundreds of times of analyzers 208 than software, still require several additional clock period to visit memory chip 212, and this may make the data processing pipeline stall.In illustrative example, 10 byte character strings in the access chip external storage 212 require 13 clock period.

Can realize whenever the circuit that when memory chip 212 execute stores read, just makes the data processing pipeline stall.Yet, make pipeline stall have shortcoming.The purpose of the byte window calculating hashed value with respect to the whole group service load is to handle the situation of polymorphic worm.But consider the situation of more general non-polymorphic worm, wherein the grouping service load of the worm traffic is intimate is identical.In this case, can produce a series of continuous couplings to the whole group service load according to method and system of the present invention.May cause serious throughput-degrade for each coupling makes pipeline stall then, this is because each memory chip 212 visit will spend a plurality of clock period.In fact, doing like this may be useful to the assailant, and this is because the system manager may be forced to turn off system.In illustrative example, solution is to finish up to previous operation but skip further storage operation when do not make pipeline stall when memory chip 212 reads.Therefore, report to the police in case produce, the data on next 13 clock period (reading and writing back to the stand-by period related in the memory chip 212) can not cause producing further warning so.

In measuring at interval, observed condition code number can be approximately equal to handled number of characters.Because owing to the sub-fraction character is skipped in memory bank RAM conflict, so handled number of characters may be less.Given measurement gap length determines that the problem of threshold value can be reduced to the boundary of determining probability, and described probability surpasses the probability of i for the element number that is hashed to identical bucket when m element hashed to the table with b bucket.Described boundary is provided by following formula:

$b{\left(\frac{\mathrm{em}}{\mathrm{ib}}\right)}^i$

In illustrative example, m condition code hashed to b counter.In above expression, i is a threshold value.Thereby given measurement gap length can change threshold value, and it is acceptably little to make counter surpass the upper limit of probability of threshold value.This has reduced the number of unnecessary memory chip 212 visits subsequently.Therefore, because the condition code that arrives hashes to the value of counter randomly, so irregular condition code may make the value of counter surpass described predetermined threshold for suitably big threshold value.The probability that counter accurately receives i element can be provided by following formula:

$\left(\begin{array}{c}m\\ i\end{array}\right){\left(\frac{1}{b}\right)}^i{(1-\frac{1}{b})}^{(m-i)}\&le;\left(\begin{array}{c}m\\ i\end{array}\right){\left(\frac{1}{b}\right)}^i\&le;\left(\frac{\mathrm{me}}{i}\right){\left(\frac{1}{b}\right)}^i={\left(\frac{\mathrm{me}}{\mathrm{bi}}\right)}^i$

Second inequality is the result of the upper limit of binomial coefficient.The probability that the value of counter is at least i can be provided by following formula:

$\mathrm{Pr}(c\&GreaterEqual;i)\&le;\underset{k=i}{\overset{m}{\mathrm{\&Sigma;}}}{\left(\frac{\mathrm{em}}{\mathrm{bk}}\right)}^k\&le;{\left(\frac{\mathrm{em}}{\mathrm{ib}}\right)}^i[1+\left(\frac{\mathrm{em}}{\mathrm{ib}}\right)+{\left(\frac{\mathrm{em}}{\mathrm{ib}}\right)}^2+...+{\left(\frac{\mathrm{em}}{\mathrm{ib}}\right)}^{(m-i)}]$

Along with i increases, the item in the square bracket approaches 1.Therefore, the value of the counter probability that is at least i can be limited by following formula:

$b{\left(\frac{\mathrm{em}}{\mathrm{ib}}\right)}^i$

In illustrative embodiment, m is 2.5 megabyte owing to measure at interval, so the number b of counter is 8192, and predetermined threshold i is 850, and the probable range that counter overflows for the Random Communication service traffics is 1.02 x 10 ^-9In view of the above, for handled communication service at interval, the probability that counter overflows is so little as expectation.

When from analyzer 208 reception warning messages, alarm generation device 214 sends User Datagram Protoco (UDP) (userdatagram protocol, UDP) control grouping to the external data disposal system of monitoring on known UDP/IP port.This grouping may comprise this harmful condition code (it is calculated the character string of the byte of hash).When analyzer_match is asserted to when high, alarm generation device 214 sends this control grouping.In view of the above, can be labeled as the character string of the most frequent appearance suspicious then.Figure 15 is a block diagram of controlling grouping 1502 from the illustrative that alarm generation device 214 is sent.

Therefore, detect the frequent condition code that occurs in the network traffic flow according to method and system of the present invention.By realizing content detection, can realize high-throughput with hardware.In addition, by utilizing, can scan more substantial traffic flows with typical comparing based on the method for software by the concurrency that hardware provided.Hash to the chip-scale block storage by several byte windows are walked abreast and keep handling capacity, wherein can walk abreast and upgrade each chip-scale block storage.This is different from traditional method based on software, and the hash of wherein following the counter renewal may need order to carry out several instructions.In addition, use the memory chip analyzer to provide low vacation to affirm rate.To each grouping carry out a plurality of hash also help system hinder simple polymorphic measurement.

The intuition that previous network monitoring instrument relies on the system manager detects unusual in the network traffic flow.Automatically detect in the network traffic flow corresponding to the peak value that content frequently occurs according to method and system of the present invention.

Provided the above description of implementation of the present invention, be used to illustrate and illustrative purposes.This is not is exhaustive and does not limit the invention to disclosed precise forms.Can make amendment and change according to above instruction, perhaps can obtain these modifications and variations by implementing the present invention.For example, described implementation comprises software, but current implementation may be implemented as the combination of hardware and software or realized separately by hardware.In addition, can carry out according to being different from above-mentioned order by the illustrative process steps that program is performed, and can comprise additional treatment step.Scope of the present invention is defined by claims and equivalent thereof.

When introducing the element of the present invention or its preferred embodiment (one or more), " one ", " one ", " this " and " described " mean and have one or more these elements.Term " comprises ", " comprising " and " having " mean included and mean the other element that can exist except that listed element.

Can in above structure, carry out various changes without departing from the present invention, should be interpreted as illustrative and do not have restricted meaning in all the elements shown in above description or the accompanying drawing.

Claims (33)

1. method that in data handling system, is used for the duplicate contents of identification data stream, described method comprises step:

Be at least one the part compute Hash functions in a plurality of parts of described data stream;

In response to institute's computed hash function result, increase the value of at least one counter in a plurality of counters, each counter is corresponding to respective computed hash function result;

When surpassing predetermined count value, the value of at least one counter in described a plurality of counters identifies described duplicate contents; And

The duplicate contents that checking is identified is not optimum string.

2. the method for claim 1 is wherein calculated a plurality of hash functions of a plurality of part parallels ground calculating that described hash function is included as described data stream.

3. method as claimed in claim 2, wherein said a plurality of counter bit are in a plurality of memory banks.

4. method as claimed in claim 3 also comprises step:

When a plurality of counters that are arranged in same bank will be increased in the identical clock period, determine the priority which counter is increased.

5. the method for claim 1 also comprises step:

Filter at least one part in a plurality of parts of described data stream to remove predetermined data.

6. the method for claim 1 also comprises step:

Usage count on average comes periodically to reduce the value of each counter in described a plurality of counter.

7. the method for claim 1 also comprises step:

Determine whether the duplicate contents that is identified is false sign.

8. method as claimed in claim 7 is wherein by comparing the duplicate contents that is identified to determine whether the duplicate contents that is identified is false sign with the duplicate contents of previous sign.

9. method as claimed in claim 8, the duplicate contents of wherein previous sign are stored in the storer away from local storage, and this local storage comprises the duplicate contents that is identified.

10. the method for claim 1 wherein uses streamline to increase the value of at least one counter in described a plurality of counter.

11. the method for claim 1, wherein said duplicate contents are the worm condition codes.

12. the method for claim 1, wherein the duplicate contents that is identified has non-predefined condition code.

13. the method for claim 1, wherein said duplicate contents is a virus signature.

14. the method for claim 1, wherein said duplicate contents is a Spam signatures.

15. the method for claim 1, wherein said duplicate contents are the contents via the network repeated exchanged.

16. the method for claim 1, wherein said duplicate contents are the user's of a plurality of access websites appearance.

17. a system that is used for identification data stream duplicate contents, described system comprises:

The hash function counting circuit is used at least one the part compute Hash functions in a plurality of parts of described data stream;

A plurality of counters in response to institute's computed hash function result, increase the value of at least one counter in a plurality of counters, and each counter is corresponding to respective computed hash function result;

The duplicate contents concentrator marker is used for identifying described duplicate contents when the value of at least one counter of described a plurality of counters surpasses predetermined count value; With

Validator is used to verify that the duplicate contents that is identified is not optimum string.

18. a plurality of hash functions of a plurality of part parallels ground calculating that described hash function is included as described data stream wherein calculate in system as claimed in claim 17.

19. system as claimed in claim 18, wherein said a plurality of counter bit are in a plurality of memory banks.

20. system as claimed in claim 19 comprises:

Priority encoder is used for determining the priority which counter is increased when a plurality of counters that are positioned at same bank are increased.

21. system as claimed in claim 17 comprises:

Filtrator, at least one part of a plurality of parts that is used for filtering described data stream is to remove predetermined data.

22. system as claimed in claim 17, wherein usage count on average comes periodically to reduce the value of each counter in described a plurality of counter.

23. system as claimed in claim 17 comprises:

Analyzer is used to determine whether the duplicate contents that is identified is false sign.

24. system as claimed in claim 23 is wherein by comparing the duplicate contents that is identified to determine whether the duplicate contents that is identified is false sign with the duplicate contents of previous sign.

25. system as claimed in claim 24, the duplicate contents of wherein previous sign is stored in the storer away from local storage, and this local storage comprises the duplicate contents that is identified.

26. system as claimed in claim 17 wherein uses streamline to increase the value of at least one counter in described a plurality of counter.

27. system as claimed in claim 17, wherein said duplicate contents is the worm condition code.

28. system as claimed in claim 17, wherein the duplicate contents that is identified has non-predefined condition code.

29. system as claimed in claim 17, wherein said duplicate contents is a virus signature.

30. system as claimed in claim 17, wherein said duplicate contents is a Spam signatures.

31. system as claimed in claim 17, wherein said duplicate contents is the content via the network repeated exchanged.

32. system as claimed in claim 17, wherein said duplicate contents is the user's of a plurality of access websites appearance.

33. a system that is used for the duplicate contents of identification data stream, described system comprises:

Be used to the device of at least one the part compute Hash functions in a plurality of parts of described data stream, at least one part of described data stream is therefrom removed optimum character to prevent that optimum string is designated duplicate contents;

Be used for increasing in response to institute's computed hash function result the device of value of at least one counter of a plurality of counters, each counter is corresponding to respective computed hash function result;

Be used for when the value of at least one counter of described a plurality of counters surpasses predetermined count value, identifying the device of described duplicate contents; With

Be used to verify that the duplicate contents that is identified is not the device of optimum string.

Images