KR20100073134A - String pattern containment decision aperture and method using hash for automatically signatures generating system - Google Patents

Abstract

PURPOSE: A character string including type determining device for signature automatic creation system and a method thereof are provided to confirm similarity and comprehensibility in a classification of document or a search engine result thereby minimizing displayed result outcome. CONSTITUTION: If input data is new data, a white list management unit(110) renews bit vector table by hash value. A data search unit(120) searches the bit vector table through the white list management unit. The data search unit searches index having appointed hash value. The data search unit traces hash value of relevant index by searching active-set index according to circumstance.

Description

String pattern containment decision aperture and method using hash for automatically signatures generating system}

The present invention relates to an apparatus and method for determining string inclusion for a signature automatic generation system. In particular, the present invention relates to a string inclusion determination for an automatic generation system that manages a large amount of data by determining whether to match or include new data when managing data. An apparatus and method are provided.

The present invention is derived from the research conducted as part of the IT growth engine technology development of the Ministry of Knowledge Economy and the Ministry of Information and Telecommunication Research and Development. Development of real-time attack signature generation and management technology for

The security device and technology for service protection may be characterized as a technology for detecting an abnormal user by extracting application characteristics from a server to be protected and applying the same to a user accessing the server.

In order to detect and block an external intrusion or access in such a security device, data for determining whether it is unconstitutional or unfair is required. In such security devices, there is a need to continuously collect and manage data, but a large amount of data requires a system that can efficiently manage the data.

For the security of the network, it is necessary to first understand the characteristics of attack packets. The signature of the attack packet is registered as a signature, and when a signature registered in the received packet is detected, the corresponding security policy is applied to protect the target network from malicious users or programs.

Most of the techniques for extracting the characteristics of attack packets on a network are based on techniques for checking or classifying the similarity of electronic documents including web documents on the Internet. In order to examine the similarity between a huge amount of electronic documents, it is necessary to first express briefly the characteristics of each document. By comparing these simplified documents, the amount of computation required for similarity verification can be minimized.

That is, in a system for continuously collecting / managing similar data, a technique for determining whether new data is included in an existing data set, includes data, or another set is required.

However, when the size of the document is large or the size of the database is too large, comparing all the hash values calculated for one document is a major factor that degrades system performance. Sampling is used as a solution. That is, by comparing only the hash values sampled using the verified sampling methods, rather than comparing all the calculated hash values, it is possible to obtain a reliable result while not degrading the system performance.

Such a system requires a comparison operation for all entries. The technique used in “On the Resemblance and Containment” presented by Andrei.Z Broder uses the m-gram Hashing to hash the data and use this hash value. Containment and Resemblacne were judged using the abbreviated data set based on this. When the comparison with N entries is needed, the conventional technique judges the result through O (n) operation time. To apply a large amount of patterns, the “Approximate Pattern matching” technique can be applied.

However, when several signatures are generated in one packet, there remains a problem of deciding which one to use as a signature. If this is not done, multiple signatures are generated for one attack, making signature management impossible. Thus, verifying the signature that is generated is accompanied by a large amount of manual work.

In this case, Exact Matching is applicable, but it is easy to miss the detection when using the existing exact pattern matching technology, and it is not possible to determine whether the current pattern is included or completely matched to the existing specific pattern. There is this.

An object of the present invention is to minimize the results presented to the user by checking similarity and inclusion in document classification or search engine results, and if the white list is applied in the security system, it meets or includes this rule. The present invention provides an apparatus and method for determining string inclusion for a signature automatic generation system capable of fast processing according to a processing rule.

An apparatus for determining string inclusions according to the present invention for solving the above-mentioned problems is based on a sliding window technique using a concatenated 3-gram hashing or overlapping hashing method. A hashing unit for hashing, a data search unit for determining whether a white list of the data is included in the white list index having the same hash value as the hash value generated by the resolution by searching from a bit vector table, and adding the data to the white list. In the case of new data including a character string to be performed, a white list management unit for updating the bit vector table corresponding to the hash value for the data and registering the data in the white list and the registration by the white list management unit are controlled. In response to the determination result of the search unit, the data is discarded or Underground includes a control unit.

In addition, the method of determining a string inclusion according to the present invention includes hashing a string included in the imported data, retrieving a whitelist index having a value equal to a hash value generated as a result of the hashing from a bit vector table. If at least one whitelist index exists, matching the data whitelist index; and if the matching result, the character string of the data appears sequentially in the character string corresponding to the whitelist index, Determining that the data is included in a pre-registered white list, and processing the data as regular data.

In addition, according to the present invention, a method for determining string inclusion includes inputting data including a string to be included in a white list, determining whether the white list is included in the data, and when the data is new data, Registering a character string included in the white list, and outputting a result value indicating that the data is already included and a white list index corresponding to the data if the data is already included in the white list.

According to the present invention, an apparatus and method for determining string inclusion for a signature automatic generation system minimizes a result of presenting to a user by checking similarity and inclusion in a classification of a document or a search engine result, and using a white list ( In case of applying whitelist), it can be processed according to the processing rule in case of meeting or including this rule, and it prevents unnecessary operation, so it can be processed using less memory and can be processed quickly. Computation time is greatly reduced.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a block diagram referred to for the description of the configuration of a character string inclusion determining apparatus according to an embodiment of the present invention.

The string inclusion determining apparatus of the present invention performs a series of operations to minimize the redundancy of the outputted results by checking the similarity and inclusion in the classification of documents or search engine results. In addition, when a white list is applied to a series of security systems including a signature auto-generation system, a predetermined task is performed based on the white list processing rule in the case of meeting or being completely included in this rule. do.

As shown in FIG. 1, the string inclusion determining device includes an input unit 180, an output unit 190, a white list managing unit 110, a data searching unit 120, a bit vector map 150, and an overall result. The table 160 includes a current result tape 170, a hashing unit, and a controller 100 for controlling the overall operation.

The character string inclusion determining apparatus is provided with an input / output interface for a user to input data or to display predetermined data. The input unit 180 or the received data is output through the graphic interface of the output unit 190, and the white list related information according to the input data is also output through the graphic interface of the output unit 190.

Since the input unit 180 includes at least one input unit, the input unit 180 inputs predetermined data in response to an operation of the input unit. The output unit 190 includes at least one of a display device, a light emitting device, and an audio output device to output an operation state or provide a graphic interface to a user. Here, the input unit and the output unit may be provided with various input / output means according to the method of the input / output interface.

The bit vector map 150, the overall result table 160, and the current result table 170 are storage areas in which data stored separately in the storage device or data thereof is stored. The bit vector map 150, the overall result table 160, and the current result table 170 are included in the data unit, and are data collections in which data associated with a specified area is stored in response to a control command of the controller 100.

The bit vector map 150 includes a bit vector table, and each bit vector table stores result data according to hashing and index data thereof.

The current result table 170 stores the white list management unit 110, the data search unit 120, intermediate data about a predetermined task performed by the hashing unit, and data calculated by the collection operation. In particular, the current result table 160 is used when the data search unit searches for an index corresponding to a predetermined hash value in the bit vector table in the bit vector map, and when the white list manager registers new data, intermediate data and result data accordingly. Is stored.

The overall result table 160 stores data stored in the current result table 160 by the white list manager 110, the data searcher 9120, and the controller 100, or accumulates data that satisfies a predetermined calculation result or condition. Stored.

The hashing unit hashes data input from the input unit 180 in response to a control command of the controller 100 using a hash function. At this time, the hashing unit performs overlapping hashing and consumable hashing according to the hashing method.

The overlapping hashing unit 130 hashes the string repeatedly in a designated unit, and the conclusive hashing unit 140 divides and hashes the string in the designated unit. At this time, each hashing unit hashes by applying a sliding window.

The white list management unit 110 registers a series of strings to be used as a whitelist through inclusion pattern matching to be reflected in a bit-vector table, which is a memory for whitelist management.

The white list manager 110 determines whether the input data is included in the white list stored in the bit vector map according to a control command of the controller 100, and if the data is new data, uses the hash value to determine the bit vector table. Register the white list by updating.

The control unit 110 selects any one of the overlapping hashing unit and the conclusive hashing unit according to the determination result of the white list management unit 110, and applies data.

In this case, when the data included in the white list previously stored by the white list management unit 110, the controller 110 outputs a result message indicating that the data is already included through the output unit 190, and the white list index corresponding thereto. Will output

In addition, when the data is newly registered in the white list, the controller 110 may output the registration result and the white list index related thereto through the output unit 190. The controller 110 outputs a registration failure and an error message when an error occurs in the registration process.

When the whitelist is applied to the incoming packet or the content, the controller 100 applies the packet or the content to the data search unit 120 to determine whether the packet or the content data is included in the whitelist. As a result, packets are sent or received or blocked.

The data search unit 120 searches the bit vector table through the white list manager 120, searches for an index having a specified hash value, and optionally searches for an active index to track the hash value of the index.

In response to a control command of the controller 100, the data search unit 120 applies a sliding-window technique to an incoming packet or content to three consecutive data in the case of 3-gram hashing. Check whether there is a value set to '1' in BT [*] [hash value] among existing BT [white_index] [hash value].

The data search unit 120 writes the white list index values of the table to be checked in the bit vector table in the current result table 170.

The data search unit 120 performs an “AND” operation with the total table 160 in the initial state, and the present result table 170 is generated according to the intersection of the current result table and the entire entire check table. The result is reflected back to the overall result table 160.

That is, the data search unit 120 finds a white list index (white_index) in which all hash values generated while being shifted by the sliding-window technique are present in the hashing unit.

If the entire result table 160 becomes NULL, that is, if the index does not exist in the entire result table 160, even if the string to be compared does not reach the end, the current search string is whitelisted. ) Does not exist.

The control unit 110 handles such contents according to a routine normally executed by the program.

If at least one white list index (white_index) exists in the entire result table 160 after performing the operation to the end of the character string to be compared, the data search unit 120 may correct the current character string with the current character string. Through exact multi-pattern matching, it is checked whether a current set of strings is sequentially present in a string corresponding to an existing whitelist index.

In this case, when the strings corresponding to the whitelist indexes are sequentially present as described above, the controller 110 determines that the currently introduced object corresponds to the whitelist, and if the order is different, the controller 110 does not match the whitelist. I do not think. In response to the determination result, the packet or contents are discarded or maintained according to the whitelist application policy.

FIG. 2 is an exemplary view referred to for describing a hashing method used to determine whether a string includes a string according to an embodiment of the present invention. FIG. 2A is a diagram illustrating overlapping hashing, and FIG. 2B is a diagram illustrating conservative hashing.

In signature auto-generation systems that use string inclusion determinants, the contents of each signature are mostly ASCII code values or strings.

The string inclusion determiner does not store all of this data, but uses a hashing value instead of the data value to store the data. In addition, the string inclusion determining apparatus does not include all the hashing values, but stores them in the bit-vector table using the hashing values of the data and the whitelist index in the form of BT [white_index] [hash values].

For example, with respect to the bit vector table, when the index of the white list is 1 and the hash value is '1234', '1' is set in BT [1] [1234].

When hashing using a hash function, the hashing hashes the data using an overlapping hash or a convolutional hashing scheme.

In the overlapping hashing to which the sliding window is applied as shown in FIG. ', The second window 12 is set to ff, 53, 4d, and the third window 13 is set to '53, 4d, 42' and hashing is performed.

On the other hand, as shown in FIG. 2B, in case of the concatenated hashing, the first window 21 is '85, ff, 53 'and the second window 22 is' 4d, 42, 72' according to the size of the designated string. Hashing is performed by dividing by.

Here, the overlapping hashing unit 130 is used when hashing to register the white list, and the conclusive hashing unit 9140 checks whether the existing white list is included in the existing white list and whether the current content is included in the white list. Is used when hashing.

FIG. 3 is a diagram referred to for describing a bit vector table of an apparatus for determining string inclusions according to an embodiment of the present invention.

When the string inclusion device hashes the data and stores it in the bit vector table of the bit vector map, the whitelist index does not store an integer value of 32 bits corresponding to the hash value as much as possible by the hash function. The number of data corresponding to the product of the bit of the and the hashable area is stored. Accordingly, the string inclusion determiner can be applied using a small amount of memory for a large number of white lists.

FIG. 3 shows 2000 whitelists in a string inclusion determining apparatus using overlapping hashing with a sliding window and a hashing function that generates a distribution of hash values by two hashing schemes, namely, convolutional hashing, and the like. ) Shows a schematic data structure of a bit-vector table generated for processing certain data.

The bit vector table can be converted into 1M area 16K, 1K, etc. according to the range of hashing function values, and can be applied to the area when a large number of white lists is required.

Therefore, the string inclusion determining apparatus of the present invention can easily manage whether the new data set is correctly matched with a database or an existing data set, and whether the current data set is included with the existing data set. It is a skill that can be judged. A large number of white lists can be managed easily and with less data for areas that manage white lists in security systems.

In this case, although most engines require execution time corresponding to the size of M whitelists, the string inclusion determining device provides a fast search time using a bitvector table as described above.

4 is a flowchart of an operation description of a method for registering a whitelist of a string inclusion determining apparatus according to an embodiment of the present invention.

As shown in FIG. 4, the string inclusion determining apparatus reflects and registers a series of strings to be used as a whitelist in a bit-vector table, which is a memory for whitelist management.

An input / output interface for inputting data or displaying predetermined data is displayed on the string inclusion determining apparatus. When data for white list is input through an input / output interface such as a graphic user interface (S310), the controller 100 Applies the input data to the white list manager 110.

The white list manager 110 first determines whether the incoming data is included in the existing white list (S320), and if it is already included, informs the controller 100 that the data is already included. In this case, the control unit 100 outputs a result indicating that the data input through the output unit 190 is pre-contained data and a white list index thereof (S370).

Meanwhile, when the data is not included in the white list, the white list manager 110 checks whether the data is new data (S330) and applies the data to the hashing unit to register the data.

Since the control unit 100 registers data in the white list, the control unit 100 applies the data to the overlapping hashing unit 130 and the overlapping hashing unit 130 among the congestive hashing unit 140.

The overlapping hashing unit 130 performs a hashing operation on the input data according to the overlapping method in the form of a sliding window as shown in FIG. 2A (S340).

The white list management unit 110 receives a hashing value from the hashing unit and updates the corresponding value of the bit vector table corresponding to the generated hash value (S350).

When the registration for the input data is completed, the white list management unit 110 notifies the control unit 100 of the successful registration, and the control unit 100 outputs the result and the white list index information thereof through the output unit 190. .

On the other hand, if an error occurs while performing the above registration process, the white list manager 110 notifies the controller of the registration failure, and the controller 100 outputs the result of registration failure and the white list information related thereto. (S380)

FIG. 5 is a flowchart of an operation description of a preprocessing procedure of an apparatus for determining a string inclusion according to an embodiment of the present invention. The preprocessing process for the white list registration and inclusion determination of FIG. 4 described above will be described in more detail.

Referring to FIG. 5, the controller 100 applies the content to be used as the current white list and the related white list index to the white list manager 110 (S410).

The white list manager 110 determines whether it is the first white list (S420), and in the case of the first white list, the hashing unit performs overlapping n-gram hashing (S430). Here, an explanation will be given taking as an example that overlapping 3-gram hashing is performed.

When the overlapping 3-gram hashing is performed on the three strings on which the current window is located, the white list manager 110 updates the bit vector map using the hash value generated as a result (S440). . At this time, the white list manager 110 sets '1' to the bit vector table corresponding to the generated hash value, BT [white_index] [hash value].

The white list management unit 110 repeatedly performs the above process until reaching the end of the content to be added (S450) (S430 and S440). When the above process is performed until the end of the content, the controller 110 outputs the result and the corresponding white list index (S570).

On the other hand, if the data such as the input content is not the first white list, the white list management unit 110 to perform a conjunctive hashing on the data through the hashing unit (S460).

The hashing unit performs a conscutive 3-gram hashing by using the sliding window, and looks up the bit vector white list map (S470), and the white list management unit 110 reaches the last string of the white list. The process of searching for a hash value is repeated until

At this time, the data search unit 120 finds a whitelist index value in which the value of the bit vector table corresponding to the hash value is set to '1' for an index from 1 to a value obtained by subtracting 1 from the current whitelist index. S480).

For example, when the hash value is '1234' and the current white list index is 8, the data search unit 120 performs a bit vector table from BT [1] [1234] to BT [7] [1234]. Search and find the index whose value is 1.

The data search unit 120 reflects the result of the search process as described above in the current result table 170 (S490).

That is, when the values of BT [1] [1234], BT [2] [1234], BT [5] [1234], and BT [7] [1234] are set to 1, the data search unit 120 may have 1, 2, 5, and 7 indexes are stored in the current result table 170 as search results for the generated hash values.

In this case, when the position of the sliding-window is the first (S500), the data search unit 120 reflects the same value in the entire result table 160 (S510). In FIG. 2, when the current window is the first window, that is, '85, ff, 53 ', the search result is also stored in the overall result table.

If the position of the sliding-window is not the first, the data search unit 120 may present a white list index existing in the current result table 170 and a white list index present in the entire result table 160 calculated in the previous step. In operation S520, the intersection of the previous full result table and the white list index included in the current result table is reflected in the full result table 170 (S530).

If the index is not present in the current result table 170 and has a null value while performing the above process, or if the index is not present in the entire result table 160, the result is null (S540). 550) In order to determine that the current whitelist is not included in the existing whitelist, and to reflect the current whitelist in the bitvector table, the process of updating the bitvector table corresponding to the overlapping hashing and the hash value is the last string. Repeatedly performed until (S430, S440). When the bit vector table is updated in this way, the control unit 100 outputs the result according to operation S570.

If at least one index exists in the current result table and the entire result table, the process is repeated until the last character string is reached (S560), including a bit vector table search using a hash value generated by convolutional hashing. It performs (S460 S550).

When the operation is completed up to the last string of the white list, if at least one index exists in the entire result table 160, the current white list is included in the existing white list. 100, and the result is output through the output unit 190 (S570).

This completes the preprocessing process of ordering the requested whitelist.

FIG. 6 is a flowchart of an operation description of a method of determining whether a new object is included in a string inclusion determining device according to an embodiment of the present invention.

As shown in FIG. 6, the apparatus for determining string inclusion includes performing the following process on the currently introduced content to determine whether it corresponds to an existing white list.

When a new object composed of a series of character strings is input (S610), the controller 100 applies data such as an object to the data search unit 120. The input object is hashed by the concatenated hashing method (S620), and the data search unit 120 looks up the bitvector whitelist map (S630), and then searches for the generated hash value in the bitvector table. In operation S640, an index having a hash value equal to the generated hash value is found.

At this time, the bit vector table from the first whitelist to the number of whitelists existing in the system is searched to find an index having the same hash value as the generated hash value. For example, if the generated hash value is '1234' and there are 8 white lists, the index of the hash value '1234' is searched for BT [1] [hash value] through BT [8] [hash value]. .

In this case, the whitelist pattern has a successive 3-gram hash value. When comparing with a large amount of data, the current object to be compared has a hash value corresponding to 1/3 to determine whether it is included. The application time is reduced compared to the conventional.

If an index having a hash value corresponding to the generated hash value exists in the bit vector table (S650), the current result table 170 and the premise result table 160 are updated (S660).

 If the index is not present in the current result table 170 and has a null value while the above process is performed, or the index is not present in the entire result table 160, or null. S670 and S680, the data search unit 120 determines that the input object does not correspond to the white list, and applies a result thereof to the control unit 100, and the control unit 100 receives a packet or traffic for the input object. This is to be discarded or passed (S750).

If at least one index exists in the current result table and the entire result table, the bit vector table having a hash value corresponding to the hash value generated by the convolutional hashing until reaching the end of the introduced object (S690). Repeat the above process including the search (S620 S690).

After performing the above repetitive operation until the end of the imported object, if one or more whitelist indexes exist in the entire result table 160, exact matching with respect to each of the whiteless indexes existing in the entire result table 160 is performed. (Exact matching) is performed (S700).

As a result of the matching, it is determined whether the strings of the objects appear sequentially according to the exact matching (S710). When the strings of the current objects appear sequentially in the string set of the white list, it is determined that the current objects correspond to the white list. (S720) The result is applied to the controller 100.

Accordingly, the control unit 100 outputs a result to inform that the introduced object is included in the white list, releases the memory, and returns (S730).

On the other hand, if the matching result, the order of the string of the object is different, it is determined that the input object does not match the white list (S740), the result is output and the current object is discarded or passed (S750).

As described above, the present invention manages the bit vector table for the distributable area of the hash value in the registration process of giving regularity to the white list, and the hash value generated by the sliding-window area exists in the entire bit vector table. By continuously managing the index of the rule to prevent the operation is no longer performed if it does not exist in the entire result table, unnecessary operations are prevented from being performed.

In addition, the present invention performs multi-pattern matching after a predetermined filtering procedure using a small amount of data set, compared to a conventional pattern matching using a large amount of data, so that the current content is applied to an existing rule. Performing a procedure to check for conformance improves computation speed and reduces data usage. In addition, it is possible to provide multi-pattern matching in a short calculation time to maintain inclusion in a large amount of patterns.

As described above, the apparatus and method for determining string inclusion for a signature automatic generation system according to the present invention have been described with reference to the illustrated drawings. However, the present invention is not limited thereto by the embodiments and drawings disclosed herein. It can be applied by those skilled in the art.

1 is a block diagram referred to for the description of the configuration of a character string inclusion determining apparatus according to an embodiment of the present invention;

FIG. 2 is an exemplary diagram referred to for describing a hashing method used for determining whether a string includes a string according to an embodiment of the present invention.

3 is a reference to the description of the bit vector table of the string inclusion determining apparatus according to an embodiment of the present invention;

4 is a flowchart referred to in an operation description of a method for registering a whitelist of a string inclusion determining apparatus according to an embodiment of the present invention;

FIG. 5 is a flowchart for referring to an operation description of a preprocessing procedure of an apparatus for determining string inclusions in an embodiment of the present invention; FIG.

FIG. 6 is a flowchart of an operation description of a method of determining whether a new object is included in a string inclusion determining device according to an embodiment of the present invention.

<Explanation of symbols on main parts of the drawings>

100: control unit

110: white list management unit 120: data search unit

130: hash part 150: bit vector map

160: full result table 170: current result table

Claims (10)

Generating a hash value by hashing a string included in the imported data; Retrieving a whitelist index having a value equal to the hash value from a bitvector table; Matching a string corresponding to the retrieved white list index with the string of data; And If the character string having the same order as the character string of the data is included in the character string corresponding to the white list index, determining that the data is included in the white list and treating the data as regular data; How to decide. The method of claim 1, After the searching step, if the whitelist index having the same value as the hash value does not exist by repeating the hashing step and the searching step until the last character string of the data, the data is not included in the white list. Determining to include, and discarding or passing the data. The method of claim 2, The searching may include searching for a whitelist index having the same value as the hash value from the beginning of the bit vector table to the last registered whitelist index corresponding to the number of registered whitelists. How to decide. The method of claim 1, If the character string of the data and the sequence of the character string corresponding to the white list index are different from each other, determining that the data does not correspond to the white list, and discarding or passing the data. How to determine inclusion. The method of claim 1, In the hashing step, the string included in the data may be hashed by a consecutive n-gram hashing method according to a sliding-window technique. Inputting data including a character string to be registered in the white list; Determining whether the data includes the whitelist; If the data is new data, registering the character string in the whitelist; And And if the data is already included in the whitelist, outputting a result value indicating that the data is registered and a whitelist index corresponding to the data. The method of claim 6, In the determining whether the whitelist is included, hashing the character string of the data by using a consecutive n-gram hashing method according to a sliding window technique. And in the bit vector table corresponding to the generated hash value, if there is at least one whitelist index having a value of 1, determining that the data is included in the whitelist. The method of claim 7, wherein In the determining whether the whitelist is included in the bit vector table corresponding to the hash value, when there is no whitelist index having a value of 1, it is determined that the data is new data not included in the whitelist. Character string inclusion determining method. The method of claim 6, The registering step may hash the string of data using an overlapping n-gram hashing method according to a sliding window technique. And updating the bit vector table corresponding to the generated hash value to register the string of data in the white list. A hashing unit hashing the imported data by a consecutive n-gram hashing or overlapping hashing method according to a sliding window technique; A data search unit for searching a white list index having a value equal to the generated hash value from a bit vector table to determine whether the data includes a white list; A white list manager for updating the bit vector table corresponding to the hash value and registering the data to the white list when the data is new data including a string to be added to the white list; And And a controller which controls registration by the white list manager and discards or maintains the data in response to a determination result of the data search unit.

Images