KR101003502B1 - Signature String clustering Method Based on the Resemblance and Containment in the Sequence - Google Patents

Abstract

The present invention relates to a signature string generation method based on similarity and inclusion of a string. The present invention enables the signature string to be quickly and efficiently by clustering the string and generating the signature string based on the similarity and inclusion of the string of the incoming data so that the signature string used in IDS can be automatically generated online.

Signature, string, similarity, inclusion, clustering

Description

Signature String clustering Method Based on the Resemblance and Containment in the Sequence}

The present invention relates to entry control through signature clustering based on similarity and inclusion of signatures used in intrusion detection systems in the field of information security.

The present invention is derived from the research carried out by the IT growth engine technology development of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [Task Management No .: 2006-S-042-02, Task name: Real time attack signature generation and management technology development].

The signature clustering associated with the conventional intrusion detection is applied by hierarchical clustering technique to the signature in the technology related to automatic signature generation. In this technique, each tokenized content of the signature to be generated is used, and then each rule corresponding to subsequence and combination between each token is used.

 If there are S flows, the technique used in the conventional polygraph constitutes S clusters, and each cluster includes one flow. Because the technology is off-line, it cannot process new signatures on-line in real time. Afterwards, clusters are repeatedly merged using a hierarchical clustering method. The clustering method used here is a greedy clustering method.

The present invention proposes a technique for processing such signature clustering algorithm on-line in real time.

Intrusion detection systems such as SNORT, which are widely used as signatures of general intrusion detection systems as open source, and in the case of commercial systems, do not take into consideration the inclusion and similarity of the applied signatures. For this reason, there are more than 10,000 signatures in case of severe signatures per event. The purpose of the present invention is to provide a method of controlling the number of rules to which a signature is applied by determining the association and inclusion of many signatures. Through the control of the number of rules, the overhead of rule management in S / W can be reduced and the capacity of TCAM and SRAM used in the security system provided in the form of hardware appliance can be reduced.

The signature string generation method based on the similarity and inclusion of a string according to the present invention comprises the steps of: tokenizing the packet into a content unit having a predetermined size in order to generate a signature from an incoming packet; Generating signature content with the set of tokenized individual contents and generating abbreviated data for the signature content; The signature content may be compared to the existing tree according to whether the contract data is included in the existing contract data, or whether the contract data is similar to the existing contract data by a predetermined criterion or more. Classifying the signature content into a tree by adding or inserting into a new tree; And separately storing a record of the signature string generated based on the tree of the signature content, and resetting the tree at predetermined time intervals.

In addition, the signature string generation method based on the similarity and inclusion of the character string according to the present invention forms a signature string corresponding to the inclusion between the abbreviated data of the signature content or similarity above a predetermined criterion, which is tokenized from the incoming data packet. Placing the signature content in a tree; Extracting a common permutation for at least one signature content included in each tree; Generating a multi-content signature string based on the extracted permutations; And generate abbreviated data of the signature string, compare the abbreviated data of the existing signature string, and if the abbreviated data of the signature string is not the same as abbreviated data of the existing signature string in response to a comparison result. It includes the process of registering as a data for identifying harmful packets.

If there are basically S documents for clustering, conventionally, S clusters are initially created, and then the associations between the S clusters are analyzed in the blocking state so that no new documents can be processed anymore. By merging repeatedly, one signature can be finally generated in one cluster.

In contrast, the present invention generates the corresponding abbreviation data each time a signature occurs, and compares the similarity and inclusion of the newly generated signature abbreviation data with the existing abbreviation data as a reference element. On the other hand, if similarity and inclusion are satisfied, clustering is performed on the same tree. If the similarity and inclusion are not satisfied, a new tree is created and data is inserted into logically different clusters.

While the conventional method is performed offline, in contrast to a method that does not provide processing for incoming data after clustering, the present invention already provides clustering based on similarity upon insertion of data. By this procedure, only homogeneous data exists in one tree.

Through the present invention, the automatic signature generation system operates online as a real-time processing system, and provides online clustering for the generated signatures.

Specific details of other embodiments are included in the detailed description and the drawings.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the embodiments make the disclosure of the present invention complete, and the general knowledge in the art to which the present invention belongs. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

For reference, the signature string generation method illustrated in the present invention is not intended to limit the scope of the present invention and is merely one embodiment. Unlike the conventional method, the signature string generation method of the present invention reduces the overhead incurred during data processing, and thus automatically generates signatures online. Therefore, a signature packet inspection system (IDS) based on signatures is provided. It may be interlocked.

1 is a flow diagram illustrating a signature string generation method based on similarity and inclusion of a string according to the present invention. Based on FIG. 1, the present invention will be described in general.

The contents of the signature are mostly ASCII code values or strings. The present invention utilizes an abbreviated bit object that can substitute for this value rather than storing all of the value of this data. Using this method, similarity and inclusion can be determined using only the abbreviated information without having all the strings of strings such as Jaccard index (Jaccard similarity coefficient).

When the signature auto-generation system is connected to a network for generating a signature and data is introduced (S100), the incoming packet is divided into tokens (contents) having a predetermined arbitrary size such as a character, a word, and a line (S110). ).

The separated tokens complete the signature content which is a candidate group of the signature string to be generated later (S120). The abbreviated data is generated by applying the Rabin Pinkerprint to the signature content (S130). Thereafter, it is checked whether the generated abbreviated data is included in the previously generated abbreviated data (S140). If it is determined to be included, the number of hits of the tree stored in the node of the signature content of the existing reduced data is increased (S150).

If it is not confirmed to be included, but is found to be similar (S160), the signature content is added as a new node to the existing tree, and the number of hits existing at the last position of the existing tree node is increased (S170). .

However, if the newly generated signature content is not included in the existing contract data or is not similar, a new tree is generated (S180).

Hereinafter, the tree generation process will be described in detail with reference to FIGS. 2 and 3.

As mentioned earlier, the task of separating each document into tokens of characters, words, lines, etc. must be preceded. Configure these tokenizers to recognize each document as a sequence of tokens in a certain pattern. In order to apply the token described here to the signature, it is determined as one content as shown in (210 to 217) of FIG. 2, and the multi-content, which is the entire signature, is mapped and applied as a document as shown in (200) of FIG.

As such, as shown in FIG. 2, one signature 200 may be configured as a multi-content that is a set of subsequences (subsequences) that are individual contents 210 to 217. In the present invention, clustering is constructed using similarity, inclusion, and distance value between contents.

In the existing invention, after clustering for all contents, hierarchical clustering of bottom-up was performed. In order to use this method, each cluster must be configured. Therefore, after clustering by offline processing, it is determined whether it is applicable as a signature to the result and false positive rate.

In the present invention, when the new signature content 301 flows in as shown in FIG. 3, the signature data 303 is applied to the signature content to generate the abbreviated data 303 matching the incoming signature. When such contract data is newly generated, the signature content is inserted as a new node in the tree 304 connected to the generated contract data, and the hit number of the signature content is recorded in the last node of the tree. In this case, the abbreviated data is added to the abbreviated data list, and the addresses of the trees are mapped in the form of a linked list with respect to the tree connected to the abbreviated data for each abbreviated data entry.

That is, in order to generate a signature, patterns of packet data that is a signature candidate group are stored in a predetermined tree. This generates a signature string composed of a plurality of contents by a combination of subsequences in which the same subsequence occurs X or more times.

The number of hits of the content at the last node means that the total number of times data has been inserted for the content starting at the root node and reaching the current node. This is used as a reference count to extract characteristics common within the cluster, with similar patterns clustered within a specific port, and used to generate a signature string.

Then, when the second signature 311 is introduced, the abbreviation data 313 is obtained through the same process 302 as above, and the similarity, distance, and the like with the value of the abbreviation data 312 having information on the signature are included. Calculate the last name.

If the inclusion 303 is close to 1, the newly added signature 311 may be included in or match the existing signature 301. In addition, when the similarity 301 is greater than or equal to a predetermined threshold value, the signature content is inserted into the tree 313 indicated by the previously abbreviated data 312, and the hit number of the content is recorded in the last node.

Unlike the above, when the inclusion 303 and the similarity 301 of the abbreviation data 313 of the second signature 311 have a value lower than the threshold value, the current abbreviation data 313 is newly inserted into the abbreviation data list, and the new Create a tree 315. In this way, clustering is performed through the procedure of creating a tree by determining similarity in advance.

Based on the above similarity and inclusion, the signature is inserted into the matching tree. If a continuous attack is repeated in a typical network, three minutes will produce a lot of similar types of signatures. In the present invention, which provides an automatic signature generation system, there is an abbreviated data list for each port, and the address of the tree is mapped in the form of a linked list for each abbreviated data entry.

That is, with respect to the harmful pattern (multi subsequence) extracted from the payload of traffic flowing for each port, the contract data of a specific site is generated by using Karp-Rabin finger print and the like, and the contract data is the same pattern. And it is used as a factor to calculate the pattern included in it. The characteristics of each entry stored in the tree, which is one cluster, can be presented through the abbreviated data list.

The following equations illustrate the equations used by the Jaccard Index to calculate similarity, distance, and inclusion between strings by the present invention.

The similarity between Document A and Document B, which includes a string, has a value between 0 and 1, as shown in Equation 1. Based on the formula in Equation 1, the logical distance between Document A and Document B is obtained by the formula shown in Equation 2.

The probability that the document A is included in the document B is calculated through the equation shown in Equation (3). Inclusion has a value between 0 and 1.

R (A, B) = | S (A) ∩S (B) | / | S (A) ∪S (B) |

D (A, B) = 1-R (A, B)

C (A, B) = | S (A) ∩S (B) | / | S (A) |

FIG. 4 is a flowchart illustrating a method of generating a signature string in each tree generated through the process of FIG. 1. A method of generating a signature string in each tree is described as follows.

First, a common permutation is extracted by applying a technique for finding a common long sequence (LCS) in each tree (S200).

On the basis of the extracted permutation, a multi-content type signature string is generated (S220), and abbreviated data for the generated signature string are generated (S230).

It is determined whether the generated abbreviated data is the same as the abbreviated data of the previously generated signature string (S240), and if it is not the same, the generated signature sheeting is registered as a signature string to be compared when the harmful packet is determined and the abbreviated data is stored ( S250). However, in the same case, the shortened data increases the number of duplicate generations of the same existing signature string.

In other words, by applying a LCS (Longest Common Subsequence) to each tree, a common permutation (subsequence) value present in the tree is extracted, and the corresponding value is generated as a multi-content signature string.

   By generating the abbreviated data for the signature string generated in this way, the signature generation record is generated so that duplicate signatures are not repeatedly generated for a predetermined period of time.

The reason for generating and managing the abbreviated data of the separately generated signature string is as follows.

There may be several pieces of abbreviated data with signature characteristics for one message. When multiple subsequences representing harmful patterns extracted from the payload are introduced into the clustering engine, the current pattern (signature candidate) is the same as the content inserted into the tree compared to the list of abbreviated data existing in each cluster. If it does, it will not be inserted into the tree. Similarly, but not included, insert into a tree used as the same cluster.

The abbreviation data is a value generated through a kind of hessing on the signature candidates (patterns) to be introduced. This can be used to determine whether or not similarity is included. However, in order to finally generate a signature string, signature signatures that have been introduced must be used. In order to use this, only the allocation of clusters is used as abbreviated data, and the data handled in the actual cluster uses a message flowing as a signature candidate.

In the case of using the tree-type signature, the tree is reset every predetermined time (15 minutes, 30 minutes) to overcome the limitation of the capacity of the tree, depending on the amount of traffic flowing from the network. Even if the tree, which is a repository of the signature contents, is reset, a record of the signature generated as a result is stored in a separate memory in the form of abbreviated data.

The reason for resetting the tree is that the network traffic has different characteristics over time, so the trees managing signature candidates must have the characteristics of the current network. To this end, a signature can be generated based on the characteristics of recently input traffic through RESET of a periodic tree. However, since the abbreviation data of the generated signature string is not reset, it is generated as a signature through clustering, but if it matches the existing signature abbreviation data, it is not generated for the signature and the number of duplicate generation counts is recorded.

The reason for recording the duplicate count count is generated as a signature through the clustering, but only the elements constituting the tree and cluster are reset while the system is running. Indicating that these particular signatures were generated frequently throughout the system means that packets of the same type continue to occur for the network. This is similar to expressing that several times an attack has occurred for a specific pattern in GUI such as IDS.

 Through the above steps, the content for the signature is generated, but the packet data (5-tuple) value associated with the signature is unknown. When the signature is generated through clustering for the generation of such data, the packet information collection flag is enabled, and when the incoming signature content and the generated signature string are associative and included, the 5-tuple is newly generated. It is extracted and combined with the signature string to provide the signature information. By providing such a 5-tuple, the reliability of the signature can be improved by dumping the payload for the signature generated based on the contents of the packet by applying a method such as real-time verification.

That is, there is a function of verifying whether a packet that is determined to be an actual attack exists in the packet through session recombination of packets used when generating a signature in the current system. The basic data of these functions is the session data of the packet. For this purpose, if 5-TUPLE data having a packet payload matching (similar to the signature) currently generated is passed, the payload data corresponding to this 5-TUPLE ( Payload data included in the session) can be compared by DUMP. When the session is dumped through the received 5-TUPLE, it has almost the same value as the payload referenced when the signature is generated. Therefore, it is possible to determine whether the signature is correctly generated.

The invention can also be embodied as computer readable code on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD_ROM, magnetic tape floppy disks, optical data suits, and the like, and may also be implemented in the form of carrier waves (for example, transmission over the Internet). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

1 is a flow diagram illustrating a method of generating a tree among a signature generation method using similarity and inclusion of characters according to the present invention;

2 and 3 are diagrams illustrating the generation of abbreviated data by tokenizing a document upon signature generation;

4 is a flowchart illustrating a method of generating a signature string from a generated tree among the signature generation methods using the similarity and inclusion of characters according to the present invention.

Claims (12)

Tokenizing the packet into content units having a predetermined random size to generate a signature from an incoming packet; Generating signature content with the set of tokenized individual contents and generating abbreviated data for the signature content; The signature content may be compared to the existing tree according to whether the contract data is included in the existing contract data, or whether the contract data is similar to the existing contract data by a predetermined criterion or more. Classifying the signature content into a tree by adding or inserting into a new tree; And Generating a signature string based on the similarity and inclusion of a string, including separately storing a record of the signature string generated on the basis of the tree of signature contents and resetting the tree at predetermined time intervals. Way. According to claim 1, wherein the classification process, If the abbreviated data is not included in the existing abbreviated data but is similar by a predetermined criterion or more, the signature content is added as a new node to the tree including the existing abbreviated data. And if not included or similar, adding the abbreviated data to the abbreviated data list, generating a new tree, and inserting the signature content. The method of claim 2, When the signature content is added to the tree of the existing contract data as a new node, the similarity and inclusion of the character string may be increased by increasing the number of hits recorded in the last node of the tree of the existing contract data. Signature string generation method based on. The method of claim 2, When the abbreviated data is included in the existing abbreviated data, the number of hits recorded in the last node of the tree including the existing abbreviated data is added without adding the signature content to the tree including the existing abbreviated data. Signature string generation method based on the similarity and inclusion of the string characterized by increasing. The method of claim 1, A signature string generation method based on the similarity and inclusion of a character string, characterized in that the inclusion indicating whether the abbreviated data is included in the existing abbreviated data is calculated based on the following equation using a Jaccard Index. C (A, B) = | S (A) ∩S (B) | / | S (A) | (However, the C function indicates whether the A and B signature contents are included, and has a value between 0 and 1, and 1 means it is included.) The method of claim 1, The similarity that indicates whether the abbreviated data is similar to the existing abbreviated data over a predetermined criterion is calculated based on the following equation using the Jaccard Index. Way. R (A, B) = | S (A) ∩S (B) | / | S (A) ∪S (B) | (However, the R function indicates whether the A and B signature contents are similar, and has a value between 0 and 1, and the closer to 1, the higher the similarity). The method of claim 1, And generating the abbreviated data by applying a rabbin fingerprint (RF) to the signature content. The method of claim 2, The abbreviation data list is a signature string generation method based on the similarity and inclusion of the character string, characterized in that the address of the tree is mapped in the form of a linked list, for each contained abbreviation data. Placing the signature content in a tree that will form a signature string in response to the inclusion between the abbreviated data of the signature content or similarity above a predetermined criterion, tokenized from the incoming data packet; Extracting a common permutation for at least one signature content included in each tree; Generating a multi-content signature string based on the extracted permutations; And Generate the abbreviated data of the signature string, compare the abbreviated data of the existing signature string, and if the abbreviated data of the signature string is not the same as the abbreviated data of the existing signature string in response to the comparison result Signature string generation method based on the similarity and inclusion of the string including the process of registering as harmful packet identification data. The method of claim 9, The method of extracting permutations extracts a common permutation of the signature content by determining a longest common subsequence (LCS) for each of the trees. The method of claim 9, In response to the comparison result, if the contract data of the signature string is not the same as the contract data of the existing signature string, separately storing the contract data of the signature string as a record of the signature for the tree to be reset. Signature string generation method based on the similarity and inclusion of the string. The method of claim 11, In response to the comparison result, when it is determined that the abbreviation data of the signature string is the same as the abbreviation data of the existing signature string, in order to prevent duplication of the signature string, the number of duplicate generation counts for the existing signature string is determined. Signature string generation method based on the similarity and inclusion of the string characterized by increasing.

Images