KR100912806B1 - Automatic Signature Extraction System and Method for Internet traffic - Google Patents

Abstract

The present invention relates to an apparatus and method for extracting a specific Internet application signature from Internet traffic, comprising: a file processing unit for sequentially extracting individual flows from an Internet traffic file stored in a flow form, in the individual flows received from the file processing unit A dictionary information collecting unit for inspecting payloads attached to respective packets stored in the packet, and collecting dictionary information representing characteristics of the payload required to generate a signature; And a signature record storage unit configured to generate a signature record while circulating the individual flows based on the prior information, so that the signature used in the content-based traffic detection apparatus can be automatically and quickly extracted. It can effectively block traffic from P2P applications, which are known to occupy excessive bandwidth among the traffic entering the Internet.

Internet traffic, flow, traffic detection, signature

Description

Device and method for extracting specific Internet application signatures from Internet traffic. {Automatic Signature Extraction System and Method for Internet traffic}

1 is a view showing a schematic configuration of a signature automatic extraction system according to an embodiment of the present invention.

2 is a diagram illustrating a configuration of a signature record according to an exemplary embodiment of the present invention.

3 is a flowchart illustrating a schematic execution procedure of a signature automatic extraction method according to an exemplary embodiment of the present invention.

4 is a flowchart illustrating a procedure of collecting dictionary information necessary for signature automatic extraction from a flow file according to an exemplary embodiment of the present invention.

5 is a flowchart illustrating a signature generation procedure according to an embodiment of the present invention.

6 is a flowchart illustrating the signature record generation and update process according to an exemplary embodiment of the present invention.

The present invention relates to an apparatus and method for extracting a specific Internet application signature from Internet traffic, and more particularly, to an apparatus for automatically extracting a signature used to detect the traffic from traffic generated by a specific Internet application. And to a method thereof.

In this case, the Internet traffic (traffic) refers to the flow of data or data transmitted through the Internet, it can be referred to as the flow of data through the Internet network.

In addition, the flow refers to a combination of packets constituting Internet traffic in the order of collection using five attributes of a protocol, a sender address, a recipient address, a sender port number, and a receiver port number as identifiers. Thus, when traffic is transmitted from port 7070 of one computer A to port 8080 of another computer B using the TCP protocol, all packets in this traffic constitute one flow. On the other hand, the TCP traffic transmitted by B to A in response to this traffic constitutes a separate flow, in which the information of the sender address, the recipient address, the sender port number, and the receiver port number is described in the opposite direction to the flow. Because it becomes.

The signature is an identifier for identifying which traffic flow originates from which internet application, and is information extracted from a payload of a packet constituting the traffic flow. Therefore, flows A-1, A-2, A-3, A-4, ... AN, etc. generated from an application A octet value of 0x00ff00ff in the fourth byte position of the payload of the second packet. If N is a large value at a statistically significant level, this 0x00ff00ff becomes Application A's signature. This is because if the octet value appears in the fourth byte position of the second packet of the traffic flow collected from the Internet, it can be concluded that the flow originates from application A.

It is well known that a significant portion of the recent Internet traffic is occupied by so-called heavy-hitter applications such as P2P programs. Therefore, in order to prevent such traffic from entering the corporate or school network, the signature used to distinguish the traffic from an intrusion detection system (IDS), which is usually installed in the corporate or school network firewall. It is necessary to register an ID so that the intrusion detection system blocks the incoming traffic. Such intrusion detection systems are often referred to as content-based packet filtering systems.

Traditionally, in order to find signatures, many traffic analysts manually analyzed the collected traffic to find signatures. The speed of development of the applications used on the Internet is difficult to catch up with this manual analysis speed. As fast as winter, there is a great need for a method and system that can automatically detect signatures from collected traffic.

Conventional technologies, which have proposed techniques for generating signatures from Internet traffic, are signature-based detection devices for detecting the occurrence of anomalous traffic on the Internet through signatures, and anomalous behaviors to extract only packets showing anomalous behavior. And a new signature generation and verification device for generating and verifying a new signature from packets showing anomalous behavior.

Although the present invention proposes a method for automatically generating signatures, it only applies to traffic generated by a program such as a worm or virus that generates anomalous traffic on the Internet. It is not possible to say that this method is not possible, and in that it does not present a specific algorithm for generating a new signature, the contents of the invention are not specific, and thus there is a limit to the actual system.

An object of the present invention is to generate a signature for detecting a traffic originating from a specific internet application, and to automatically generate a signature by using flows generated from the traffic generated by the application as an input. And a method.

In accordance with an aspect of the present invention, an apparatus for extracting a specific Internet application signature from Internet traffic includes a file processor sequentially extracting individual flows from an Internet traffic file stored in a flow form, and the individual flows received from the file processor. A dictionary information collecting unit which inspects payloads attached to respective packets stored in the packet and collects dictionary information representing characteristics of the payload required to generate a signature; And a signature record storage unit configured to generate a signature record while iterating through the individual flows based on the prior information.

The dictionary information collecting unit may include a number of times a specific value appears in a specific byte of the payload, a total amount of all flows belonging to a packet having a specific value at a specific position of the payload, a total amount of all flows belonging to the traffic file, and The number of all packets belonging to an individual flow is recorded as prior information.

delete

The signature record storage unit may also include a histogram of a location within a signature flow, a histogram of a location within a packet of a signature, a signature string, a signature string mask, a protocol, and a packet including a packet having a corresponding signature. The sum of the number of packets having the signature and the size of flows including the packet having the signature is recorded as a signature record.

The signature record storage unit may generate a signature by inspecting the payload, and when the signature is a new signature, an element containing the signature byte using the dictionary information together with the signature. And converting the byte unit length into a signature record format and storing the converted byte.

The signature record storage unit may generate a signature list in which the signature record is stored, and if the signature generated while traversing the individual flow exists in the signature list, updates the corresponding signature record. If it does not exist in the signature list, it is characterized in that a new signature record is created in the signature list.

The method may further include a signature record converting unit converting the signature record generated by the signature record storage unit into a report form in an arbitrary format when there is no individual flow delivered from the file processing unit. .

In order to solve the above technical problem, a method for extracting a specific Internet application signature from the Internet traffic proposed by the present invention comprises the steps of sequentially extracting individual flows from an Internet traffic file stored in a flow form, and stored in the individual flows. Inspecting the payload attached to each of the packets present therein, and collecting advance information indicative of the characteristics of the payload required to generate a signature; And a signature record storing step of generating a signature record while iterating through the individual flows based on the prior information.

The collecting of the prior information may include: a number of times a specific value appears in a specific byte of the payload, a total amount of all flows belonging to a packet having a specific value at a specific position of the payload, and a total of all flows belonging to the traffic file. The total amount and the number of all packets belonging to the individual flows are recorded as prior information.

delete

In addition, the signature record storing step may include a histogram for a location in a signature flow, a histogram for a location in a signature packet, a signature string, a signature string mask, a protocol, and a packet having a corresponding signature. The sum of the number, the number of packets having the signature and the size of flows including the packet having the signature is recorded as a signature record.

The signature record storing step may include generating a signature by inspecting the payload, and storing the signature byte using the dictionary information together with the signature when the signature is a new signature. Characteristic and byte unit lengths are converted into signature records and stored.

In addition, the signature record storing step may generate a signature list in which the signature record is stored, and update the corresponding signature record when the signature generated while traversing the individual flow exists in the signature list. If it does not exist in the signature list, a new signature record is generated in the signature list.

The method may further include a signature record converting step of converting the signature record generated in the signature record storing step into a report form in any format when the extracted individual flow does not exist.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

1 is a view showing a schematic configuration of a signature automatic extraction system according to an embodiment of the present invention. Referring to FIG. 1, the signature automatic extraction device 110 includes a file processor 120 and a signature generator 130.

First, the file processor 120 sequentially extracts individual flows from the traffic file 100 in which the flows are sequentially connected according to the request of the signature generator 130, and delivers the individual flows to the signature generator 130. It performs the function. At this time, the flows are delivered to the signature generator 130 individually.

As one reference example in FIG. 1, the respective flows from the traffic file 100 composed of four flows, that is, flow 1 101, flow 2 102, flow 3 103, and flow 4 104, The process of extraction and delivery to the signature generator is presented. In flow X, X represents a sequential number numbered for each flow. Accordingly, the signature generator 130 traverses the signature file through the file processor 120.

The signature generator 130 sequentially processes the individual flows transferred from the file processor 120 to generate a signature. The generated signatures are stored in the memory until the file processing unit 120 finishes the flow extraction and delivery process, and is converted into a report form after the operation of the file processing unit 120 ends. In particular, the signature generator generates a signature by inspecting a payload attached to each packet stored in the flow.

2 is a diagram illustrating a configuration of a signature record according to an exemplary embodiment of the present invention. Referring to FIG. 2, one signature record 200 includes histogram information 210 for a location in a signature flow, histogram information 220 for a location in a packet, and a signature string 230. ), The length of the signature string 240, the protocol 250, the number of flows including the packet with the signature 260, the number of packets with the signature 270, and the packet with the signature It consists of the information of the sum 280 of the size of the flows including.

The histogram information 210 for the location in the flow of the signature is a data structure for recording the number of packets in the flow of the packet having the signature. For example, if the signature string 0x00ff00ff appears in the third of flow 1, the second of flow 2, the third of flow 3, and the fourth of flow 4, the signature 0x00ff00ff in the second packet in the flow within that data structure. From the ordered pair (2, 1) to record that is observed, from the two thirdly located packets in the flow, from the ordered pair (3, 2) to record that the signature 0x00ff00ff is observed, and from one of the fourth located packets in the flow. An ordered pair (4, 1) will be stored to record that signature 0x00ff00ff was observed.

In addition, the histogram information 220 about the position in the packet of the signature is a data structure for recording the position of the byte in the payload. For example, if the signature string 0x00ff00ff appeared once in the 0th position, 3rd in the 2nd position, and 1st in the 4th position in the payload, then the histogram information 220 for the position in the packet of the signature is (0). Three ordered pairs, (1), (2, 3) and (4, 1) will be stored.

In addition, the sum 280 of the sizes of flows including the packet with the signature is calculated for the total amount of the traffic (ie, the sum of the sizes of the individual flows) for all flows including the packets with the signature. will be.

3 is a flowchart illustrating a schematic execution procedure of a signature automatic extraction method according to an exemplary embodiment of the present invention.

The signature generator 130 collects prior information necessary to generate the signature during the first traversal of the traffic file 100 (S304), and the signature during the second traversal of the traffic file 100. Records are generated and stored in the memory (S305), and when there is no more flow transmitted from the file processor 110, the signature records stored in the memory are converted into a report form (S306). Reports can be generated in any format using signature records stored in memory.

4 is a flowchart illustrating a procedure of collecting dictionary information necessary for signature automatic extraction from a flow file according to an exemplary embodiment of the present invention.

The prior information collection process (S304) required for signature generation of FIG. 3 initially initializes the values of two-dimensional array variables PACKET_COUNTERS and FLOW_BYTE_COUNTERS and general variables TOTAL_FLOWS_IN_BYTES and TOTAL_PACKETS to 0 (S401). PACKET_COUNTERS and FLOW_BYTE_COUNTERS are two-dimensional arrays of size 64 x 256, respectively.

Then, the flow is requested to the file processing unit (S402), and if the response that there is no remaining flow is transmitted from the file processing unit 110, the execution ends, otherwise the flow received from the file processing unit 110 Read F (S404). Thereafter, the temporary two-dimensional array TMP_MATRIX to be used in the process of flow F is initialized, the total number of packets contained in F is added to TOTAL_PACKETS, and the size of F is added to TOTAL_FLOWS_IN_BYTES (S405). TMP_MATRIX has a size of 64 x 256. Then, it is checked whether the flow F is a data flow (S406).

If no value is given, a flow consisting of 100 or more packets and having an average size of 1000 or more packets is considered a data flow. If no value is given, 10 is substituted into BOUND for data flow (S407), otherwise, the number of packets belonging to F is substituted (S408). It then checks the first BOUND packets in flow F in order.

In this process, each byte of the payload belonging to each packet is checked in order. If the byte being checked is called the J th byte of the payload and the value of that byte is B (where B is greater than or equal to 0 and greater than or equal to 255), PACKET_COUNTERS [J] [B] Is increased by 1, and the value of TMP_MATRIX [J] [B] is set to 1 (S413). After all of the above steps have been completed and all the packets in the flow F have been examined, the value of FLOW_BYTE_COUNTERS [J] [K] is assigned to the size of F for every (J, K) th element whose value is stored in TMP_MATRIX. Increase by (S420).

After the process is completed for all flows stored in the flow file 100, four variables, PACKET_COUNTERS, FLOW_BYTE_COUNTERS, TOTAL_FLOWS_IN_BYTES and TOTAL_PACKETS, contain information to be used in the actual signature generation process (S305) of FIG.

5 is a flowchart illustrating a signature generation procedure according to an embodiment of the present invention.

The signature generator first generates a new two-dimensional array SIGMATRIX using the information generated in the prior information collection process S304 of FIG. 3. SIGMATRIX measures 64 x 255.

The value of the (I, J) th element SIGMATRIX [I] [J] of SIGMATRIX has a value of PACKET_COUNTERS [I] [J] greater than 10 and the value of FLOW_BYTE_COUNTERS [I] [J] is 0.9 and TOTAL_FLOWS_IN_BYTES. It is set to 1 only when it is equal to or larger than the value obtained by multiplying, otherwise it is set to 0 (S505). The above values such as 10 or 0.9 are the default values unless otherwise specified. After the above process is completed, the signature generation unit requests a flow to a file processing unit (S508) to bring the flows stored in the traffic file 100 in order (S510).

After reading the new flow F (S510), the payload belonging to each packet in the flow F is checked as follows. First, the array SIGNATURE to hold the signature to be created, the array SIGNATURE_MASK used to mark the elements of the array SIGNATURE that actually contain the SIGNATURE bytes, and the length in bytes of the signature to be created. The value of the variable SIGNATURE_LEN is initialized (S514).

Then, for every byte position J in the payload of that packet, if the value of SIGMATRIX [J] [B] is nonzero when the value of that byte is B, then assign B to SIGNATURE [J] and replace SIGNATURE_MASK [J]. Set the value to 0xFF and increase SIGNATURE_LEN. Otherwise, SIGNATURE [J] is set to 0 and the value of SIGNATURE_MASK [J] is set to 0 (S518).

If the signature created by the above process is a new signature, the signature is converted into the signature record of FIG. 2 and inserted into the signature list. If the signature is not a new signature, the signature already exists in the signature list. The statistics stored in the signature record are updated (S520). The above process is repeated sequentially for all packets of all flows.

6 is a flowchart illustrating the signature record generation and update process according to an exemplary embodiment of the present invention.

If there is no signature record having the same signature string and protocol number as the signature stored in the array SIGNATURE of FIG. 5 in the signature list, a new signature record is created and inserted (S602). At this time, an ordered pair (I, 1) is substituted into the histogram 210 for the position in the signature flow in the signature record. In addition, an ordered pair (J, 1) is stored in the histogram 220 for the location in the packet of the signature.

The value stored in the array SIGNATURE used in FIG. 5 is stored in the signature string 230, the value stored in the array SIGNATURE_MASK used in FIG. 5 is stored in the signature string mask 240, and the signature is extracted in the protocol 250. The protocol value used by the received packet is recorded. The number of flows 260 including the packet having the signature is 1, the number 270 of the packet having the signature is 1, and the packet having the signature is stored. The size of the flow F used in FIG. 5 is recorded in the size sum 280 of the included flows.

On the other hand, if there is a signature in the signature list having the same signature string and the same protocol number as the signature recorded in the array SIGNATURE of Fig. 5, the histogram (21) for the position in the flow of the signature in the signature record is shown. If there is an ordered pair (I, X) for I, it is searched for and updated to (I, X + 1) if present, and (I, 1) is inserted if not present (S605).

In addition, the histogram 220 for the position in the packet of the signature in the signature record is searched for the existence of an ordered pair (J, Y) for M, and if present, updated to (J, Y + 1), If it does not exist, a new ordered pair (J, 1) is inserted (S605). In addition, the number of flows 260 including the packet having the signature in the signature record is increased by 1 only when the signature record has never been visited in the process of processing the packet of the flow F (S607).

In addition, the value of the number of packets 270 with signatures in the signature record is increased by 1, and the value of the sum of sizes 280 of flows including packets with signatures in the signature record is the corresponding signature. The record is incremented by the size of F only if it has never been visited in the process of processing the packet of the flow F (S607).

The invention can also be embodied as computer readable code on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, which are also implemented in the form of carrier waves (for example, transmission over the Internet). It also includes. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

As described above, optimal embodiments have been disclosed in the drawings and the specification. Although specific terms have been used herein, they are used only for the purpose of describing the present invention and are not intended to limit the scope of the invention as defined in the claims or the claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

As described above, according to the present invention, the signature can be automatically extracted for the traffic generated by a specific Internet application program.

Such a signature automatic extraction apparatus and method may be useful for network managers interested in automatically extracting signatures to be used for IDS to create signatures for blocking traffic flowing into their networks.

Claims (18)

delete A file processor that sequentially extracts individual flows from an internet traffic file stored in a flow form; A dictionary information collection unit which inspects payloads attached to respective packets stored in the individual flows received from the file processing unit, and collects dictionary information indicating characteristics of the payloads required to generate a signature; And And a signature record storage unit configured to generate a signature record while circulating the individual flows based on the prior information. The method of claim 2, The pre-information collecting unit includes the number of times a specific value appears in a specific byte of the payload, the total amount of all flows belonging to a packet having a specific value at a specific position of the payload, the total amount of all flows belonging to the traffic file, and the individual flow. Apparatus for extracting a specific Internet application signature from the Internet traffic, characterized in that the number of all packets belonging to and recorded as the prior information. The method of claim 2, The signature record storage unit includes a histogram for a location in a signature flow, a histogram for a location in a packet of a signature, a signature string, a signature string mask, a protocol, and a flow number including a packet having a corresponding signature. And recording the sum of the number of packets with signatures and the size of flows including the packets with signatures as a signature record and recording a specific Internet application signature from Internet traffic. The method of claim 2, The signature record storage unit generates a signature by inspecting the payload, and when the signature is a new signature, an element and a byte containing the signature byte using the dictionary information together with the signature. Apparatus for extracting a specific Internet application signature from the Internet traffic, characterized in that the unit length is converted into a signature record form and stored. The method of claim 2, The signature record storage unit generates a signature list in which the signature record is stored and updates the corresponding signature record when the signature generated while traversing the individual flow exists in the signature list. And generate a new signature record in the signature list if it does not exist in the signature list. The method of claim 2, And a signature record converting unit converting the signature record generated by the signature record storage unit into a report form in an arbitrary format when there is no individual flow delivered from the file processing unit. A device that extracts specific Internet application signatures from Internet traffic. delete Sequentially extracting individual flows from an internet traffic file stored in a flow form; Inspecting the payload attached to each of the packets stored in the respective flows and collecting prior information indicative of the characteristics of the payload required to generate a signature; And And a signature record storing step of generating a signature record while iterating through the individual flows based on the prior information. The method of claim 9, The collecting of the prior information may include the number of times a specific value appears in a specific byte of the payload, the total amount of all flows belonging to a packet having a specific value at a specific position of the payload, the total amount of all flows belonging to the traffic file, and And extracting a specific internet application signature from the internet traffic, wherein the number of all packets belonging to each individual flow is recorded as prior information. The method of claim 9, The signature record storing step may include a histogram for a location in a signature flow, a histogram for a location in a packet of a signature, a signature string, a signature string mask, a protocol, a number of flows including a packet having a corresponding signature, And recording the sum of the number of packets having the signature and the size of the flows including the packet having the signature as a signature record and recording the specific Internet application signature from the Internet traffic. The method of claim 9, The signature record storing step may include generating an signature by inspecting the payload, and when the signature is a new signature, an element containing a byte of the signature using the dictionary information together with the signature; and A method of extracting a specific Internet application signature from Internet traffic, comprising converting and storing a byte length into a signature record format. The method of claim 9, The signature record storing step generates a signature list in which the signature record is stored and updates the corresponding signature record when the signature generated while traversing the individual flow exists in the signature list, and updates the signature. Creating a new signature record in the signature list if it does not exist in the signature list. The method of claim 9, A signature record converting step of converting the signature record generated in the signature record storing step into a report form of any format when the extracted individual flow does not exist. How to extract specific internet application signatures. delete 10. The method of claim 9, wherein collecting the dictionary information A two-dimensional array variable having a size of 64 × 256 that stores the number of times a specific value appears in a specific byte of the payload, and the total amount of all flows belonging to a packet having a specific value in a specific position of the payload, and belongs to the traffic file Initializing the values of the first general variable storing the total amount of all flows and the second general variable storing the number of all packets belonging to the individual flows; When the individual flow is delivered, initializes a two-dimensional array variable for temporarily storing data having a size of 64 × 256, adds the total number of packets included in the individual flow to the initialized second general variable, and adds the first general variable. Adding a size of the delivered individual flow to a variable; In the case of the data flow by checking whether the individual flow is a data flow, the first 10 packets of the individual flow and the payload belonging to each packet are sequentially examined. Increasing the number of times a particular value appears in a particular byte and the value of an array variable that temporarily stores the data; And After the inspection of all the packets in the delivered individual flow is finished, all the flows belonging to a packet having a specific value at a specific position of the payload for all elements whose value of the array variable for temporarily storing the data is not '0'. Increasing the total amount of P by the size of the individual flow; and extracting a particular Internet application signature from the Internet traffic. The method of claim 9, (a) the value of the two-dimensional array variable having a size of 64 × 256, in which the number of times a specific value appears in a specific byte of the payload generated in collecting the dictionary information is greater than '10' and the payload is specified. The value of a two-dimensional array variable having a size of 64 × 256 stored in the total amount of all flows belonging to a packet having a specific value at a position is greater than the value obtained by multiplying '0.9' by the total amount of all flows belonging to the traffic file. Generating a two-dimensional array variable storing '1' if it is equal to or greater than '0' otherwise; (b) a first array variable for storing a signature to be generated for each packet in the respective flow and a second array variable for indicating an element containing a signature byte of each element of the signature stored in the array variable; And initializing a third variable that stores the length in bytes of the signature. (c) if the value of the first array variable is not '0' when all byte positions in the payload of the packet have a specific byte value, store the specific byte value in a corresponding element of the first array variable; Store '0xFF' in the corresponding element of the second array variable and increase the third variable; otherwise, the corresponding element of the first array variable and the corresponding element of the second array variable are '0' Storing; And (d) If the signature corresponds to a new signature, convert the signature into a signature record form and insert the signature into the signature list.If the signature is not a new signature, the statistics stored in the signature list are updated. And further extracting a specific internet application signature from the internet traffic. 18. The method of claim 17, wherein step (d) If there is no signature record having the same signature string and protocol as the signature in the signature list, the signature list has a histogram for the location in the flow of the signature, a histogram for the location in the packet, a string, and a signature. Generating the signature record in which a sum of a string string mask, a protocol, a number of flows including a packet with a signature, a number of packets with a signature and a size of flows including a packet with a signature is recorded; And When a signature record having the same signature string and protocol as the signature exists in the signature list, a sequence pair for the histogram for the position in the flow of the signature in the signature record exists. Update to the next ordered pair, create the first ordered pair if none exists, If there is an ordered pair for the histogram of the location in the packet of the signature in the signature record, the existing ordered pair is updated. The number of flows that contain packets with signatures in the signature record is increased only if the signature record has never been visited in the course of processing the packets of the individual flows. Increase the number of packets with signatures in the signature record, The sum of the sizes of flows containing packets with signatures in the signature record is increased by the size of the individual flows only if the signature record has never been visited in the course of processing the packets of the individual flows. Step; extracting a particular internet application signature from the internet traffic.

Images