KR100877911B1 - Method for detection of p2p-based botnets using a translation model of network traffic - Google Patents

Abstract

A method for detecting a P2P(Peer to Peer)-based bot-net by using a network traffic transfer model is provided to detect a bot-net quickly and correctly by generating and matching a detection model with a suspicious traffic in the P2P network. A traffic flow group is generated by collecting and clustering network connection flow on the network. Information, which represents that a state of the traffic flow group is changed, is collected and a multi-frequency matrix is formed and trained based on the collected information. A multi-state transfer probability model is obtained by generating a probability matrix for all attacks of the bot-net and the probability matrix for normality through a learning process. The probability model is matched with the model for the state transfer information, which is collected from the traffic of the network in real time. It is detected whether the malicious bot-net is found by comparing a model matching result value with a preset threshold.

Description

Method for detection of P2P-based botnets using a translation model of network traffic}

The present invention relates to a P2P based botnet detection method using a network traffic transition model. In detail, the present invention finds and models the attack characteristics of the botnet by extracting and analyzing the behavior-based information of the P2P botnet at the network level in order to formalize the gradual behavior characteristics that appear from the initial formation and construction of the botnet. The present invention relates to a peer-to-peer-based botnet detection method using a network traffic transition model that can quickly and accurately cope with the damage of a P2P-based network system caused by a botnet by providing a detection process through matching.

Since the emergence of the Morris worm, the first Internet worm virus, in 1988, there have been constant attacks on the Internet by many worms. In contrast to malicious behaviors that have been used for no specific purpose in the past and attacks that have only been rapidly spreading, many worms that have been discovered in recent years are trying to collect and leak personal information for financial gain. It is evolving to give the attacker control over the infected system to exploit as a waypoint for the attack. As a result of this trend, a new type of attack called a malicious bot is derived from an existing worm, and the damage has soared, and research for active countermeasures is being actively conducted.

Systems infected by malicious bot viruses have built their own network, the botnet, which has evolved into a threat that can rapidly and effectively spread various types of attacks such as spam mail and distributed denial of service attacks. An attacker needs a separate botnet command and control (C & C) server to maintain bot infected systems, and has been using IRC (Internet Relay Chat) server for this purpose. In this case, individual bots are IRC clients. Acts as.

Currently, there are more than 7,000 variants of bots due to the leak of source code, and even antivirus companies are unable to detect all bots. There are more than 1,500 botnet C & C servers already known worldwide, and the 2007 World Economic Forum (WEF) warned that the `` net pandemic '' is the biggest threat to the future of the Internet. Of the 600 million PCs, about 100 million to 150 million are already used as botnets.

In order to prevent such botnet damage, a botnet has been established since 2005 to cope with malicious bots by operating DNS sinkholes with domestic ISP / IDC operators and overseas CERT teams to control bot-infected PCs. It is blocking the connection to the C & C server. DNS Sinkhole is a policy that reverses the process by which a bot attempting to connect to a botnet C & C server queries a domain name to obtain a server's dynamic IP address. This is a technique to prevent the process that is connected to the source. In this case, the control model by the botnet C & C server is a fatal weakness of the botnet.

However, as the number of bots blocked by these methods increased, attackers increasingly needed botnets that didn't require servers, which eventually led to the emergence of peer-to-peer-based botnets. Building a peer-to-peer botnet model can be difficult to spread and control, but it is popular among bot developers because it offers the great benefit of building a network without relying on a centralized server. It is expected to increase further.

In general, virus detection techniques at the network level include quantitative analysis of incoming traffic, signature analysis of data areas for individual packets, and deep packet inspection (DPI) techniques. In addition, attempts have been made to extract and analyze network traffic information in the unit of integrated information and flow of multiple packets for universal intrusion detection. In botnets, however, a large number of distributed traffic generation features, such as worms, and independent and specialized traffic generation characteristics due to communication with attackers or individual peers coexist. The spread of worms is a step in the growth of botnets, and if you don't see big flows, you may be mistaken for spreading malware. In addition, communication traffic between peers generated by P2P-based botnets may involve encryption, thus excluding dependence on data areas.

Therefore, in order to detect P2P-based botnets more quickly and accurately, analysis by step-by-step definition and classification of comprehensive botnet traffic and detection methods irrespective of the characteristics of the data area are required.

The present invention has been invented to meet the above requirements.

Accordingly, the present invention generates a detection model according to the traffic characteristics of a P2P-based botnet, and uses the generated detection model to detect botnets more quickly and accurately by matching suspected traffic on a P2P network. The purpose is to provide a method.

In order to achieve the above object, the present invention collects and aggregates traffic connection flows on a network to generate a traffic flow group, and then aggregates information to which the state of the traffic flow group is transferred, and multi-frequency matrix based on the aggregated information. Forming a and performing a learning process; Obtaining a multi-state transition probability model by generating a probability matrix for all attacks of the botnet and all normal probability matrices through the learning process; Performing model matching on the state transition information collected from the probability model and the traffic of the network in real time and comparing the result of model matching with a preset threshold to detect whether a malicious botnet exists; .

In addition, the multi-state transition probability model classifies the probability model group according to the traffic state transition for the already discovered botnet and the probability model group according to the traffic state transition for the undetected botnet, and performs the model matching. When detecting the existence of a botnet, if the detected botnet belongs to a model group of traffic state transitions to the already discovered botnets, it is considered a known attack and is considered a misuse detection. Otherwise, the detected botnet is found on an undetected botnet. It is characterized in that the traffic state war for the traffic belongs to the model group to determine it as an unknown attack and regard it as an abnormal detection.

Here, the multi-state transition probability model is generated by a Markov chain process, and the model matching is performed by a log-likelihood ratio test for calculating a ratio of normal likelihood for generalization of likelihood.

As described above, the present invention can provide a safe and stable P2P network environment by detecting the botnet more quickly and accurately on the P2P network, so that it can quickly respond to the damage caused by the botnet. You get an effect.

An embodiment to which the present invention as described above is applied will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for performing a botnet detection process according to the present invention.

Referring to the drawings, the system consists of a learning module, a multi-state transition probability model, and a detection module. Such a configuration can be built into a program by a computer connected to the P2P network.

The learning module collects and aggregates the traffic connection flows on the network to generate a traffic flow group, and then aggregates information to which the state of the traffic flow group is transitioned, and forms a multi-frequency matrix based on the aggregated information. Will perform.

The multi-state transition probability model is formed by generating a probability matrix for all attacks of the botnet and all normal probability matrices through the learning process. In particular, such a probability model may be classified into a probability model group according to a traffic state transition for an already discovered botnet and a probability model group according to a traffic state transition for an undiscovered botnet.

The detection module detects whether or not a malicious botnet exists by performing a model matching on the state transition information collected from the traffic of the network and the state transition probability model in real time by comparing the result of model matching with a threshold value already set. Done.

The process of detecting the botnet by the system will be described in detail with reference to the accompanying drawings.

1. Describe the process for performing the stepwise reduction of network traffic.

In a peer-to-peer botnet, a single peer bot generates a large amount of phased traffic according to the number of peer discovery and information exchange processes in order to connect and form a network with as many external peers as possible. In other words, traffic having the same flow scheme is continuously generated with time difference.

Figure 2 shows the packet growth that a known SpamThru bot communicates with individual peers and generates every five minutes. Although there is a time difference according to the order of connection, it can be seen that the models of traffic variation are similar.

They need to be properly classified and group similar traffic by the staged behavioral characteristics that emerge from the initial formation and deployment of botnets. In this paper, to reduce the massive amount of TCP and UDP traffic generated by P2P-based botnets into clusters of phased connection flows, we generate a pattern for each individual traffic and perform clustering based on a similarity comparison algorithm.

end. Network traffic link flow split

The amount of traffic that P2P applications take today is already well above that of HTTP and FTP. In particular, with the development of P2P sharing programs, network traffic has brought a great change in the direction and type of the flow. Characteristic segmentation of unit traffic flows is required for such traffic analysis.

1) TCP connection flow split

TCP is a connection-oriented protocol. Basically, a session-managed transmission unit is provided for a trusted transmission system. This is a relatively easy three-way handshake for the session establishment process and a procedure for terminating the connection, making connection flow splitting relatively easy.

But for more accurate traffic analysis, it is necessary to distinguish between protocol flooding attacks and simple packet retransmissions. 3 (a) shows that the same SYN packet is repeatedly transmitted in the initial connection establishment procedure. This case is of significant importance as part of the SYN packet flood attack attribute, so even a single packet should be separated into one independent traffic flow. However, if a response by the SYN and ACK packet is subsequently received from the receiver as shown in (b) of FIG. Improved accuracy

2) UDP connection flow split

The UDP protocol does not follow a reliable transport protocol like TCP. Therefore, since the criteria to be used for traffic link flow segmentation are ambiguous, some difficulties are inherent. In this study, UDP traffic having the same socket pair is divided by time unit.

Of course, since the same characteristics as in TCP may be present, the meaning of the connection request packet retransmitted and duplicated as shown in (a) of FIG. 4 is later canceled by receiving a response as shown in (b) of FIG. It is classified as a normal retransmission packet and loses its existence in the traffic link flow segmentation process.

I. Comparison of Similarity of Traffic Flow Units

Based on the traffic link flow segmentation scheme of the network protocol, we generate a pattern for each traffic link flow. By comparing the similarities of these patterns, we attempt to extract the state transition information of the entire P2P botnet traffic by clustering the same class of connection flow patterns, defining the states of these clusters, and analyzing the transition process.

1) Create traffic link flow pattern

In the present embodiment, the unit network flow to be distinguished means an active connection between two computers for one purpose of communication. Through this connected flow, computers can exchange data with each other and perform and process the smallest unit of communication.

Table 1 shows the metrics used for traffic link flow analysis. The TCP protocol is a connection-oriented protocol that can distinguish each flow relatively accurately using its transport characteristics. The initial connection setup of the TCP protocol is accomplished by a three-stage handshake. In addition, the connection termination indicating the end of the session recognizes that the communication is terminated by exchanging a packet having the FIN or RST bit set in the flag field between the hosts. On the other hand, the UDP protocol is excluded because no flag exists.

TCP UDP Measure Range of values Measure Range of values Transmission direction 0, 1 Transmission direction 0, 1 flag 1 to 63 Data length 0 to MTU Data length 0 to MTU

 Table 2 lists one TCP traffic connection flow for each SMTP service sent and received on port 25 of the server based on each measure. During the first three phase handshake, the connection request packet record that was retransmitted prior to connection establishment is removed to distinguish between protocol flooding attacks and packet retransmissions. In addition, the ACK response packet is ignored, and if the packet having the FIN or RST flag is transmitted to terminate the connection, the subsequent traffic is ignored. This is because there is no difference in common properties of all TCP traffic.

(Transmission direction) : (flag) : (Data size) pkt__1_ 0 : 2 : 0 ← SYN 0 : 2 : 0 ← SYN pkt__2_ One : 18 : 0 → SYN, ACK 0 : 16 : 0 ← ACK pkt__3_ One : 24 : 84 → PSH, ACK 0 : 16 : 0 ← ACK pkt__4_ 0 : 24 : 26 ← PSH, ACK pkt__5_ One : 24 : 26 → PSH, ACK pkt__6_ 0 : 24 : 26 ← PSH, ACK pkt__7_ One : 24 : 48 → PSH, ACK pkt__8_ 0 : 24 : 41 ← PSH, ACK pkt__9_ One : 24 : 48 → PSH, ACK pkt_10_ 0 : 24 : 35 ← PSH, ACK pkt_11_ One : 24 : 34 → PSH, ACK pkt_12_ 0 : 24 : 6 ← PSH, ACK pkt_13_ One : 24 : 50 → PSH, ACK pkt_14_ 0 : 24 : 1024 ← PSH, ACK One : 16 : 0 → ACK pkt_15_ 0 : 24 : 295 ← PSH, ACK pkt_16_ One : 24 : 19 → PSH, ACK pkt_17_ 0 : 24 : 6 ← PSH, ACK pkt_18_ One : 24 : 24 → PSH, ACK pkt_19_ 0 : 17 : 0 ← FIN, ACK One : 16 : 0 → ACK One : 17 : 0 → FIN, ACK 0 : 16 : 0 ← ACK

<Protocol> | <Start time> | <Internal port> _ <external port> | <Transmission and reception pattern> Header Contents T | 1179824184.443993 | 1379_250 : 2: 0-0 + 1,1: 18: 0-0 + 2,1: 24: 19-84 + 3; 5; 7; 9; 11; 13; 16; 18,0: 24: 6-295 + 4; 6; 8; 10; 12; 15; 17,0: 24: 1024-1024 + 14,0: 17: 0-0 + 19

In this embodiment, such unit traffic is formalized into a sequential pattern. The formal pattern is divided into a header part and a content part. The header part includes a protocol classification, a time when a connection request is started (microseconds), and a port number between both ends of communication. The content part includes packets of the actual traffic. Sequential flow information is sorted based on the scale values. Table 3 shows an example of sequential pattern generation for the connection flow illustrated in Table 2. The sequential pattern is constructed based on the syntax symbols shown in Table 4. The content part is basically constructed by listing the occurrence sequence numbers of the same packets after the '+' sign.

sign meaning | Header attributes and content separators _ Header symbol inside / outside port of header part : Each measure delimiter in the packet - Scale delimiter for scales with consecutive attribute values + Delimiter for Listing Packet Sequence Numbers ; Each packet sequence number separator , Separators for Different Packets

2) Similarity Measurement Function for Sequential Data

In the previous section, we looked at how to format the sequential patterns for each traffic flow. This is ultimately used as a comparison unit for similarity test. In this paper, we use the gap penalty matrix based on dynamic programming to compare the similarity between unit traffic flows.

For example, two sequences consisting of a set of overlapping elements S _i = {A, B, G, C, C, B, A, E, C, F} and S _j = {C, B, C Suppose there are, D, D, E, E, B, D, C, F, F}. To illustrate the differences in word order between two different languages, the concepts of forward alignment (Equations 1 and 5) and reverse alignment (Equations 2 and 6) can be applied.

Regular alignment: i → j = S _i

Inverted alignment: j → i = S _j

On the other hand, the embodiment proposes a combined alignment method because an element of one sequence cannot be exactly one-to-one matched to an element of another sequence on the alignment path due to the nature of the packet flow. As shown in Fig. 7, this alignment scheme is symmetrical. Here, an arrow connected to two B ¹ has a double arrowhead for symmetric matching.

This mixed alignment scheme requires cross alignment for independent partial orders, such as elements such as B ¹ | C ¹ and B ² | E ¹ . Thus, the following elements C ² and C ³ have a path to the previous element duplicated as shown in FIG. 8. This makes it difficult to assign scores for comparing similarities between two sequences. This problem is solved based on predefined rules.

In order to compare the similarity of two sequences, the method of applying the similarity comparison method of categorical data is effective. The similarity comparison function for the sequences S _i and S _j is shown in Equation 3.

The similarity between the two sequences can basically be obtained as the ratio of union and intersection. However, one thing to consider here is that when you look at the sequence you want to compare as a set, the elements of the set are in order. Therefore, the intersection of sequences cannot be obtained simply by the number of elements overlapping each other, but should include the processing of the order. To solve this problem, the similarity comparison algorithm proposed in this paper applies a dynamic programming technique.

To calculate the intersection, we first construct a gap deduction matrix. In FIG. 9, a score calculation method is shown for each intersecting block. The score of the final intersection is determined by the sum of these scores. If block (x _prev , y _prev ) is the previous crossed block of block (x, y), then (x _prev <x) and (y _prev <y) must be satisfied. The score is found as the inverse of the distance from the previously crossed block. In this paper, we use the concept of gap deduction to calculate distance.

,

,

BS _{(x, y)} means Block Scoring of the intersection block (x, y). D _{(x, y)} is the distance from the nearest previous intersection block (x _prev , y _prev ) to block (x, y). This distance also means the number of blocks between (x _prev , y _prev ) and (x, y). Rely on the three rules defined in Table 5 to find the nearest previous intersection block (x _prev , y _prev ) of the current intersection block (x, y).

number rule Rule 1 The previous matched coordinate is | x _any * y _any | The value is | x * y | The coordinate that has the maximum value among all values smaller than the value. Rule 2 If two or more previously matched coordinates are | x _any * y _any | If you have a value, among them | x _any + y _any | The coordinate with the larger number is finally selected as the previously matched coordinate. Rule three If x _any * y _any | Value and | x _any + y _any | If two or more previously matched coordinates with the same value are found, follow the next two steps of calculation again. Rule 3-1 If the x and y coordinates are the same, the deduction value is the same no matter which previous matched coordinate is selected. Rule 3-2 If the x coordinate is not equal to the y coordinate, then | x-x _any | Value and | y-y _any | The value is calculated to extract the larger of the two, and finally the coordinate with the minimum of all extracted values is finally selected as the previously matched grid.

In FIG. 9, the initial intersection block is (0, 0) and BS is 0 at this time. By rule 2, the nearest previous intersection block in the intersection block (5, 3) becomes the block (4, 1). In addition, according to Rule 3-2, the nearest previous intersection block of the intersection block 9, 10 becomes the block 8, 6. The intersection blocks 10 and 11 have a value of 1 as the maximum BS . This is because element F intersects immediately after element C intersected on both sequences.

As such, comparison scoring is based on gradual transitions from previous sequences.

Therefore, SCS (Sequence Comparison Scoring) is equal to the sum of all BSs , and SCS of the two sequences S _i and S _j is 2.56 (= 1/4 + 1/4 + 1/2 + 1/5 + 1/9 + 1 / 4 + 1), which is the number of intersections you want to find.

The similarity value according to Equation 6 has a value between 0 and 1. Here, the union is used for generalization of the similarity comparison function. The number of unions is simply equal to the number of overlapping elements of the entire union as in FIG.

Therefore, finally, the similarity of the two sequences S _i and S _j is 0.17 (= 2.56 / 15). This similarity comparison method between sequences is applied to the sequential pattern as it is, Table 6 shows the algorithm of the similarity measurement function proposed in this embodiment.

procedure SeqSim ( S _i , S _j ) begin set I [ x , y ] for every intersected blocks between S _i and S _j for each ( x , y ) ∈ I do { ( x _prev , y _prev ): = select_nearest_prev-intersected_block ( x , y ) D : = compute_distance ( x , y , x _prev , y _prev ) BS : = reciprocal ( D ) intersection : = intersection + BS } union : = size of bag { S _i ∪ S _j } similarity : = intersection / union end

All. Representative Traffic Flow Group Generation by Clustering

In this embodiment, in order to effectively reduce a myriad of traffic connection flows, clustering of each connection flow sequential pattern was performed. In this case, ROCK, which is one of data mining clustering algorithms, was used. ROCK enables clustering of categorical data rather than numerical data, and forms clusters based on the connection relationship of each cluster. In this case, the closer the clusters are to each other, the more secure the connection is to each other. That is, when two clusters are above a certain threshold in terms of similarity by the similarity function presented above, the neighbors are determined to be neighbors, and the number of common neighbors between the clusters is defined as the number of connections between the two clusters. Subgroups belonging to the same cluster generally have a large number of common neighbors and a large number of connections at the same time. Therefore, when merging clusters, the ones with the most connections are merged first to create meaningful clusters. Here, a goodness was proposed for the merging of clusters and scaled up by using random sampling.

The ROCK algorithm generates an adjacency matrix as shown in FIG. 11 as a preprocessing step for clustering, which is composed of results from numerical values obtained by comparing similarities between all clusters. If the similarity value of the two sequential patterns is smaller than the set threshold value θ, it has a value of 0, and if it is greater than or equal to the threshold value, it has a value of 1. In this way, the adjacent matrix becomes a matrix composed only of values of zero or one. In addition, the threshold value at this time is used again for fitness calculation.

Based on the generated neighbor matrix, a link matrix is again generated as shown in FIG. 12.

An example of a connection haengryeong generation process, the connection can link [D, F] for the cluster D and F are obtained by the following calculation procedure:

link [F, D] = adjacency matrix (horizontal axis) x adjacency matrix (vertical axis)

= 0x0 + 0x0 + 1x1 + 1x0 + 1x1 + 0x1 = 2

Subsequently, clustering is repeated based on the connection relationship between the clusters. At this time, one cluster is reduced per clustering stage. This process is repeated until clustering is no longer possible, which means it is time for a connection to not be established between all clusters.

13 shows a process in which clusters are formed after clustering is performed. Two clustered clusters share a number of connections and reshape common connections with other clusters that were connected to each other.

In this case, the ROCK algorithm uses the goodness of fit, not the number of connections for clustering. Clustering by simple concatenation is repeated because the larger the cluster, the greater the number of connections with neighboring clusters, and the larger cluster continues to grow in the clustering process. In order to prevent this phenomenon, as in Equation 7, fitness is defined by generalizing the number of connections.

n _i and n _{j denote the number} of cross-connections, and θ applies the threshold that was previously applied to generate the adjacent matrix. In this paper, the function f (θ) is declared as (1-) / (1+).

2. Describe network traffic transition model and botnet detection process.

end. Extract individual traffic state information

A method of obtaining a single packet state from field information of a protocol header may be used for intrusion detection. The state information has a 9-bit state value consisting of Dir (transmission direction), TCP-flag , and IP-flag as shown in Tables 7 and 8. The transmission direction of the packet is binary-divided into (Server → Client) and (Client → Server) directions.

In this embodiment, the state information of the representative traffic flow group is extracted, not individual packets. In this case, the state value is composed of 7 bits.

property Measure Scale value Configuration status protocol protocol TCP (0) PT UDP (1) port Internal port Random Ports (0) LP Reserved Ports (1) External port Random Ports (0) RP Reserved Ports (1) Trust standard Minimum connection flow quantity (0) OP Minimum connection flow quantity (1) traffic Successful connection Unidirectional Communication (0) RS Bidirectional Communication (1) Packet Count Comparison Internal> = external (0) PC Inside <outside (1) Byte count comparison Internal> = external (0) DC Inside <outside (1)

7 bits of status information Configuration protocol port traffic Configuration status PT LP RP OP RS PC DC Configuration value 64 32 16 8 4 2 One

A traffic flow group that owns a sufficient amount of connection flows communicates over the UDP protocol and consists of transport traffic between an internal random port and an external specific reserved port. When the two-way communication is performed according to the request success and the number of packets and bytes of traffic flowing out from the inside is larger than the number of packets and bytes flowing from the outside, the status value of the representative traffic flow group can be obtained as shown in FIG.

In this case, the state of the traffic flow group represented by the bit string is 1010100 _(2), that is, it has a 84 ₍₁₀₎ state value. In this way, each cluster can be represented by 128 (2 ⁷ ) state values from 0 to 127.

I. Multistate Transition Probability Modeling

1) Traffic state transition information

The state transition matrix is generated based on state information. Stores transition information of an observed state changed from a previous state in a traffic flow. The matrix consists of the rows of the previous state and the columns of the current state. This matrix contains information about the statistical frequency of each state transition. FIG. 15 shows a frequency matrix for transition information of a state in which traffic flows as follows.

(64 → 42 → 84 → 126 → 21 → 126 → 21 → 86 → 21 → 23 → 84)

In this way, one traffic flow has a frequency matrix for the state transition information and undergoes a statistical learning process based on the state transition information.

2) Transition model based on Markov chain process

The method of using frequency behavior has the drawback of maintaining separate matrices for all traffic flows. Therefore, it is necessary to propose a single state transition matrix generation method based on Markov model theory for all flow data.

The Markov model includes transition probabilities for each state. Based on the Ergodic model as shown in FIG. 16, a special state transition model is proposed. The model implies statistical causality between the states of sequentially occurring packets in a traffic flow.

The transition probability from the previous state p to the current state o is defined as follows.

, and

, and

In Equation 8, N means the total number of possible states. And the probability of the initial state p is defined as

,

,

On the other hand, the probability of transition is calculated by the frequency, which is

,

,

은 이전 상태에 이어 발생하였던 모든 현재 상태 총 빈도수를 의미한다. 이는 이전 상태들과의 관계를 나타내기 위하여 적용된다. 만일 모든 이전 상태들의 총 빈도수에 의하여 나누어 진다면, 그것은 모든 이전 상태들에 대한 현재 상태로서의 관계성을 지니게 될 것이다. 그러나 위 연산은 수학식 8의 전이 확률에 반하는 의 결과를 초래한다.In Equation 10,

Means the total frequency of all current states that occurred after the previous state. This applies to indicate the relationship with previous states. If divided by the total frequency of all previous states, it will have a relationship as the current state to all previous states. However, the above operation results in the opposite of the transition probability of Equation 8.

The probability of the sequence S = { S ₁ , S ₂ , ..., S _T } may be defined as shown in Equation 11 by the Markov characteristic,

In other words, the equation (12) can be obtained.

The probability defined in Equation 12 becomes likelihood, which can be used as a reference for model recognition. Here, the probability of the initial state p is to be ignored. Thus, likelihood L can be defined as

However, likelihood will cause underflow by multiplication of probabilities. Therefore, as a result, a log-likelihood as shown in Equation 14 is defined.

Log-likelihood is the sum of the transition log-probabilities for the states. This is an optimal criterion for model matching, and the model thus constructed includes multi-state transition information expressed as a probability matrix.

17 shows an example of generating a probability matrix. For example, the following three traffic flows can have a total of four types of states (0, 1, 2, 3).

 Traffic link flow # 1 = {2,2,2,3,3,1}: 2 → 2 → 2 → 3 → 3 → 1

 Traffic link flow # 2 = {2,2,2,3,3,0,1}: 2 → 2 → 2 → 3 → 3 → 0 → 1

 Traffic link flow # 3 = {3,3,2,0,1}: 3 → 3 → 2 → 0 → 1

If the frequency is zero, the minimum value is given to cover all cases of state transition. The minimum frequency, Fr _min , should be as small as 0.001.

As a further example, the log-likelihood between any traffic flow {2,2,3,0,1} and the probability matrix of FIG. 17 is calculated as follows. As the log-likelihood value increases, the similarity also increases proportionally.

All. Model matching methodology

Prior to suggesting the intrusion detection method by model matching, the ratio of the normal likelihood is calculated for the normalization of the likelihood, and the log-likelihood ratio at this time is defined as in Equation 15.

, M ∈ {A ₁, A ₂, ..., A _k, A, N}

, M ∈ { A ₁ , A ₂ , ..., A _k , A , N }

Finally, the detection system proposed in this embodiment has a structure as shown in FIG. 1 as described above. If the maximum log-likelihood ratio MLLR is larger than the threshold, it means that it is similar to the attack flow.

MLLR = max ( LLR ), MLLR ≥ 0

18 shows a specific matching process by LLR comparison. Here, if the MLLR have a value of 0 indicates that the type of traffic and matching normal model, and thus MLLR can not have a negative value. If we have the maximum LLR when matched with a mixed model of all attacks, then this means that the input traffic is not a specific attack but is also normal traffic. Eventually, it is classified as an abnormal traffic that may be an unknown attack.

In the present embodiment, detection is performed based on the state transition information of each unit traffic flow group forming one real-time total traffic. Detection based on individual sessions or traffic flows is inadequate for effective detection of traffic that is particularly difficult to detect malicious behavior until large quantities of similar types of transport packets, such as botnets, or actual attacks occur. Because.

Therefore, a traffic modeling method and a matching technique based on the information in which the traffic flow groups are transferred are separated from each other in order to escape from the viewpoint of monitoring only a single traffic, and further expand it to distinguish the stepwise properties of the entire traffic.

Experimental Example

1) Experimental sample composition

Sinit and Phatbot, which are not currently active, were excluded from the experiment. Nugache, shown in Table 9, also showed a sluggish state, which was not active while generating a large amount of traffic, but experimented based on the communication traffic between the two peers in the established internal experimental environment.

Attack type Sample Binary MD5 Spamthru d844d871225484e3fd90c1ffe1e5706a 6183f1a6d780c70c95d20798090b0828 Nugache 0c859cfad2fa154f007042a1dca8d75b 1720155bf90614866392b6b655d15cbe 74600e5bc19538a3b6a0b4086f4e0053 9007e2a98f5e0399a798e35338e63d98 Peacomm dc49dfc97d9698ebede280e8daa7fd7b 562d6dad245497e6c95d1bb33e4bedda 9441cab2287dd6a5713e9e285c862e5f

2) Experiment data composition

Two-thirds of all source data were used for learning and the other 1/3 for experiments (detection). Here, the number of unit traffic as shown in Table 10 refers to a series of continuous traffic generation data from the initial execution phase to the end of the process by emulating one attack sample.

Attack type Unit traffic Data configuration learning Experiment Spamthru 157 105 52 Nugache 120 80 40 Peacomm 151 101 50

I. Traffic Characteristics and Detection Results Analysis

1) Analysis of traffic to be detected

SpamThru attempts to connect with externally active peers in the early stages. Prior to this, SpamThru downloads an antivirus program from any website to limit the activity of other malware. 19 and 20 show the traffic change amount at this time. At the beginning of execution, the traffic receiving file transfers increased significantly.

Subsequently, there may be peers that do not respond to the peer discovery process. In this case, as shown in FIGS. 21 and 22, the number of outgoing packets increases more than the number of incoming packets. Since only TCP communication is performed, there is no big difference in the total number of packets, but the incoming traffic is much higher in data size. This means that the packets mainly generated by the outgoing traffic were ACK response packets without data. In addition, the amount of traffic that is gradually generated is rather large, because the step of transmitting and receiving another peer list from the peers that are connected. The server then receives a list of template servers at the same time as the peer list exchange, and when these actions are completed, SpamThru simply checks its SMTP function before preparing to launch a full-scale spam attack. From this stage, traffic is somewhat reduced and ready for attack.

Looking at the amount of traffic change by protocol, the SpamThru botnet communicates using its own P2P protocol. However, as shown in FIG. 23, the initial vaccine program download process communicates with the web server using the HTTP protocol, and in the preparation process for sending spam mail, the SMTP protocol and the DNS protocol for querying the mail address domain as shown in FIG. Show the pattern you use.

25 and 26 show the distribution of ports used in the communication process. File is received from port 80 of web server and shows random usage in peer discovery and communication process. The spam mailing process uses port 25 for SMTP service and port 53 for DNS query.

In Nugache's sample emulation, outgoing traffic will continue to attempt to connect, but as incoming traffic, response packets are rarely received. This is because the line server is closed by passing an updated list of infected peers, making the activity difficult. As a result, it has recently become a botnet that is no longer effective. It is presumed that the main cause was the disadvantage of being hardcoded into the binary and attempting to access the IP lists of predefined servers. In this experiment, we built an internal experimental environment and extracted and analyzed the traffic transmitted and received between two peers.

27 and 28 show the traffic variation of the Nugache botnet. In general, it can be seen that the amount of outgoing traffic is significantly higher than inflow traffic, which is analyzed by the result of transmitting its peer information to a single connected peer with few responding peers. The Nugache botnet is a self-encrypted peer-to-peer communication based only on the TCP protocol.

As shown in the distribution of the ports used, communication is mainly performed with TCP port 8, and as shown in FIG. 29, the port numbers used for peer discovery are randomly increased.

Peacomm generates a lot of eDonkey / Overnet traffic and retrieves its query code. However, the query does not actually correspond to a file. 30-32 is unusual in inflow and outflow traffic in terms of the number of packets and data size. This is not because all of the files are downloaded. Peacomm does not download files from eDonkey by default. It just needs information from peers to find a site that downloads files. In this case, a list of peers is sent in response to the search query. Most of the Peacomm traffic consists of such file search and forwarding of neighbor peer list and response traffic of search results.

Most of the Peacomm traffic comes from the UDP protocol. This is because the eDonkey / Overnet access protocol, which is the protocol for peer discovery and file retrieval, is based on UDP. Sometimes the HTTP protocol is generated to download a file from the web server's path. In this case, file transfer traffic is very intermittent.

FIG. 33 shows the distribution of each Peacomm botnet traffic by port. As shown in the sample analysis result, the UDP port 4000, 7871, and 11271 are mainly selected to communicate with eDonkey peers. In this emulation result, communication is made using port 11271.

2) Detection result analysis

34 shows the detection performance according to the likelihood ratio threshold of the multi-state transition probability model for the analyzed traffic. In this experiment, the detection model for each attack was generated and the matched detection result was shown. The results at the threshold 0.2 of MLLR are shown in Table 11. In fact, SpamThru was mostly detected in traffic during the 'SMTP performance check' phase, which is almost the last step in preparation for the attack.

Botnet attack Detection Not detected False positives Detection rate Classification rate Not detected False positive rate Classification Unclassified Spamthru 52 50 47 3 2 - 96.15% 90.39% 3.85% - Nugache 40 38 37 One 2 - 95.00% 92.50% 5.00% - Peacomm 46 46 46 0 0 - 100.00% 100.00% 0.00% - Normal traffic 764 - - - - 22 - - - 2.88%

35 shows detection results based on the unified model of all attack patterns, and the results at the threshold value 0.2 of the MLLR are shown in Table 12. It can be seen that the detection by individual attack-specific models yielded better performance than that based on the unified model.

Botnet attack Detection Not detected False positives Detection rate Not detected False positive rate Spamthru 52 48 4 - 92.31% 7.69% - Nugache 40 37 3 - 92.50% 7.50% - Peacomm 46 45 One - 97.83% 2.17% - Normal traffic 764 - - 39 - - 5.11%

In this experiment, although a large amount of attack data was not constructed because the total traffic was one detection unit, this study suggests the possibility of future research in that the success rate of classification was also measured relatively high.

All. Probability distribution distance based detection result analysis

Not all transitions of a learned attack are involved in improving detection performance. In some cases, transition processes similar to normal patterns may have a negative effect on detection performance. In this experiment, we apply the effective detection scale selection method based on probability distribution distance (PDD) to analyze the effect on detection performance when only selected transition process is used for matching.

As a method of selecting a valid measure for detection, a method using experimental analysis results in the learning process has been mainly used. This is based on the experience of selecting the final scale by repeating the experiment by changing the scale several times and extracting the rule and checking the detection rate using the rule to check the detection rate. Involves the method. However, the detection rate in the verification process through learning does not represent the detection rate in the actual experiment, and the standard is ambiguous as an experimental method considering only the detection result ignoring the different meanings of the scales in the normal and the attack.

In this experiment, each measure is weighted according to its importance to identify the measures that most effectively represent the characteristics of the attack, and to select the optimal detection measures. The distance between the distribution of the scale values in the attack data and the distribution of the scale values in the normal data is measured. The larger the distance is, the more the attack discrimination is recognized. That is, the PDD between the attack and the normal data for each scale was obtained and applied as the weight of the scale. In this experiment, we used the Coolback-Leibler distance, which is defined as relative entropy, to calculate the PDD of each scale between normal behavior and attack behavior distribution. Coolback-Libler entropy is based on information theory as a distance calculation method applied in various applications. This entropy value defines the distance between two distributions as shown in Equation 17.

X is the vector set of detection scales, p is the probability distribution for normal behavior, and q _i is the probability distribution for attacking behavior. Each scale is f ∈ { scale ₁ , scale ₂ , ..., scale _m }, and each attack is i ∈ { attack ₁ , attack ₂ , ..., attack _n }. However, the above equations hold on the assumption that the measures are independent of each other.

Since PDDs are different from each other based on normal behavior and PDDs are different based on attack behavior, PDD is the maximum distance distance between two distributions for symmetric entropy. Set by value.

In the above equation, D is the standard reference distance for determining the distance (similarity) of two individuals, and the maximum distance between the attack and normal behavior distribution was used as the PDD to be obtained in this paper.

Table 13 shows a partial example of detection scale selection using PDD. First, a PDD value for each scale is obtained and stored, and only a scale whose PDD is greater than or equal to an arbitrary threshold distance is selected as a detection scale. Here, the scale means the transition process in the present experimental example.

According to the change in the threshold PDD, the number change amount of the selected scale may be confirmed in FIG. 36. It can be seen that the distance and the number of scales are inversely related, and when the distance is over a certain distance, the decrease curve is abruptly gentle, indicating that many non-significant scales are simultaneously removed. The critical distance should be adjusted to take into account the correlation between detection performance and detection rate.

The detection result of the SpamThru botnet to which the PDD is applied is shown in FIG. 37. Only the results of applying the PDD to the integrated detection model were tested. The best performance is shown when the threshold PDD value between SpamThru attack traffic and normal traffic transition is 67. In addition, when the threshold PDD value of 68 is used, the performance is shown to be deteriorated, which means that some valid measures have been canceled out. In addition, the results show a marked improvement compared to the performance in Figure 5.17.

In this experiment, we showed the effectiveness of detection scale selection using PDD. It is expected to further increase the discrimination ability between attacks and normal traffic. In addition, it facilitates the understanding of attack traffic characteristics and facilitates the cause analysis of detection results.

1 is a block diagram of a system for performing a botnet detection process according to the present invention.

2 is a graph of packet growth of communication traffic with individual peers of the SpamThru bot.

3 is an exemplary diagram illustrating packet retransmission of a TCP protocol.

4 is an exemplary diagram illustrating packet retransmission of the UDP protocol.

5 is an illustration of forward alignment of the gap reduction matrix.

6 is an illustration of reverse alignment of a gap penalty matrix.

7 is an illustration of a mixed alignment of a gap deduction matrix.

8 is a transition direction graph of two sequences by a mixed alignment of a gap reduction matrix.

9 is a diagram illustrating the result of a gap deduction matrix;

FIG. 10 is a band diagram illustrating a mixed sorted union. FIG.

11 is an exemplary view in which an adjacent matrix is generated.

12 is a view illustrating a state in which a connection matrix is generated.

13 is an exemplary view in which a connection is reformed through a connection relationship between clusters.

14 is an exemplary diagram of a process of calculating a state value of a representative traffic flow.

15 is a diagrammatic illustration of a frequency matrix of state transition information.

16 illustrates an Ergodic model.

17 illustrates an exemplary multi-state transition probability matrix of three traffic link flows.

18 is an exemplary view in which a model matching process is performed.

19 is a graph of transmit / receive packet count of SpanThru botnet traffic # 1.

20 is a graph of transmit / receive bytes of SpanThru botnet traffic # 1.

21 is a graph of transmit / receive bytes of SpanThru botnet traffic # 2.

22 is a graph of transmit / receive bytes of SpanThru botnet traffic # 2.

Fig. 23 is a graph of the number of packets for each protocol of SpanThru botnet traffic # 1.

24 is a graph of packet count for each protocol of SpanThru botnet traffic # 2.

25 is a distribution graph of ports of SpanThru botnet traffic # 1;

Fig. 26 is a graph showing distribution of ports of SpanThru botnet traffic # 2.

27 is a graph of transmit / receive packets of Nugache botnet traffic.

28 is a graph of transmit and receive bytes of Nugache botnet traffic.

29 is a distribution graph of transmission and reception ports of Nugache botnet traffic.

30 is a graph of the number of transmission and reception of Peacomm botnet traffic.

31 is a graph of transmit / receive bytes of Peacomm botnet traffic.

32 is a graph of the number of packets for each transmission / reception protocol of Peacomm botnet traffic.

Fig. 33 is a distribution graph of ports of Peacomm botnet traffic.

34 is a graph of ROC curves according to individual attack detection models.

35 is a graph of ROC curves according to an integrated detection model.

36 is a graph of the number of measures selected according to the change of probability distribution distance.

37 is a PDD applied ROC curve graph of the SpamRhru botnet according to the integrated detection model.

Claims (12)

Collecting and clustering traffic connection flows on a network to generate a traffic flow group, and then aggregating information of the state of the traffic flow group transition, and forming a multi-frequency matrix based on the aggregated information to perform a learning process; ; Obtaining a multi-state transition probability model by generating a probability matrix for a botnet attack and a normal probability matrix through the learning process; Performing model matching on the state transition information collected from the probability model and the traffic of the network in real time to detect whether a malicious botnet exists by comparing a result value of model matching with a preset threshold value; Pepipi-based botnet detection method using a network traffic transition model comprising a. The method of claim 1, The multi-state transition probability model distinguishes between a probability model group according to traffic state transitions for botnets already found and a probability model group according to traffic state transitions for botnets not found, When the existence of the botnet is detected as a result of performing the model matching, if the detected botnet belongs to the model group of traffic state transition to the already discovered botnet, it is regarded as a known attack and is regarded as a misuse detection. A traffic-penetration-based botnet detection method using a network traffic transition model, characterized in that the botnet is regarded as an abnormal detection by determining that the traffic state transition for the botnet which is not found belongs to a model group. The method of claim 1, wherein the multi-state transition probability model Pepipi-based botnet detection method using a network traffic transition model characterized in that generated by the Markov chain process. The method of claim 1, wherein the model matching is A peer-to-peer based botnet detection method using a network traffic transition model, which is performed by a log-likelihood ratio test for calculating a ratio of normal likelihood for generalization of likelihood. The method of claim 1, In the performing of the learning process: using the network traffic transition model, the collected traffic connection flows are divided into individual unit traffic connection flows, and similar unit traffic connection flows are grouped by sequential patterns according to behavior characteristics. P2P based botnet detection method. The method of claim 5, wherein In the step of performing the learning process: Peptide-based botnet detection method using a network traffic transition model characterized in that the gap deduction matrix is applied to compare the similarity between the unit traffic connection flow. The method of claim 6, A function for determining similarity between the unit traffic flows,

Pepipi-based botnet detection method using a network traffic transition model, characterized in that. The method of claim 1, A peer-to-peer based botnet detection method using a network traffic transition model, characterized in that the ROCK algorithm, which is a data mining clustering algorithm, is applied to cluster the traffic connection flow. The method of claim 8, In the preprocessing step of the clustering, a neighboring matrix is generated by a result value obtained by comparing similarities, and a linking matrix is generated based on the neighboring matrix, and the number of links of the linking matrix link = neighboring matrix (horizontal axis) x Pepipi-based botnet detection method using a network traffic transition model, characterized in that obtained by the adjacent matrix (vertical axis). The method of claim 9, The ROCK algorithm is applied to the fitness for clustering traffic link flow, the fitness is applied by generalizing the number of connections, which

Pepipi-based botnet detection method using a network traffic transition model, characterized in that calculated by. The method of claim 3, wherein The model generated by the Markov chain process presents a state transition model based on an Ergodic model, and based on this Ergodic model

Ptpi-based botnet detection method using a network traffic transition model characterized in that it calculates the log-likelihood, and the model thus constructed includes multi-state transition information represented as a probability matrix. The method of claim 4, wherein The log-likelihood ratio is

, M ∈ { A ₁ , A ₂ , ..., A _k , A , N }, and when the maximum log-likelihood MLLR is greater than the threshold, it is determined to be similar to the attack flow. Peopi based botnet detection method using network traffic transition model.

Images