JP4209897B2 - Mass flow generation host identification method and system - Google Patents

Description

  The present invention relates to a method and system for identifying mass flow generation hosts.

  In recent years, it has become a problem that abnormal traffic caused by worms, viruses, DDoS, and the like causes serious damage to networks and servers. A host that has been used as a platform for DDoS, or a host that is infected with a worm or virus, has a feature that a large number of flows are generated in a short time although the individual flow size is small. For this reason, it is important to quickly identify a host that generates a large number of flows in a short time, and take measures such as calling attention to the identified host or disconnecting it from the network.

  As a technique for collecting various statistical values of a flow such as a flow size and a flow time, packet sampling in which packets are extracted from a packet stream at regular intervals or at random is attracting attention. As the line capacity of the link increases to 2.5 Gbps and 10 Gbps, it becomes difficult to manage all the flow states. Therefore, only some flow states are managed by packet sampling, and various flow statistics are obtained. Is very useful in practice. Therefore, a method for identifying a high-rate flow in which the number of packets transmitted within a measurement period is equal to or greater than a threshold from information obtained by packet sampling is being studied. However, since the size of each flow generated by the mass flow generation host is small, such a host cannot be effectively identified by the high-rate flow identification method.

In order to specify a large number of flow generation hosts, it is effective to perform sampling for each flow, not for each packet, and total the number of generated flows for each host. From such a viewpoint, a method is considered in which a host in which one or more flows are sampled is regarded as a mass flow generation host, and the number of flows generated by the host within a measurement period is measured. However, it is not possible to explicitly give a specific threshold value for the flow rate ratio q (the flow number m generated by a host within the measurement period divided by the total flow number M), and a host having an arbitrary flow rate ratio q A specific curve that is the probability that is specified is not obtained.
Refer to the following Non-Patent Documents 1 to 3 for the above technique.

Mori, Kawahara, Kamiyama, Ishibashi, Abe, “Worm infected host detection method based on communication pattern analysis”, IEICE Tech. CQ, 2005-11. Kamiyama, Mori, “Study on Improvement of Specific Accuracy of High Rate Flow Identification Method”, IEICE Technical Report IN, 2005-7. K. Keys, D. Moore, C. Estan, “A Robust System for Accurate Real-time Summaries of Internet Traffic,” ACM AIGMETRICS 2005.

In the conventional mass flow generation host specifying method, the specific probability with respect to the flow number ratio is not formulated, and it is not clear how to specify the threshold value for the specified flow number ratio. When specifying a mass flow generation host, it is practically important to be able to design the threshold q ^* of the flow number ratio defined as a mass flow and its specific probability H ^* . Furthermore, the probability that a flow with the flow number ratio q q <q ^* is erroneously specified (FPR: False Positive Ratio) and the probability that a flow with q ≧ q ^* is not erroneously specified (FNR: False Negative Ratio) are clarified. It is important to clarify the specific accuracy. For this purpose, it is necessary to clarify the specific probability H for an arbitrary q when q ^* and H ^* are given (a curve representing the relationship between q and H is called a specific curve).

An object of the present invention is to construct a technique capable of specifying a mass flow generation host. Also, large amounts probability H ^* threshold q ^* host threshold q ^* and flow ratio of the number of number of flows ratio obtained by dividing the number of flows that host generated within the measuring period by the total number of flows arriving to the target link is identified When a specific threshold Y that is a threshold of the number of flows for specifying as a flow generation host is given, a flow sampling probability f that is a probability of sampling a flow is derived, or when q ^* , H ^*, and f are given It is to construct a mass flow generation host identification technique using flow sampling information that can derive Y in

  Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.

The mass flow generation host identification method of the present invention is a mass flow generation host identification method in a mass flow generation host identification system comprising a flow sampling device, a new flow determination device, a host table, and a mass flow generation host identification device, The flow sampling device obtains a β-bit value v obtained through a hash function prepared in advance for each packet arriving at the link of interest, using information for identifying a flow in the packet as a key. by the value of v is the sample the packet at the threshold w is less than for flow sampling is the product of the flow sampling probability f and 2 ^beta, samples the flow in the flow sampling probability f, the new flow determining device, Sampled by the flow sampling device within the measurement period. Is determined to be a new flow, and if it is determined to be a new flow, the number of flows of the host in the host table is updated, and the mass flow generation host specifying device A host whose number of flows is equal to or greater than a predetermined threshold Y is specified as a mass flow generation host . Further, the judgment sample packets of whether it belongs to the non-count of the flow, characterized by using a Bloom filter.

The mass flow generation host specifying system of the present invention uses, for each packet arriving at the link of interest, a value v of β bits obtained through a hash function prepared in advance using information for identifying a flow in the packet as a key. obtained, by the obtained value of v samples the said packets when a threshold w less than for flow sampling is the product of the flow sampling probability f and 2 ^beta, flow sampling to sample flow in the flow sampling probability f A device, a new flow determination device that determines whether or not the flow sampled by the flow sampling device within the measurement period is a new flow, and the new flow determination device that determines that the flow is a new flow Host table in which the number of flows of the corresponding host is updated, and the number of flows in the host table Characterized in that it comprises a Flow Hogs specific equipment for identifying the host from which the predetermined threshold value Y or higher as Flow Hogs. Also comprises a parameter designing apparatus for designing a flow sampling probability f and the threshold Y, and the flow sampling probability f and threshold Y for the parameter designing apparatus were designed with the flow sampling device respectively the Flow Hogs particular device It is characterized by being set.

  According to the present invention, it is possible to specify a mass flow generation host in which the number of flows generated within a measurement period exceeds a threshold using data obtained from flow measurement.

  FIG. 1 is a system configuration diagram showing an example of an embodiment of a mass flow generation host identification system that identifies a mass flow generation host by the mass flow generation host identification method of the present invention. In FIG. 1, 101 is a parameter design device, 102 is a flow sampling device, 103 is a new flow determination device, 104 is a host table, 105 is a mass flow generation host specifying device, and 106 is a mass flow generation host list.

  A flow sampling probability f (probability of sampling a flow) and a specific threshold Y (threshold for the number of flows for specifying as a mass flow generation host) are designed by the parameter design device 101, and each flow sampling device 102 and the mass flow generation host specifying device 105. During the measurement period, the flow that flows through the link of interest by the flow sampling device 102 is sampled with a certain probability (flow sampling probability f), and whether or not the sampled flow is a new flow (uncounted) is determined as a new flow determination device. If the result is 103 and the flow is a new flow, the number of flows of the corresponding host in the host table 104 is updated. The mass flow generation host specifying device 105 specifies a mass flow generation host from the number of flows of the hosts registered in the host table 104 and outputs it to the mass flow generation host list 106.

  That is, the mass flow generation host specifying system according to the present embodiment has a new flow determination apparatus 103 that determines whether or not a flow sampled within the measurement period is a new flow, and a new flow determination apparatus 103 that is new. A host table 104 in which the number of flows of the corresponding host is updated when it is determined to be a flow, and a mass that identifies a host in which the number of flows in the host table 104 is equal to or greater than a predetermined threshold Y as a mass flow generation host A flow generation host specifying device 105. Further, the flow sampling apparatus 102 that samples the flow with the flow sampling probability f, and the parameter design apparatus 101 that designs the flow sampling probability f and the threshold value Y, the flow sampling probability f and the threshold value Y designed by the parameter design apparatus 101 are provided. Are set in the flow sampling device 102 and the mass flow generation host specifying device 105, respectively. Then, the mass flow generation host specifying device 105 outputs the specified list of mass flow generation hosts to the mass flow generation host 106.

  Next, with regard to the mass flow generation host specifying method according to the embodiment of the present invention, 1. Principle of operation 2. Parameter setting method 3. Derivation of required memory size Derivation of the required maximum memory access number will be described respectively.

1. Principle of Operation A flow is defined as a set of packets having a common destination IP address and destination port number. Then, an arbitrary measurement period Φ having a length of φ (s) is defined, and the total number of flows arriving at the link of interest within Φ is defined as M. For each host, when the number of flows generated in Φ is m, the flow number ratio q is defined as q≡m / M. For an arbitrarily defined threshold value q ^* , a host with q ≧ q ^* is defined as a mass flow generation host, and the mass flow generation host is specified in Φ in the link of interest.

For each packet arriving at the link of interest, v <w, where v is the β-bit value obtained through a hash function prepared in advance using a total of 96 bits of the incoming and outgoing IP addresses and outgoing and outgoing port numbers as keys. Only when input to the new flow determination device. However, w is a threshold value for flow sampling. If f is a flow sampling probability, f = w / ^2β . Since flows belonging to the same flow have the same key, all flows are sampled with a probability f regardless of the flow size.

When v <w, it is determined whether the flow to which the next arrival packet belongs has already been counted. The most intuitive method is to manage all sampled flows in the flow table and determine whether there is an entry for the flow in the flow table, but it is necessary to check all the entries at the maximum. As a result, the processing time increases. Therefore, here, a determination method using a Bloom filter is used. Bloom filter is a method for determining whether a candidate is a member of a certain set, k different hash functions (each returning a b-bit value for the input key) and 2 ^b Binary variables (initially all reset to zero) are prepared. In this case, the value of a binary variable corresponding to k addresses obtained through k hash functions using the 96-bit key of the packet is examined, and if one or more is zero, the flow to which the packet belongs is a new flow. Is determined. Then, all k corresponding binary variables are set to 1. When the Bloom filter is used, there is no possibility that a flow that has already been counted is erroneously regarded as a new flow, but there is a possibility that a new flow may be mistakenly missed due to a hash value collision.

  The host table 104 stores the source IP address and the number of sample flows. If it is determined that the flow is a new flow, it is checked whether the source IP address of the packet has already been entered in the host table (for example, the hash value address is checked using the source IP address as a key), and an entry already exists. When doing so, the number of sample flows is increased by one. As a result, when the number of sample flows reaches the threshold Y, the host is specified as a mass flow generation host, and is written in the mass flow generation host list. On the other hand, if the source IP address is not entered in the host table, a new entry is generated.

  FIG. 2 shows a flowchart of the processing of the present invention. In FIG. 2, when a packet arrives (step 201), a hash value v is calculated (step 202). It is determined whether or not the calculated hash value v is smaller than a threshold w for flow sampling (step 203). If N in step 203, the process ends (not sampled). In the case of Y in Step 203, a new flow is determined by the Bloom filter (Step 204), and when it is not a new flow (in the case of N in Step 205), the process ends. If the flow is a new flow (Y in Step 205), the sample flow number c of the corresponding host in the host table 104 is incremented by 1 (Step 206). If c is not equal to the threshold Y (N in Step 207) If finished). When c is equal to the threshold value Y, the originating host is specified as a mass flow generation host (step 208). Here, steps 201 to 203 are processed by the flow sampling apparatus 102, steps 204 to 206 are processed by the new flow determination apparatus 103, and steps 207 to 208 are processed by the mass flow generation host specifying apparatus. As a timing at which the mass flow generation host specifying device 105 writes out to the mass flow generation host list 106, a method of collectively writing out at the end of the measurement period can be considered, but other timings may be used.

  That is, the mass flow generation host specifying method of the present embodiment uses only statistical data obtained by flow sampling to specify a host whose number of flows generated within a measurement period is equal to or greater than a threshold as a mass flow generation host. Is. In addition, a Bloom filter is used to determine whether or not a sampled packet belongs to an uncounted flow. Also, the number of flows sampled for each host is aggregated, and a host whose aggregate value is equal to or greater than a predetermined threshold Y is specified as a mass flow generation host.

2. Parameter setting method The probability that j flows from a host having a flow rate ratio of q are sampled within the measurement period Φ is given by _F C _j (1-q) ^F−j q ^j (F is the total number of samples in Φ) Number of flows, F = fM). Therefore, the probability H that this host is specified is

となる。フロー数比率の閾値ｑ^＊のホストを確率Ｈ^＊で特定するためには、

It becomes. In order to identify the host with the flow rate ratio threshold q ^* with probability H ^* ,

となるようにＹとＦを設定すればよい。すなわち、ＹとＦのどちらか一方を与えれば、式（２）を数値的に解くことによりもう一方を決めることができる。Ｍの値として例えば前測定期間における実測値を用いれば、Ｙとｆのどちらか一方が自動的に決まると考えることもできる。

Y and F may be set so that That is, if one of Y and F is given, the other can be determined by solving Equation (2) numerically. For example, if an actual measurement value in the previous measurement period is used as the value of M, it can be considered that either Y or f is automatically determined.

  It is expected that the greater the Y, the greater the required accuracy, while the specific accuracy improves. Therefore, by adjusting Y, a trade-off is obtained between the specific accuracy and the required memory amount. For example, it is possible to select the maximum Y within the range of the allowable system cost (memory amount) and design the flow sampling probability f. Similarly, as f increases, the specific accuracy improves, but the required memory amount is expected to increase. Therefore, the maximum f is selected within the allowable memory amount range, and the threshold value Y is designed. Is possible.

  The parameter design apparatus 101 performs the design of the above parameters, and the flow sampling probability f and the threshold Y designed by the parameter design apparatus 101 are set in the flow sampling apparatus 102 and the mass flow generation host specifying apparatus 105, respectively.

3. Derivation of required memory amount It is assumed that H ₃ that is known for high performance is used for the hash function. The amount of memory required for the flow sampling hash function is 96β [bits]. The new flow determination process requires 2 ^b [bits] for accommodating binary variables and 96 bk [bits] for the hash function. Finally, for the host table, if the maximum number of hosts entered is E _max ,

が必要になる。ここで、ｌｏｇ_２Ｙを挟む記号は、挟んだ数を下まわらない最小の整数を意味する記号であり、例えばｌｏｇ_２Ｙが３．６である場合は４になる。これらを合わせ、所要メモリ量［bits］は次式で算出できる。

Is required. Here, the symbol that sandwiches log ₂ Y is a symbol that means the smallest integer that does not fall below the sandwiched number. For example, when log ₂ Y is 3.6, the symbol is 4. By combining these, the required memory amount [bits] can be calculated by the following equation.

４．所要最大メモリアクセス数の導出
１個の到着パケットに対して必要になる最大メモリアクセス数を考える。Ｈ_３を用いればハードウェアで容易にハッシュ関数を構築できるためハッシュ値を求める処理は考慮しない。新規フローの判定処理にｋ回のメモリアクセスが必要である。ホストテーブルの更新には、ホストのＩＰアドレスをキーにハッシュ関数を用いてエントリを収容することを想定し、ハッシュ値の衝突による収容アドレスの調整が生じる回数の最大値をＣ_ｍａｘとするとＣ_ｍａｘ＋１となる。よって所要最大メモリアクセス数は次式で求まる。

4). Derivation of required maximum memory access number Consider the maximum memory access number required for one incoming packet. Process of obtaining the hash value for the use of H ₃ can be constructed easily hash function hardware is not considered. A new flow determination process requires k memory accesses. To update the host table, assuming that accommodates the entry by using a hash function to the key IP address of the host, when the maximum number of times the adjustment of the accommodating addresses by the collision of the hash value is generated as C _max C _max +1. Therefore, the required maximum memory access number is obtained by the following equation.

以上説明した実施の形態において、ホストテーブルおよび大量フロー生成ホストリストは記憶装置に記憶される。また、各装置はコンピュータと記憶装置に記憶されたプログラムで構成できる。また、そのプログラムの一部または全部に代えてハードウェアで構成してもよい。

In the embodiment described above, the host table and the mass flow generation host list are stored in the storage device. Each device can be configured by a program stored in a computer and a storage device. Further, it may be configured by hardware instead of part or all of the program.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Of course.

System configuration diagram showing an example of an embodiment of the present invention The flowchart which shows the flow of the process of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Parameter design apparatus, 102 ... Flow sampling apparatus, 103 ... New flow determination apparatus, 104 ... Host table, 105 ... Mass flow generation host specification apparatus, 106 ... Mass flow generation host list

Claims (8)

A mass flow generation host identification method in a mass flow generation host identification system comprising a flow sampling apparatus, a new flow determination apparatus, a host table, and a mass flow generation host identification apparatus,
For each packet arriving at the link of interest, the flow sampling device obtains a β-bit value v obtained through a hash function prepared in advance using information for identifying a flow in the packet as a key. the value of v is by sampling the packets at the time of the threshold value w is smaller than for the flow sampling is the product of the flow sampling probability f and 2 ^beta were samples the flow in the flow sampling probability f,
When the new flow determination device determines whether or not the flow sampled by the flow sampling device within the measurement period is a new flow, and determines that the flow is a new flow, the host in the host table Update the number of flows
The mass flow generation host identification device identifies a host whose number of flows in the host table is equal to or greater than a predetermined threshold Y as a mass flow generation host;
A mass flow generation host identification method characterized by the above. In the mass flow generation host specifying method according to claim 1,
A method for specifying a mass flow generation host, wherein the new flow determination apparatus uses a Bloom filter to determine whether or not a sampled packet belongs to an uncounted flow. In the mass flow generation host specifying method according to claim 1 ,
The mass flow generation host specifying system includes a parameter design device,
The parameter design apparatus has a total number M of flows arriving at the link of interest within the measurement period, a threshold q ^* of a flow number ratio obtained by dividing the number of flows generated by the host within the measurement period by the total number of flows M, Given a probability H ^* that a host with a ratio q ^* is identified and a threshold Y, and F = fM,

A mass flow generation host specifying method characterized by deriving a flow sampling probability f which is a probability of sampling a flow. In the mass flow generation host specifying method according to claim 1 ,
The mass flow generation host specifying system includes a parameter design device,
The parameter design apparatus has a total number M of flows arriving at the link of interest within the measurement period, a threshold q ^* of a flow number ratio obtained by dividing the number of flows generated by the host within the measurement period by the total number of flows M, and the number of flows. Given a probability H ^* that a host with a ratio threshold q ^* is identified and a flow sampling probability f that is the probability of sampling the flow, where F = fM,

A threshold value Y is derived from the mass flow generation host identification method. In the mass flow generation host specifying method according to claim 1 ,
The mass flow generation host specifying system includes a parameter design device,
The parameter design device divides the total flow number M that arrives at the link of interest within the measurement period, the flow sampling probability f that is the probability of sampling the flow, and the total number of flows M that the host generated during the measurement period. Given the threshold q ^{* of} the flow number ratio and the probability H ^* that the host of the threshold q ^* of the flow number ratio is specified, the probability H of specifying the host having an arbitrary flow number ratio q is derived, A mass flow generation host identification method characterized by obtaining a specific curve which is a curve representing the relationship between q and H. In the mass flow generation host specifying method according to claim 1 ,
The mass flow generation host specifying system includes a parameter design device,
The method for specifying a mass flow generation host , wherein the parameter design apparatus obtains a required memory amount and a required maximum memory access number. For each packet arriving at the link of interest, using the information for identifying the flow in the packet as a key, a β-bit value v obtained through a hash function prepared in advance is obtained. by sampling the packets at the time of the threshold value w is smaller than for the flow sampling is the product of the sampling probability f and 2 ^beta, and flow sampling device for sampling the flow in flow sampling probability f,
A new flow determination device for determining whether or not a flow sampled by the flow sampling device within a measurement period is a new flow;
A host table in which the number of flows of the corresponding host is updated when the new flow determination device determines that the flow is a new flow;
A mass flow generation host identifying device that identifies a host whose number of flows in the host table is equal to or greater than a predetermined threshold Y as a mass flow generation host;
A mass flow generation host identification system characterized by comprising: The mass flow generation host identification system according to claim 7 ,
Includes a parameter design equipment for designing the flow sampling probability f and threshold Y,
A mass flow generation host specifying system, wherein a flow sampling probability f and a threshold value Y designed by the parameter design apparatus are set in the flow sampling device and the mass flow generation host specifying device, respectively.

Images