JP2007173907A - Abnormal traffic detection method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a detection method of an abnormal traffic for performing monitoring with a small amount of memory capacity, monitoring data of N upper field values that are important as the objects to be monitored and a counter value, and detecting abnormality in time series. <P>SOLUTION: The method includes a step for acquiring traffic data; a step for designating at least one field to be monitored and the kind of counter to the field; a step for calculating N (N≥2) upper field values to the designation counter of a designated field at a preset interval from the acquired traffic data, and for counting the counter value; a step for storing the N calculated upper field values and the data of the counter value to calculate the degree of similarity between newly calculated data and past ones; and a fifth step for issuing an alarm as a fault when the calculated degree of similarity is smaller than a preset threshold to extract the field value estimated to be the cause of a decrease in the degree of similarity. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an abnormal traffic detection method and apparatus, and more particularly to a method and apparatus for detecting an abnormality in computer network traffic.

With an increase in abnormal traffic such as attacks on specific servers on the Internet, there is an increasing demand for a method for detecting abnormal traffic.
Current anomalous traffic detection methods are roughly classified into those based on signatures and those based on statistical information.
In the former, a traffic pattern determined to be abnormal in advance is prepared as a signature, and it is determined whether or not the observed traffic is abnormal by comparing the observed traffic with the same signature.
While this method can reliably determine whether or not the observed traffic is abnormal, it is difficult to prepare all the abnormal traffic patterns. In particular, it can detect new abnormal traffic for which no signature has been obtained. There is a problem that it cannot be done.

The method based on statistical information creates statistical information from past observed traffic history, and judges whether newly observed traffic is abnormal or not based on statistical deviation from past traffic. It is. The observation target is generally based on the total traffic such as the number of packets.
However, in statistical anomaly detection based on total traffic volume observation, it is difficult to detect anomalies due to some changes in traffic data that do not appear as changes in total traffic, and it is difficult to identify the cause of anomalies after anomaly detection There's a problem.
On the other hand, instead of monitoring the total traffic volume, a specified counter (for example, the number of packets for each transmission IP address or the number of bytes for each transmission IP address) for each field value of the specified field (for example, the transmission IP address field) is held. In the method of detecting an abnormality from the statistical information of the counter value, there is no such problem, and it is possible to detect a change in the number of packets of some transmission IP addresses and to specify the transmission IP address that has caused the change.
However, this method requires the same number of counters as the number of field values to be observed, and it requires a huge amount of memory to apply this method to traffic that generates a large number of field values. Is difficult.

As prior art documents related to the invention of the present application, there are the following.
JP 2005-65294 A Robert Schweller, Ashish Gupta, Elliot Parsons, Yan Chen, “Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams,” Proceedings of the ACM SIGCOMM Internet Measurrment Conference; (IMC), Octber 2004. C. Estan and G. Varghese, “New directions in traffic measurement and accounting,” in Proc. ACM SIGCOMM, August 2002. G.Cormode and S.Muthukrishnan.WHat's hot and what's not: Tracking most frequent items dynamically.In Proceeding of ACM Principles of Database sSystems, June 2003. RFC 3577-Introduction to the Remote Monitoring (RMON) Family of MIB Modules Benoit Claise, “NetFlow features update,” Proceedings of Sampling 2005, July 2005

In order to solve the above-described problem, the above-mentioned Patent Document 1 has not a counter for each field value but a counter for each value called a sketch, which is a field value converted into a fixed area value using a hash function or the like. Have proposed a method of monitoring counters of a large number of field values with a fixed memory capacity.
However, this method has a problem that it is difficult to identify the field value that caused the abnormality after detecting the abnormality because the field value is converted by the hash function. In the above-mentioned Non-Patent Document 1, although a method for alleviating this difficulty has been proposed, iterative calculation is necessary for specifying the cause of the abnormality, and there remains a problem in terms of calculation amount.
On the other hand, methods for extracting only a field value having a large counter value, which is important for traffic monitoring, with a limited memory capacity have been proposed in the aforementioned Non-Patent Documents 2 and 3.
In addition, as described in Non-Patent Documents 4 and 5 described above, some network devices implement a method of outputting the top N field values in descending order of counter values and the counter values. There is also.

However, the method of extracting the top N field value and the counter value only provides a snapshot of the top N field value and the counter value data at a certain point in time, and detects an abnormality in the top N field value data on the time series. Not what you want.
The present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to monitor the top N field values that can be monitored with a small memory capacity and are important for monitoring. An object of the present invention is to provide an abnormal traffic detection method and apparatus for monitoring data of a counter value and detecting an abnormality in the time series.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.
In order to achieve the above object, the present invention provides an abnormal traffic detection method for detecting an anomaly in time series traffic data, the first step of acquiring traffic data, and at least one field to be monitored. , And a second step of designating the type of counter for the field, and the top N (N ≧ 2) field values for the designated counter of the designated field at a predetermined time interval from the acquired traffic data , And a third step of calculating the counter value, the calculated top N field values, and the counter value data are accumulated, and the similarity between the newly calculated data and the past data is calculated. Step 4 and if the calculated similarity is below a predetermined threshold, an alarm is given as an abnormality, and Degrees is characterized in that it comprises a fifth step of extracting a field value that is estimated to cause of the reduction.
In the present invention, the similarity calculated in the fourth step is based on a correlation coefficient between the newly calculated counter value of the upper N field value and the past counter value of the field value. .

In the present invention, the similarity calculated in the fourth step is based on a correlation coefficient between the counter value of the upper N field value in a certain past monitoring period and the newly calculated counter value of the field value. Is.
In the present invention, the similarity calculated in the fourth step is a phase difference between the newly calculated counter value of the upper N field value and the counter value of the upper N field value calculated in the past in the same order. Based on the number of relationships.
In the present invention, in the above-described methods, the similarity calculated in the fourth step is based on the sum of the distances between the counter values instead of the correlation coefficient.
In the present invention, the similarity calculated in the fourth step is based on a correlation coefficient between the newly calculated rank of the top N field value and the past rank of the field value.
In the present invention, the similarity calculated in the fourth step is a correlation coefficient between the rank of the top N field value in a certain past monitoring period and the rank of the field value in the newly calculated counter value. Is based.
In the present invention, the similarity calculated in the fourth step is calculated based on the time series prediction of the counter value for each field value calculated from the data calculated in the past, and the newly calculated upper N field value. This is based on an error from the counter value.

In the present invention, when the top N field values are arranged in descending order with respect to the distance between the counter values in the fifth step, the counter value of the field value is omitted from the similarity calculation from the top one. The field value that does not fall below the threshold value is estimated as the field value that caused the similarity to decrease.
In the present invention, in the fifth step, when the top N field values are arranged in descending order with respect to the rank distance, the threshold value is set when the rank of the field value is omitted from the similarity calculation from the top rank. The field value that does not fall below is estimated as the field value that caused the similarity to decrease.
In the present invention, in the fifth step, when the counter value time series prediction of the top N field value and the error distance of the counter value of the newly calculated top N field value are arranged in descending order, The field value that does not fall below the threshold when the counter value of the field value is omitted from the similarity calculation from the higher rank is estimated as the field value that caused the similarity to decrease.
The present invention is also an abnormal traffic detection apparatus that executes each of the abnormal traffic detection methods described above.

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.
According to the present invention, by monitoring only the top N field values and counter value data that can be monitored with a small amount of memory, an abnormality in those data is detected, and a field value that causes the abnormality is detected. It becomes possible to do.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In all the drawings for explaining the embodiments, parts having the same functions are given the same reference numerals, and repeated explanation thereof is omitted.
FIG. 1 is a block diagram showing a basic configuration of an abnormal traffic detection apparatus according to an embodiment of the present invention. The apparatus for detecting abnormal traffic according to the present embodiment is arranged in an IP network.
The traffic acquisition unit 10 acquires traffic information transferred on a link in the network and within a node in the network, and transfers the acquired traffic information to the upper N field value calculation unit 12.
The field / counter type designation unit 11 designates the field / counter type to the upper N field value calculation unit 12. Here, as a field specification example, a set of header information such as a transmission IP address, a reception IP address, {transmission IP address, reception IP address, protocol, transmission port number, reception port number} (referred to as a normal flow), URI, which is payload information, and DNS domain name.
Examples of specifying the counter type include the number of packets, the number of bytes, the field value of the originating / receiving IP address, the number of flows, the field number of the URI, DNS domain name, etc. Can be mentioned.
The upper N field value calculation unit 12 receives the traffic information from the traffic acquisition unit 10 and sets a combination of the upper N field values and counter values related to the designated counter of the designated field designated by the field and counter type designation unit 11. calculate.

FIG. 2 shows an example of the output of the upper N field value calculation unit 12 when the originating IP address is designated as the designated field and the number of packets is designated as the counter value. Here, N = 5 and the output interval is 5 minutes.
The similarity calculation unit 13 compares the data of the output top N field values and the counter values with past data, and calculates how similar the latest data is to the past data.
There are various methods for calculating the similarity, but in this embodiment, a correlation coefficient with past data is used.
Hereinafter, an example of a method for calculating the similarity will be described.
The field value at the ranking i in the time slot (t) is f (i, t), and the counter value of the field value f in the time slot (t) is c (f, t).
Therefore, in the example of FIG. 2, f (1,1) = 192.168.0.1, f (1,2) = 192.168.0.2, and c (f (1,1), 1 ) = 5000, c (f (1,1), 2) = 6000.
At this time, in the second method of the present embodiment (the method according to claim 2 of the present application), the counter value of the upper N (N is a natural number, N ≧ 2) field values of the time slot (t). The correlation coefficient r (t, t-1) between the data and the data in the time slot (t-1) which is the latest data is calculated using the following equation (1).

ここで、ｃ_ａｖｇ（ｔ）、ｃ_ａｖｇ（ｔ−１）は、それぞれ時刻スロット（ｔ，ｔ−１）における、ｆ（ｉ，ｔ）のカウンタ値の平均であり、それぞれ、下記（２）式、（３）式で表される。

Here, c _avg (t) and c _avg (t−1) are averages of the counter values of f (i, t) in the time slot (t, t−1), respectively. It is represented by the formula (3).

As shown in the above equation (1), the correlation coefficient takes a value from “−1” to “1”, and in the time slot (t, t−1), the top N field values and their counters If the values are the same, the correlation coefficient r (t, t−1) is “1”. Therefore, the closer the correlation coefficient is to “1”, the more similar.
For example, as shown in the counter value in time slot 2 for the first field value “10.0.0.1” in time slot 3 in FIG. It is also conceivable that the value does not exist in the top N in the time slot (t−1) and there is no corresponding counter value data. In this case, for example, a method of calculating the correlation coefficient with such a counter value as “0” can be considered.
FIG. 3 shows the counter values of time slot 2 and time slot 3 in the example of FIG.
In normal traffic, the top N field values calculated at regular intervals and their counter value data are less changed by the law of large numbers than other field values and their counter value data, and are similar to each other. It is expected that
FIG. 4 shows a time series of correlation coefficients r (t, t−1) between adjacent time slots for data of the top 100 field values of traffic data for a certain 24 hours.
As expected at most times, the top 100 data are similar and the correlation coefficient is between (0.95 and 1), but the correlation coefficient suddenly increases at around 3 o'clock and 20 o'clock. There is a slight decline.
This is because the field values that existed in the lower order in other time periods suddenly appeared in the upper order during this time period.

On the other hand, in order to detect an abnormality when the top N field values appearing in the past data are not in the new data, it is necessary to calculate the correlation coefficient of the counter value based on the past top N field values. The third method of the present embodiment (the method described in claim 3 of the present application) is a method for calculating such a correlation coefficient.
Further, in the case of a field such as a flow that ends in a short time, the upper N field value varies greatly, but the corresponding counter value is expected to vary relatively little. In this case, the field value is fixed and the correlation coefficient between the corresponding past and new counter values is compared, and the order is fixed and the correlation coefficient between the corresponding past and new counter values is compared. There are cases where the method is effective.
In the fourth method of the present embodiment (the method described in claim 4 of the present application), the correlation coefficient of the counter value corresponding to each rank is used in such a case.
In the fifth method of the present embodiment (the method according to claim 5 of the present application), the difference between the counter values in the past and new data, not the correlation coefficient, in the above-described second, third, and fourth methods. The reciprocal of the sum of absolute values of is used. This method is useful when it is desired to perform anomaly detection that emphasizes the absolute value of the difference between counter values, not the similarity normalized from “−1” to “1”, such as a correlation coefficient.

In addition, when the upper N field value is obtained, but the counter value is not obtained, or obtained, but the fluctuation is large, a method of using a change in rank instead of the counter value may be considered.
The sixth method of the present embodiment (the method described in claim 6 of the present application) detects an abnormality based on the rank without using the counter value in such a case.
Here, the field value of i-th ranking in the time slot (t) is f (i, t), and the ranking of the top N (N is a natural number, N ≧ 2) field values in the time slot (t) is r. When (f, t) is set, the rank correlation coefficient r _c (t, t−1) is calculated by the following equation (4).
Furthermore, in order to detect an abnormality when the top N field values appearing in the past data are not in the new data, it is necessary to calculate a correlation coefficient of ranks based on the past top N field values. The seventh method of the present embodiment (the method according to claim 7 of the present application) is a method for calculating the correlation coefficient of such rank.

本実施例の第８の方法（本願の請求項８に記載の方法）においては、過去の上位フィールド値のカウンタ値の観測データに基づき、上位Ｎフィールド値毎の時系列予測を、例えば、ＡＲＩＭＡモデルなどの線形予測モデルなどで生成し、新規の上位Ｎフィールド値のカウンタ値と予測値との差の絶対値の和の逆数を類似度として計算する。
本方法は、線形予測モデルを生成するために計算量が多くなる反面、トレンド、周期変動等を異常検出の対象から除くことが出来ると期待される。

In the eighth method of the present embodiment (the method according to claim 8 of the present application), time series prediction for each upper N field value is performed based on the observation data of the past counter value of the upper field value, for example, ARIMA It is generated by a linear prediction model such as a model, and the reciprocal of the sum of absolute values of the difference between the counter value of the new top N field value and the predicted value is calculated as the similarity.
Although this method requires a large amount of calculation to generate a linear prediction model, it is expected that trends, periodic fluctuations, and the like can be excluded from an abnormality detection target.

The abnormality determination and abnormality cause identification unit 14 compares whether or not the similarity output by the similarity calculation unit 13 is below a predetermined threshold, and issues an alarm if the similarity is below.
As an example of how to set the threshold value, a method of generating a statistical distribution from past similarity data and using the 99% value as the threshold value is conceivable. Further, when the value is below the threshold value, the field value estimated as the cause is extracted.
As an example of the extraction method, in the ninth method of the present embodiment (the method described in claim 9 of the present application), when the absolute values of the differences between the past and the new counter values are arranged in descending order, the higher ones are used. When the counter value of the field value is omitted from the similarity calculation, a field value that does not fall below the threshold value is estimated as a field value that causes the similarity to decrease.
In the example of FIG. 3, the field value that is the farthest from the straight line with the slope 1 is used to calculate whether or not the field value is the cause of the similarity decrease. First, the field value “10.0.0.1” decreases the similarity It will be calculated whether it is the cause.
Further, in the tenth method of the present embodiment (the method according to claim 10 of the present application), when the top N field values are arranged in descending order with respect to the rank distance, the ranks of the field values from the top are similar. A field value that does not fall below the threshold when omitted from the degree calculation is estimated as a field value that causes the similarity to decrease.

Furthermore, in the eleventh method of the present embodiment (the method according to claim 11 of the present application), the time series prediction of the counter value for each upper N field value and the newly calculated counter value of the upper N field value are performed. When the error values are arranged in descending order, the field value that does not fall below the threshold when the counter value of the field value is omitted from the calculation of the similarity from the top is used as the field value that caused the similarity to decrease. presume.
In normal traffic, the top N field values calculated at regular intervals and their counter value data are less changed by the law of large numbers than other field values and their counter value data, and are similar to each other. It is expected that
Therefore, according to the present embodiment, by monitoring the degree of similarity with past data, if there is an abnormality in the top N field values and the counter value data, it is relatively easy to perform statistics with a decrease in degree of similarity. This is advantageous in that it can detect the degree of mechanical anomaly and easily detect the field value that causes it.
As mentioned above, the invention made by the present inventor has been specifically described based on the above embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Of course.

It is a block diagram which shows the basic composition of the abnormal traffic detection apparatus of the Example of this invention. It is a figure which shows an example of an upper N field value and its counter value. In the example shown in FIG. 2, it is a graph which shows the relationship between the time slot 2 time and the counter value of slot 3. It is a figure which shows an example of the correlation coefficient between adjacent time slots of an upper N field value and its counter value.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Traffic acquisition part 11 Field, counter kind designation | designated part 12 Upper N field value calculation part 13 Similarity calculation part 14 Abnormality determination, abnormal cause identification part

Claims (12)

An anomaly traffic detection method for detecting anomalies in time series traffic data,
A first step of obtaining traffic data;
A second step of specifying at least one field to be monitored and a counter type for the field;
A third step of calculating the top N (N ≧ 2) field values for the designated counter of the designated field, and the counter value from the acquired traffic data at a predetermined time interval;
A fourth step of accumulating the calculated top N field values and the counter value data, and calculating the similarity between the newly calculated data and past data;
And a fifth step of extracting a field value that is presumed to be the cause of the decrease in similarity, and that issues a warning as an abnormality if the calculated similarity is below a predetermined threshold. To detect abnormal traffic.   The similarity calculated in the fourth step is based on a correlation coefficient between a newly calculated counter value of the top N field value and a past counter value of the field value. Item 6. The abnormal traffic detection method according to Item 1.   The similarity calculated in the fourth step is based on a correlation coefficient between a counter value of the top N field value in a certain past monitoring period and a newly calculated counter value of the field value. The method for detecting abnormal traffic according to claim 1.   The similarity calculated in the fourth step is based on a correlation coefficient between a newly calculated counter value of the top N field value and a counter value of the same rank of the top N field value calculated in the past. The abnormal traffic detection method according to claim 1, wherein:   5. The similarity calculated in the fourth step is based on a sum of distances between counter values instead of the correlation coefficient. 6. The anomaly traffic detection method described in 1.   The similarity calculated in the fourth step is based on a correlation coefficient between the rank of the newly calculated top N field value and the past rank of the field value. The anomaly traffic detection method described in 1.   The similarity calculated in the fourth step is based on the correlation coefficient between the rank of the top N field value in a certain past monitoring period and the rank of the field value in the newly calculated counter value. The method for detecting abnormal traffic according to claim 1, wherein:   The similarity calculated in the fourth step is an error between the time series prediction of the counter value for each field value calculated from the data calculated in the past and the counter value of the newly calculated upper N field value. The abnormal traffic detection method according to claim 1, wherein the method is based on the above.   In the fifth step, when the upper N field values are arranged in descending order with respect to the distance between the counter values, the field that does not fall below the threshold when the counter value of the field value is omitted from the similarity calculation from the upper one. 6. The abnormal traffic detection method according to claim 2, wherein the value is estimated as a field value that causes the similarity to decrease.   In the fifth step, when the top N field values are arranged in descending order with respect to the rank distance, a field value that does not fall below the threshold when the rank of the field value is omitted from the similarity calculation from the top rank, The abnormal traffic detection method according to claim 6, wherein the field value of the cause of the decrease in the similarity is estimated.   In the fifth step, when the time series prediction of the counter value for each top N field value and the error distance of the counter value of the newly calculated top N field value are arranged in descending order, 9. The abnormal traffic detection according to claim 8, wherein a field value that does not fall below a threshold when the counter value of the field value is omitted from the similarity calculation is estimated as a field value that causes the similarity to decrease. Method. An anomaly traffic detection device for detecting anomalies in time series traffic data,
Means for obtaining traffic data;
Means for designating at least one field to be monitored and a counter type for the field;
Means for calculating upper (N ≧ 2) field values for a designated counter of a designated field and a counter value thereof at a predetermined time interval from the acquired traffic data;
Means for accumulating the calculated top N field values and their counter value data, and calculating the similarity between the newly calculated data and past data;
An abnormal traffic characterized by including a means for issuing an alarm as an abnormality if the calculated similarity is below a predetermined threshold value and extracting a field value presumed to be the cause of the decrease in the similarity Detection device.

Images