JP5310094B2 - Anomaly detection system, anomaly detection method and anomaly detection program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an abnormality detection system for efficiently detecting in what data system an abnormality or variation is generated even when there are a very large number of data systems. <P>SOLUTION: An integration means 71 integrates data systems determined to belong to the same group by calculating data values or the sum of power of the data values of the data systems determined to belong to the same group. A statistic calculation means 72 calculates statistic of the data values of the data systems before they are integrated. A group detection means 73 detects a group including a data system in which the abnormality or the variation is generated based on the sum calculated by each group. A data system specification means 74 specifies the data system in which the abnormality or the variation is detected based on the statistic from among the data systems which belong to the group detected by the group detection means 73. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an anomaly detection system, an anomaly detection method, and an anomaly detection program, and more particularly to an anomaly detection system, an anomaly detection method, and an anomaly detection for detecting which data series among a plurality of data series are abnormal or changing. Related to the program.

  Various abnormality detection systems have been proposed. For example, Patent Document 1 describes a network abnormality detection system that aggregates network traffic in units of flows using its characteristics and performs abnormality detection by reducing the data series to be inspected.

  Patent Document 2 summarizes a plurality of abnormality detection means for detecting an abnormality and a plurality of abnormality detection information derived from the same cause and detected from the plurality of abnormality detection means as a single abnormality information. A system that generates as one piece of detection information is described.

  Non-Patent Document 1 describes a method for detecting anomalies after aggregating a plurality of data series. In the method described in Non-Patent Document 1, after an abnormality is detected, a person specifies which data series before aggregation is the cause of the abnormality.

  Patent Document 3 discloses a plurality of collection devices that collect packet information and flow statistical information for each predetermined type, and information collected by the collection device from one or a plurality of collection devices. Describes a network monitoring system including an aggregation device that aggregates collected information according to the above.

  Patent Document 4 describes a process monitoring device that calculates statistical data and diagnoses process abnormality.

  Patent Document 5 calculates the top N (N ≧ 2) field values and the like of a specified field, accumulates the calculated top N field values and the like, and newly calculated data and past data An abnormal traffic detection method is described in which the degree of similarity is calculated and an abnormality is detected by comparing the degree of similarity with a threshold. Patent Document 5 describes that a statistical distribution is calculated from past similarity data, and the 99% value is used as a threshold value.

  Examples of abnormality detection algorithms include outlier detection (see Patent Literatures 6 to 9), change point detection (see Patent Literatures 10 and 11), abnormal behavior detection (see Patent Literature 12), and the like.

JP 2006-246325 A (paragraph 0008, FIG. 1) Japanese Patent Laid-Open No. 2001-314037 (paragraph 0007, FIG. 1) Japanese Unexamined Patent Publication No. 2007-13590 (paragraphs 0021-0023 and 0032-0043) JP 2007-65883 A (paragraphs 0020, 0032) JP 2008-167484 A (paragraphs 0007, 0019) JP 2001-101154 A JP 2003-5970 A Japanese Patent Application Laid-Open No. 2004-78981 JP 2007-18530 A JP 2004-54370 A JP 2004-4658 A JP 2004-309998 A

Masaya Yamagata, Yasuhiro Murata, Daisuke Inoue, Masafumi Eto, Katsunari Yoshioka, Koji Nakao, “Extension of Change Point Detection Engine Considering All Port Access Monitoring in Macro Analysis Environment”, 2008 Symposium on Cryptography and Information Security (SCIS2008), 3C1-4, 2008

  As described in Patent Document 1, there is a method for detecting anomalies by aggregating a plurality of data series. However, in the technique described in Patent Document 1, when an abnormality is detected, It is not possible to automatically identify in which data series an abnormality or change has occurred. Further, when the flow itself increases rapidly as in a denial of service attack (DoS: Denial of Service) that causes excessive access, it is difficult to reduce the number of abnormality detection targets by aggregation.

  Further, in the system described in Patent Document 2, since abnormality detection is performed for all data series, there is a problem that the amount of abnormality detection processing increases in proportion to the number of data series. Further, in the system described in Patent Document 2, abnormal information that does not originate from the same factor is not collected. In addition, in order to detect an abnormality individually for each of a plurality of data series, it is necessary to set a threshold for detecting an abnormality for each data series. Further, when the data tendency in the data series changes, it is necessary to reset the threshold. In this way, detecting an abnormality for each data series requires a separate threshold setting for each data series, and there is a problem that the work load for that is increased. Although it is conceivable to detect anomalies automatically by data mining, anomaly detection is performed at a certain rate due to the influence of statistical processing (for example, the top 0.1% of the distribution of anomaly scores). For this reason, the number of abnormality detections increases in proportion to the number of data series, and it becomes difficult to specify the data series originally desired to be specified.

  In the method described in Non-Patent Document 1, a person specifies which data series is a cause of abnormality. Therefore, when there are many types of data series before aggregation, it may be difficult to specify the data series. For example, it is assumed that one abnormality occurs per day in one data series. If there are 1000 data series, 1000 abnormalities occur per day. In this case, one or more abnormalities occur every two minutes. Further, if there are 100,000 data series, 100,000 abnormalities occur per day. In this case, one or more abnormalities occur per second. When an abnormality occurs at such a pace, it is difficult to manually specify a data series. In addition, when manually specifying a data series, the specific accuracy and time varies depending on the skill of the operator, and it is difficult to continue to specify the data series according to a certain standard.

  Therefore, the present invention provides an anomaly detection system, an anomaly detection method, and an anomaly detection program capable of efficiently detecting which data series has an abnormality or change even when the number of data series is very large. The purpose is to provide.

  An anomaly detection system according to the present invention calculates a data series determined to belong to the same group by calculating the sum of the data values of the data series determined to belong to the same group or the power of the data values. An aggregation means for aggregating data, a statistic calculation means for calculating a statistic of data values of the data series before being aggregated, and a data series in which an abnormality or change has occurred based on the sum calculated for each group A group detecting means for detecting a group including the data sequence specifying means for specifying a data series in which an abnormality or change has occurred based on a statistic from among data series belonging to the group detected by the group detecting means; It is characterized by providing.

  Further, the abnormality detection method according to the present invention is determined to belong to the same group by calculating the sum of the data values of the data series determined to belong to the same group or the power of the data values. Aggregate the data series, calculate the statistics of the data values of the data series before being aggregated, and detect the group containing the data series that is abnormal or changing based on the sum calculated for each group A data series in which an abnormality or a change has occurred is specified based on a statistic from among the data series belonging to the detected group.

  Further, the abnormality detection program according to the present invention belongs to the same group by calculating the sum of the data values of the data series or the powers of the data values determined to belong to the same group in the computer. An anomaly or change occurs based on the aggregation process that aggregates the data series defined as follows, the statistic calculation process that calculates the statistic of the data values of the data series before being aggregated, and the sum calculated for each group A group detection process that detects a group that includes the data series that is being detected, and a data series that is abnormal or changes based on a statistic from among the data series that belong to the group detected by the group detection process A data series specifying process is executed.

  According to the present invention, even if the number of data series is very large, it is possible to efficiently detect which data series has an abnormality or change.

It is a block diagram which shows the example of the abnormality detection system of the 1st Embodiment of this invention. It is a flowchart which shows the example of the process progress of 1st Embodiment. 6 is an explanatory diagram illustrating an example of the number of data generated within a certain period W. FIG. It is explanatory drawing which shows an outlier detection typically. It is explanatory drawing which shows a conversion point detection typically. It is explanatory drawing which shows an abnormal action detection typically. It is explanatory drawing which shows the example of the data value of a some data series. It is explanatory drawing which shows the change of the aggregation result of the data series shown in FIG. The calculated threshold T _k is an explanatory view schematically showing. It is a block diagram which shows the example of the abnormality detection system of the 2nd Embodiment of this invention. It is a flowchart which shows the example of the process progress of 3rd Embodiment. It is a block diagram which shows the minimum structure of this invention. It is a block diagram which shows the other structural example of this invention. It is a block diagram which shows the other structural example of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Embodiment 1. FIG.
FIG. 1 is a block diagram illustrating an example of an abnormality detection system according to the first embodiment of this invention. The abnormality detection system 10 of the present invention includes a data series aggregation means 2, statistical information calculation means 6, conversion information storage means 3, abnormality detection means 4, and data series identification means 5. Then, the abnormality detection system 10 receives a plurality of data series data from the information source 1.

  The information source 1 generates data of a plurality of data series. The data of each data series is numerical data represented by numerical values, and the numerical value represented by this data is referred to as a data value. Note that the information source 1 is not realized by a single device, but may be realized by a plurality of devices. Data of each data series generated by the information source 1 changes with time. Further, the total number of data series is not limited to several, and may be a large number. For example, the total number of data series may be 1000 or more, 10,000 or more.

  The data of each data series may be, for example, the number of traffic per unit time. The information source 1 may generate such data as a log relating to communication traffic. Each data series may be received by the abnormality detection system 10 as a TCP (Transmission Control Protocol) or UDP (User Datagram Protocol) packet, and individual data of each data series may be stored in the payload of the packet. .

  Each data series may correspond to a different port number. That is, the port number corresponding to the data series may be different for each data series.

The data series is grouped in advance. In the example shown in FIG. 1, it is determined that each data series from data series 1 to data series N belongs to the same group, and similarly each data series from data series S _k to data series E _k. Exemplifies a case in which it is determined that they belong to the same group.

More specifically, the data series may correspond to TCP or UDP source port number units (65536 types) and destination port number units (65536 types). In addition, the data series may be determined in units of transmission source IP addresses and transmission destination IP addresses. Type of source IP address and destination IP address, the IPv4 (Internet Protocol Version4), is about 4.3 billion kinds ^{(2 23} kinds).

  The data series aggregating means 2 aggregates data series that are determined to belong to the same group for each group. Specifically, the data series aggregation unit 2 calculates the sum of the data values of the data series determined to belong to the same group or the sum of the powers of the data values. The sum obtained by this calculation is the aggregation result of the data series. The data series aggregation means 2 performs a process for obtaining an aggregation result for each group at regular intervals. This fixed period is represented by the symbol W. Note that when calculating the sum of the powers of the data values, the exponent in the calculation of the power is not limited to a natural number. For example, the exponent in the power calculation may be ½, and the sum of the data values to the power of ½ (that is, the sum of the square roots of the data values) may be calculated.

In the example shown in FIG. 1, the data series aggregation unit 2, and aggregates for the data sequence from the data series 1 to the data series N, likewise, to aggregate with respect to data sequence from the data sequence S _k to the data series E _k . Similarly, other data series are aggregated for each group.

  The data series aggregating means 2 calculates the sum of the data values of each data series in the same group or the sum of the powers of the data values, and obtains a numerical value as an aggregation result. Therefore, the data is collected, and the number of data series is reduced from that before the sum operation.

  The statistical information calculation means 6 calculates the statistic (statistical information) of the data values of the data series before being aggregated by the data series aggregation means 2. Examples of the statistic include an average value and variance. However, the statistic calculated by the statistical information calculation unit 6 is not limited to the average value and variance.

  Hereinafter, the statistical information calculated by the statistical information calculation unit 6 is referred to as conversion information. The conversion information may include information other than statistical information (specifically, exclusion information described in the second embodiment or the like). However, in the first embodiment, the conversion information includes only statistical information. explain.

  The conversion information storage means 3 is a storage device that stores conversion information. The statistical information calculation unit 6 stores the calculated statistical information (conversion information) in the conversion information storage unit 3. A set of conversion information stored in the conversion information storage means 3 can be referred to as a conversion information database.

In the example shown in FIG. 1, the statistical information calculation means 6 calculates statistical information 3 _1a (conversion information 3 ₁ ) for data series 1 to data series N, and stores the statistical information 3 _1a in the conversion information storage means 3. Let Similarly, the statistical information 3 _ka (conversion information 3 _k ) is calculated for the data series S _k to the data series E _k , and the statistical information 3 _ka is stored in the conversion information storage unit 3. Similarly, the statistical information calculation means 6 calculates statistical information for the data series of other groups, and stores the calculated statistical information in the conversion information storage means 3. As a result, the statistical information calculated for each group is stored in the conversion information storage device 3.

The abnormality detection means 4 includes a group including a data series in which an abnormality or a change has occurred based on a transition of an aggregation result aggregated for each group (that is, a data value calculated for each group or a sum of powers of data values). Is detected. In other words, it is determined for each group whether or not a data series in which an abnormality or change occurs exists in the group. An abnormality detection algorithm for determining whether or not a data series in which an abnormality or change has occurred exists in a group, and is determined in advance for each group. The abnormality detection algorithm may be a separate algorithm for each group. For example, the abnormality detection algorithm for determining from the data series 1 whether abnormality in the group until the data sequence N has occurred, normal, etc. in the group from the data sequence S _k to the data series E _k is generated The abnormality detection algorithm for determining whether or not there may be different.

  The outlier detection algorithm described in Patent Documents 6 to 9, the change point detection algorithm described in Patent Documents 10 and 11, or the abnormal behavior detection algorithm described in Patent Document 12 is applied to the abnormality detection means 4. It may be. Other abnormality detection algorithms may be applied as the abnormality detection algorithm.

When a group including a data series in which an abnormality or change has occurred is detected by the abnormality detection unit 4, the data series specifying unit 5 uses the statistical information calculated from the group to assign any data series in the group. Identify whether an anomaly or change has occurred. For example, when the group from the data series 1 to the data series N is detected by the abnormality detecting unit 4, the data series specifying unit 5 uses the statistical information _31a to determine the data series from the data series 1 to the data series N. The data series in which an abnormality or change has occurred is identified. The same applies when other groups are detected.

  The data series aggregation means 2, the statistical information calculation means 6, the abnormality detection means 4, and the data series identification means 5 are realized by a CPU of a computer that operates according to a program (abnormality detection program), for example. In this case, the CPU reads the abnormality detection program from the program storage means (not shown) in which the abnormality detection program is stored, and according to the program, the data series aggregation means 2, the statistical information calculation means 6, the abnormality detection means 4, and What is necessary is just to operate | move as the data series identification means 5. Further, the data series aggregation means 2, the statistical information calculation means 6, the abnormality detection means 4, and the data series identification means 5 may be realized as separate dedicated circuits.

Next, the operation will be described. FIG. 2 is a flowchart illustrating an example of processing progress of the first embodiment. In the following example, description will be given focusing on the data series from the data series S _k to the data series E _k, but the same operation is performed for other groups.

First, the abnormality detection system 10 acquires data of the data series S _{k to} E _k from the information source 1 (step A1). For example, data of each data series transmitted by the information source 1 is received. The number of data series from the data series S _k to the data series E _k is N _k .

Next, the data series aggregation unit 2, aggregate data series to data series N _k from data sequence S _k obtained within a certain period of time W (step A2). For example, the data series aggregation unit 2 calculates the sum of the data values of the data series S _{k to} E _k acquired within a certain period W, and uses the result as the aggregation result. Also, the sum of the powers of the data values may be calculated and the result may be used as the aggregate result. As already described, the square root may be calculated as the power calculation of the data value.

  Further, the number of data acquired within a certain time W for one data series may be one. Or two or more may be sufficient. FIG. 3 is an explanatory diagram showing an example of the number of data generated within a certain period W. In the example shown in FIG. 3, in the data series 1, three data are generated within a certain period W, and in the data series 2, one data is generated within a certain period W. In such a case, all the data acquired within a certain period W may be subject to aggregation. For example, the data series aggregation means 2 calculates the sum of the data values for the three data of the data series 1 shown in FIG. 3 and one data of the data series 2, or calculates the sum of the powers of the data values. do it.

The statistical information calculation means 6 calculates the statistic of the data value acquired from the information source 1, and stores the calculated statistic in the conversion information storage means 3 (step A3). An example of calculating statistics is shown below. However, in each of formulas shown below, v _i is the data value of each data acquired as data for each data series. Acquired up to statistic calculation time, the data value for each data series in the group in question may be v _i. Alternatively, the data value received since the time of going back a predetermined period from the statistic calculation time may be v _i. That is, it is possible to ignore old data before the point in time that goes back for a predetermined period. α _k is an average value of the data values of the data of each data series, and may be obtained by the following equation (1). N _k is the number of data series in the group of interest (in this example, the group of data series S _{k to} E _k ).

For example, the statistical information calculation unit 6 may obtain the statistic σ _k by performing the following equation (2) for the group of interest.

Equation (2) is an equation for obtaining a standard deviation as the statistic σ _k .

Further, for example, the statistical information calculation unit 6 may obtain the statistic σ _k by performing the following equation (3) for the group of interest.

Expression (3) is an expression for obtaining a harmonic average as the statistic σ _k .

In addition, for example, the statistical information calculation unit 6 may obtain the statistic σ _k by performing the following equation (4) for the group of interest.

Equation (4) is the exponential function value of the arithmetic mean of the data values as the statistic σ _k . The base of this exponential function is e. When the statistic is obtained by the equation (4), even if a large value appears in the data value, it is possible to prevent the calculation from becoming impossible.

  Here, three types of formulas (2) to (4) are shown as statistical formulas, but the statistical calculation method is not limited to the above example, and the statistical information calculation means 6 is another formula. Statistics may be calculated.

Subsequent to step A3, the abnormality detection means 4 detects a group including a data series in which an abnormality or change has occurred based on the transition of the aggregation result calculated for each group of data series (step A4). For example, with respect to the group of data series S _{k to} E _k , it is determined whether or not a data series in which an abnormality or change has occurred exists in the group. Then, a group determined to have a data series in which an abnormality or change has occurred is identified.

  The abnormality detection unit 4 may use, for example, outlier detection as an abnormality detection algorithm, and may detect the group that is the source of the aggregation result when the aggregation result is an outlier. FIG. 4 is an explanatory diagram schematically showing outlier detection. Outlier detection is an algorithm that detects, among the obtained data, data 91 that is statistically determined to be out of other data as abnormal (outlier).

  Further, the abnormality detection means 4 may use, for example, conversion point detection as the abnormality detection algorithm, and if the aggregation result changes abruptly, the abnormality detection unit 4 may detect the group that is the source of the aggregation result. FIG. 5 is an explanatory view schematically showing conversion point detection. Change point detection is an algorithm that detects a sudden change in data. In the example shown in FIG. 5, the tendency of data fluctuations changes at time P. The abnormality detection unit 4 may detect such a rapid change in the aggregation result.

  In addition, the abnormality detection unit 4 may detect the group that is the source of the aggregation result if the appearance pattern of the aggregation result is changed by using, for example, abnormal behavior detection as the abnormality detection algorithm. FIG. 6 is an explanatory view schematically showing abnormal behavior detection. In abnormal behavior detection, data transition is monitored, a normal data transition pattern is specified, and if data appears in a pattern not corresponding to the transition pattern, it is detected as abnormal. The abnormality detection unit 4 may identify a normal transition pattern of the aggregation result from the past aggregation results, and detect a group having an abnormality if a change in the aggregation result not corresponding to the pattern is detected.

  When the abnormality detection unit 4 determines that there is no group in which an abnormality or change has occurred (No in step A5), the process returns to step A1.

  When the abnormality detection unit 4 determines that there is a group in which abnormality or change has occurred (Yes in step A5), the data series identification unit 5 determines the statistics of the group (group in which abnormality or change has occurred) detected in step A4. Using the information, it is determined in which data series in the group an abnormality or change has occurred (step A6). In step A6, the data series specifying unit 5 calculates a threshold value from the statistical information of the group detected in step A4, and the data series in which the data value used in the aggregation in step A2 exceeds the threshold value is abnormal or changed. It is determined that the data series is generated.

In step A6, the data series specifying means 5 sets the threshold value as the number of data series in the group in which anomaly or change is detected, the statistic calculated in step A3 for the group, and the average value of the data values of the data series in the group It may be calculated as a function of If there is an abnormality or change in the group of data series S _{k to} E _k, the number of data series is N _k . Further, when the threshold value calculated for the group is T _k , T _k can be expressed as a function of N _k , statistic σ _k , and average value α _k of data values as shown in Expression (5). As such a function, for example, a function exemplified in Expression (6) or Expression (7) can be used. As shown in Expression (6), the threshold value T _k may be calculated without using N _k .

For example, it is assumed that the standard deviation (see formula (2)) is calculated as the statistic σ _k and the data series specifying unit 5 calculates the threshold by the calculation of formula (6). In this case, a value obtained by adding 1/2 of the deviation to the average of the data values becomes a threshold value, and the data series specifying unit 5 detects a data series in which a data value equal to or greater than the threshold value has occurred.

Further, for example, it is assumed that a harmonic average (see formula (3)) is calculated as the statistic σ _k , and the data series identification unit 5 calculates the threshold value by the calculation of formula (7). In this case, the larger the average value, the smaller the difference between the threshold value and the average value. Therefore, when the average value of the data values is small, it is difficult to determine that the data series is abnormal even if there is a data value having a large difference from the average value. When the average value of the data values is large, Even a data value having a small difference from the average value is likely to be determined as a data series in which an abnormality or the like has occurred.

  The abnormality detection system 10 executes the above steps A1 to A6 for each predetermined group.

Next, a process for specifying a group in which an abnormality has occurred and further specifying a data series in which an abnormality has occurred will be described using a specific example. In each graph shown in FIGS. 7A to 7D, the horizontal axis is a data series, and the vertical axis is a data value. FIG. 7 shows data values of the data series S _{k to} E _k . Specifically, the data series S _{k to} E _k indicate a case where there are 10 data series of the data series 3000 to 3009. 7A to 7D show data values of the data series 3000 to 3009 at times “10:00”, “10:01”, “10:02”, and “10:09”, respectively. Yes.

  FIG. 8 is an explanatory diagram showing a change in the aggregation result of the data series shown in FIG. 7, and shows a change in the aggregation result between 10:00 and 10:10. The horizontal axis in FIG. 8 is time, and the vertical axis is the aggregation result at each time. For example, when the data values of the ten data series shown in FIG. 7A are acquired at time 10:00, the data series aggregation unit 2 aggregates the data values. In FIG. 8, a symbol (a) is attached to a location corresponding to the aggregation result obtained by aggregating the data illustrated in FIG. Similarly, symbols (b), (c), and (d) are assigned to locations corresponding to the data aggregation results shown in FIGS. 7 (b), (c), and (d), respectively.

  Further, when the data series are aggregated at each time, the statistical information calculation unit 6 calculates the statistics for the group of the data series 3000 to 3009 and stores them in the conversion information storage unit 3.

The abnormality detection unit 4 determines that the aggregation result of FIG. 8D is different from the other aggregation results, and determines that an abnormality has occurred in the group of the data series 3000 to 3009. Then, the data series identification unit 5 calculates the threshold value T _k using the statistics calculated for this group. 9, the calculated threshold T _k is an explanatory view schematically showing. The graph shown in FIG. 9 is the same as the graph shown in FIG. 7D, and shows data values of the data series 3000 to 3009 at the time when the aggregation result has changed. The threshold value _Tk is superimposed on this graph. When the threshold value T _k shown in FIG. 9 is calculated, the data series specifying unit 5 determines that an abnormality has occurred in the data series in which the data value at the time when the change has occurred in the aggregation result exceeds the threshold value T _k . In this example, it is determined that an abnormality has occurred in the two data series.

  In addition, even if the individual data series does not vary so much, the variation for each data series may be added to cause a large variation in the aggregation result. Even in such a case, the data value of each data series is equal to or less than the threshold value, and the data series specifying unit 5 determines that there is no data series in which an abnormality or fluctuation occurs.

  According to this embodiment, the data series aggregating means 2 aggregates a large amount of data series for each group by calculating the sum, and the abnormality detecting means 4 detects a group in which an abnormality or a change has occurred. Rather than directly determining the presence or absence of an abnormality or the like for each individual data series, the data values are aggregated for each group, thereby reducing the number of determination targets for the presence or absence of an abnormality or change. Then, the data series specifying means 5 detects a data series in which an abnormality or change has occurred from the detected group. As described above, since the determination target of abnormality or change is reduced by aggregation, it is possible to efficiently detect in which data series abnormality or change has occurred.

When a data series is detected from the group, a threshold value is calculated from the statistics (α _k , σ _k, etc.) calculated by the statistical information calculation means 6 to detect a data series exceeding the threshold value. Therefore, it is not necessary to manually set a threshold value for each data series in advance or update the threshold value, and the data series can be detected efficiently.

  Further, even if a certain data value fluctuates, it may not appear as a fluctuation of the aggregation result. Even if such a change in data value is caused by an abnormality, if the change is not an extremely large change that appears as a change in the aggregation result, the abnormality is minor and may be ignored. According to the present invention, it is possible to efficiently detect a data series in which a large abnormality or change has occurred by preventing a minor abnormality from being detected by aggregation.

  In general, when an abnormality or change occurs, the data value of the data series becomes extremely large. Therefore, when determining groups, data series having different data value orders (digits) can be included in the same group. For example, a data series having a value of about 10 at normal time and a data series having a value of about 1000 at normal time can be included in the same group. When an abnormality occurs in a data series having a value of about 10, the data value becomes a value that is larger than 1000. Therefore, the aggregation result itself varies greatly, and it can be detected that abnormality or the like has occurred in the group. Here, when the fluctuation of the data value of the data series having a small order is small (for example, when the value of about 10 is usually about 50), the data series of about 1000 is also included. The fluctuation is not noticeable. In this case, as described above, an abnormality in a data series having a small order may be ignored. In general, when an abnormality or the like occurs, the data value fluctuates greatly. Therefore, the value of the group aggregation result also fluctuates greatly, and it can be determined that an abnormality has occurred in any data series in the group.

  Since the data series with a large order is considered to be more likely to affect the aggregation result when an abnormality occurs, the data series specifying means 5 detects that the data series with the largest order in the detected group is abnormal. Alternatively, it may be determined that a change has occurred. However, even if a group is detected, there is a case where no abnormality occurs in each data series, or there are cases where abnormality occurs in a plurality of data series as shown in FIG. It is preferable to detect the data series by comparing the value with the data value.

Embodiment 2. FIG.
FIG. 10 is a block diagram illustrating an example of the abnormality detection system according to the second embodiment of this invention. Constituent elements similar to those in the first embodiment are denoted by the same reference numerals as those in FIG. 1, and detailed description thereof is omitted. The abnormality detection system 10 of the present invention includes a data series aggregation means 2, statistical information calculation means 6, conversion information storage means 3, abnormality detection means 4, data series identification means 5, and exclusion information editing means 7. Prepare.

In the first embodiment, the conversion information includes only statistical information (statistics). However, in the present embodiment, exclusion information is also included in the conversion information. The exclusion information is information indicating a data series to be excluded from the aggregation target, and is determined for each group, for example. In the example shown in FIG. 10, exclusion information 3 _1b is defined for the groups of data series 1 to N, and exclusion information 3 _kb is defined for the groups of data series S _{k to} E _k . Further, if there is no data series to be excluded in the group, there is no need for exclusion information.

  The exclusion information editing unit 7 edits the exclusion information of each group. Here, the case where the exclusion information editing unit 7 edits the exclusion information according to the user's operation will be described as an example. However, the exclusion information may be edited in another manner. The exclusion information editing unit 7 stores new exclusion information in the conversion information storage unit 3 in accordance with a user operation. Further, the exclusion information stored in the conversion information storage unit 3 may be updated or deleted according to the user's operation.

In the present embodiment, the data series aggregating means _2a aggregates the data series for each group. This is the same as in the first embodiment. However, the data series aggregation unit _2a excludes the data series indicated by the exclusion information of the group of interest from the aggregation target. For example, among the data series _S k to E _k, and data series _{S k} is specified by the excluded information. In this case, the data series aggregating unit 2 _a performs aggregation using data values of data series other than the data series S _k among the data series S _{k to} E _k . The other points are the same as those of the data series aggregation unit 2 of the first embodiment.

The statistical information calculation means 6a calculates _a statistic for each group. This is the same as in the first embodiment. However, the statistics calculating unit 6 _a is a data sequence indicating the exclusion information of the group of interest, to exclude from the calculation of statistics. For example, among the data series _S k to E _k, and data series _{S k} is specified by the excluded information. In this case, the statistical information calculation unit 6 _a calculates σ _k and α _k using data values of data series other than the data series S _k among the data series S _{k to} E _k . At this time, as the number of data series N _k , a value obtained by subtracting the number of data series indicated by the exclusion information (the number of excluded data series) from a predetermined N _k may be used.

Data series aggregation unit 2 _a and statistics calculation means 6 _a is realized by a CPU that operates according to example program. In this case, the CPU may operate as the data series aggregation means 2 _a , the statistical information calculation means 6 _a and other means according to the program.

  Also in the second embodiment, the processing of steps A1 to A6 may be performed as in the first embodiment.

  According to the second embodiment, it is possible to prevent such a data series from being detected by designating a data series that does not need to be detected as an exception information. Examples of data series that do not require abnormality detection include a data series that is already known to be under constant attack, a data series that cannot acquire data due to a failure, and the like. In such a data series, abnormalities continue to appear, data does not exist, or does not change. It is possible not to detect such a data series in which occurrence of abnormality or the like is known in advance.

Embodiment 3. FIG.
The third embodiment has the same configuration as that of the second embodiment (see FIG. 10). However, the data series specifying means 5 also adds exclusion information. When a group is detected by the abnormality detection unit 4, the data series identification unit 5 detects a data series in which an abnormality or change has occurred from the group. This operation is the same as in the first embodiment and the second embodiment. In the third embodiment, when the data series identification unit 5 detects the data series, the data series identification unit 5 stores the exclusion information indicating the data series in the conversion information storage unit 3 as the exclusion information of the group to which the data series belongs. . Other components are the same as those in the second embodiment.

  FIG. 11 is a flowchart illustrating an example of processing progress of the third embodiment. Steps A1 to A6 are the same as those in the first and second embodiments. After detecting the data series in which an abnormality or the like has occurred (after step A6), the data series specifying unit 5 converts the exclusion information indicating the detected data series as the exclusion information of the group to which the data series belongs. The data is stored in the storage unit 3 (step A7).

  As a result, the data series detected as a data series in which an abnormality or the like has occurred is excluded from the aggregation target. Then, when data is continuously acquired and aggregated, the data series is treated as an object of aggregation processing.

  In the third embodiment, since the data series specifying unit 5 stores the exclusion information indicating the detected data series in the conversion information storage unit 3, the data series in which an abnormality has occurred is continuously detected many times. Can be prevented.

  In addition, the data value of the detected data series has fluctuated greatly, but if such a data series continues to be included as an aggregation target, if there is an abnormality in another data series, the data The fluctuation of the series is buried in the aggregation result (in other words, it is difficult to be reflected in the aggregation result), and it becomes difficult to detect an abnormality in another data series. In the third embodiment, since the detected data series is excluded from aggregation targets, such a problem can be prevented.

  When it is confirmed that the data series detected in step A6 is not abnormal or the abnormality has converged, the original detection state can be restored by deleting the information of the data series from the exclusion information. .

  In the third embodiment, the exclusion information editing unit 7 may not be provided. In other words, the exclusion information editing unit 7 may not edit the exclusion information, and only the data series identification unit 5 may add the exclusion information.

  In each of the first to third embodiments, a case has been described in which the statistical information calculation means 6 calculates statistics (step A3) in conjunction with the data series aggregation processing (step A2). When the group is detected by the abnormality detection unit 4 (that is, when a group that is determined to include a data series in which an abnormality or change has occurred is detected), the statistical information calculation unit 6 uses the data value of the group. Statistics may be calculated and stored in the conversion information storage unit 3. The statistic calculation method itself is the same as that of the embodiment already described.

  Thus, by calculating the statistics only for the groups detected by the abnormality detection means 4, the statistical information stored in the conversion information storage means 3 can be reduced.

  Next, the minimum configuration of the present invention will be described. FIG. 12 is a block diagram showing the minimum configuration of the present invention. The anomaly detection system of the present invention comprises an aggregation means 71, a statistic calculation means 72, a group detection means 73, and a data series identification means 74.

Aggregating means 71 (for example, data series aggregating means 2, 2 _a ) belongs to the same group by calculating the sum of the data values of the data series determined to belong to the same group or the powers of the data values. Aggregate data series determined to be

The statistic calculation means 72 (for example, the statistical information calculation means 6, 6 _a ) calculates the statistic of the data value of the data series before being aggregated.

  The group detection unit 73 (for example, the abnormality detection unit 4) detects a group including a data series in which an abnormality or a change has occurred based on the sum calculated for each group.

  The data series specifying unit 74 (for example, the data series specifying unit 6) specifies the data series in which an abnormality or change has occurred based on the statistic from the data series belonging to the group detected by the group detecting unit 73. .

  With such a configuration, even if the number of data series is very large, it is possible to efficiently detect which data series has an abnormality or change.

  13 and 14 are block diagrams showing other configuration examples of the present invention. As shown in FIG. 13, the above embodiment includes an exclusion information storage unit 75 (for example, conversion information storage unit 3) that stores exclusion information that is information indicating a data series excluded from aggregation. 71 discloses a configuration in which the data series indicated by the exclusion information is excluded from the calculation target of the data value or the sum of the power of the data value. According to such a configuration, it is possible to prevent such a data series from being detected by designating a data series that does not require abnormality detection with the exclusion information.

  In the above embodiment, as shown in FIG. 14, an exclusion information editing unit 76 (for example, the exclusion information editing unit 7) that edits exclusion information stored in the exclusion information storage unit 74 in accordance with a user operation. ) Is disclosed.

  In the above-described embodiment, a configuration is disclosed in which exclusion information storage means stores exclusion information indicating a data series specified by the data series identification unit 74 as a data series in which an abnormality or change has occurred. According to such a configuration, it is possible to prevent the same data series from being detected continuously.

  Further, the above embodiment discloses a configuration in which the statistic calculation means 72 calculates the statistic of the data value of the data series when the data series determined to belong to the same group is aggregated. Has been.

  In the above embodiment, the statistic calculation means 72 calculates the statistic of the data value of each data series belonging to the group and aggregated by the aggregation means 71 when the group detection means 73 detects the group. The structure to perform is disclosed.

  Further, the above embodiment discloses a case where each data series is a data series corresponding to a different port number.

  Further, the above embodiment discloses a case where each data series is a data series generated as a log relating to communication traffic.

  The present invention is preferably applied to an abnormality detection system that detects which data series in a plurality of data series are abnormal or changed. For example, the present invention can be applied to a system for monitoring a change in a communication packet and detecting an abnormality in units of TCP or UDP transmission source port numbers, transmission destination port numbers, transmission source IP addresses, and transmission destination IP addresses. Also, for example, monitoring of changes in the number of accesses in units of URLs (Uniform Resource Locators) in large-scale Web servers, changes in the amount of mail flow in units of sender and destination mail addresses in large-scale mail servers, and abnormality detection Applicable to systems that perform

DESCRIPTION OF SYMBOLS 1 Information source 2, 2 _a Data series aggregation means 3 Conversion information storage means 4 Abnormality detection means 5 Data series specification means 6, 6 _a Statistical information calculation means 7 Exclusion information editing means 10 Abnormality detection system

Claims (12)

An aggregation means for aggregating the data series determined to belong to the same group by calculating the sum of the data values of the data series determined to belong to the same group or the power of the data values;
A statistic calculation means for calculating a statistic of data values of the data series before being aggregated;
Group detection means for detecting a group including a data series in which an abnormality or change occurs based on the sum calculated for each group;
An abnormality detection system comprising: a data series specifying means for specifying a data series in which an abnormality or a change has occurred based on the statistic from among data series belonging to the group detected by the group detection means. . Exclusion information storage means for storing exclusion information that is information indicating a data series excluded from aggregation,
The abnormality detection system according to claim 1, wherein the aggregation unit excludes the data series indicated by the exclusion information from a calculation target of a data value or a sum of data value powers. The abnormality detection system according to claim 2, further comprising an exclusion information editing unit that edits the exclusion information stored in the exclusion information storage unit in response to a user operation. The abnormality detection system according to claim 2 or 3, wherein the data series specifying unit stores exclusion information indicating the data series specified as a data series in which an abnormality or change has occurred in the exclusion information storage unit. The statistic calculation means calculates a statistic of a data value of the data series when the data series determined to belong to the same group is aggregated. The abnormality detection system according to item 1. The statistic calculation means, when the group detection means detects a group, calculates a statistic of the data value of each data series belonging to the group and aggregated by the aggregation means. The abnormality detection system according to claim 1. The abnormality detection system according to any one of claims 1 to 6, wherein each data series is a data series corresponding to a different port number. The abnormality detection system according to any one of claims 1 to 7, wherein each data series is a data series generated as a log relating to communication traffic. Aggregate data series defined as belonging to the same group by calculating the sum of the data values or data value powers of the data series determined to belong to the same group,
Calculate the statistics of the data values of the data series before being aggregated,
Based on the sum calculated for each group, detect groups that contain data series that are abnormal or changing,
An abnormality detection method, characterized in that, from among data series belonging to a detected group, a data series in which an abnormality or change has occurred is identified based on the statistics. The abnormality detection method according to claim 9, wherein a data series indicated by exclusion information that is information indicating a data series excluded from aggregation is excluded from a calculation target of a data value or a sum of powers of data values. On the computer,
An aggregation process for aggregating data series determined to belong to the same group by calculating the sum of the data values of the data series determined to belong to the same group or the power of the data values;
A statistic calculation process that calculates the statistics of the data values of the data series before being aggregated,
A group detection process for detecting a group including a data series in which an abnormality or change occurs based on a sum calculated for each group; and
An abnormality detection program for executing a data series identification process for identifying a data series in which an abnormality or a change has occurred based on the statistic from among data series belonging to a group detected by a group detection process. In a computer comprising exclusion information storage means for storing exclusion information that is information indicating a data series excluded from aggregation,
The abnormality detection program according to claim 11, wherein the data series indicated by the exclusion information is excluded from a calculation target of a data value or a sum of powers of data values in the aggregation processing.

Images