JP2017215765A - Abnormality detector, abnormality detection method and abnormality detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect abnormality of data to be detected, based on relationship among data.SOLUTION: An abnormality detector 10 includes: a divergence value vector calculation unit 122 configured to calculate a divergence value vector indicating divergence of detection data as a detection object from relationship among data, based on the relationship among data, and a divergence value vector indicating divergence of comparison data as a comparison object from the relationship among data; an abnormality degree calculation unit 123 configured to calculate discrete degree of the divergence value vector of the detection data from aggregation of the divergence value vectors of the comparison data, as degree indicating abnormality; and an abnormality determination unit 124 configured to determine that the detection data is abnormal when the discrete degree exceeds a predetermined threshold.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an abnormality detection device, an abnormality detection method, and an abnormality detection program.

  In a system composed of a large number of devices such as servers and routers, it is necessary to detect hardware abnormalities such as hard disks and memories of each device. There is a demand not only for information / communication equipment but also for industrial machines and automobiles having a large number of sensors to detect machine abnormalities from sensor data such as temperature and acceleration. In addition to devices, for example, detection of unauthorized use by analyzing credit card usage, abnormal detection technology used in the field of DDoS (Distributed Denial of Service) attacks and malware detection in information security It has been.

  As the simplest abnormality detection method, a method in which an abnormality in data is detected by a human directly viewing the data can be considered. However, it is difficult to determine an abnormality with the human eye because it requires considerable skill to grasp the tendency when a lot of data shows an abnormality. Even if an abnormality can be detected by human eyes, it depends on a specific skilled worker, and if the worker disappears, the abnormality cannot be detected. In view of the above, it is necessary to mechanically execute the mechanism of abnormality detection by some method.

  One of the simplest methods for detecting such mechanical anomalies is a method in which a threshold value is provided in the data range, and an abnormality is determined when the threshold value is exceeded. Although this method may work properly, it is generally difficult to determine a reasonable threshold. This is because, if the threshold value is increased, an event that is originally supposed to be detected as an abnormality may be missed. On the other hand, if the threshold value is decreased, an event that is determined to be abnormal although not abnormal is increased.

  Thus, LOF has been proposed as a widely used method for detecting an abnormality that has not occurred so far and determining an abnormality (for example, see Non-Patent Document 1). LOF is a method for calculating local density in data space.

  Specifically, the LOF outputs a small numerical value indicating the degree of abnormality when newly obtained data is present at a high density location in the data space obtained so far. In other words, the LOF determines that the newly obtained data is not abnormal when the newly obtained data is similar to the data obtained so far. On the other hand, when newly obtained data is present at a location with low density, a numerical value representing the degree of abnormality is output to a large value. In other words, the LOF determines that the newly obtained data is abnormal when the newly obtained data is not similar to the data obtained so far.

  In addition, when there is some relationship between the data, there is a method of detecting an abnormality because the relationship is broken. As this method, a method of analyzing whether or not correlation between a plurality of types of data is maintained and finding an abnormality from the result has been proposed. In this method, two sets of data are selected, a function that predicts one from the other by simple regression, for example, is constructed, and the correlation is considered to be destroyed because the predicted value is more than a certain distance from the observed value. Yes.

M. Breunig, H. Kriegel, R. Ng, and J. Sander, “LOF: Identifying Density-Based Local Outliers”, SIGMOD, Volume 29 Issue 2, 93-104, 2000

  However, LOF has a problem that when there is a correlation between data, even if the data conforms to the correlation and is normally normal, data that is out of the data set is detected as abnormal. Specifically, the problems of the prior art will be described with reference to FIGS. 13 to 15 are diagrams for explaining an abnormality detection method according to the related art. In FIGS. 13 to 15, assuming that a set of X and Y is given as data, the set is plotted on a coordinate plane.

  In FIG. 13, as a point for a set of X and Y, a two-dimensional normal distribution with an average vector of (3, 3) and a covariance matrix of [[1, 0.9], [0.9, 1]] 200 points (white circles) are plotted according to the above. The point Pb located at the upper right is the point (6.5, 6.5). The upper left point Pr is the point (2, 5).

  In FIG. 13, points for a set of X and Y are plotted in the image of the case where the following correlation is found between data. For example, let X and Y represent the CPU (Central Processing Unit) usage rates of the application server A and the application server B, respectively. Then, as an example, the application server A and the application server B receive requests from the Web server equally.

  In this case, if the number of accesses to the Web server increases and the number of requests to the application server increases, both X and Y are considered to increase. Conversely, when the number of accesses to the Web server decreases, both X and Y are considered to decrease. In FIG. 13, white circles are examples of data in a normal state. The point Pb is an example in a case where the correlation is maintained and a value that has not existed until then is taken. The point Pr is an example when the correlation is broken. Hereinafter, a situation is considered in which white circles, which are data in a normal state, are given first, and then points Pb and Pr are given as abnormality detection target data.

  Among these points, the point Pb maintains a correlation that holds in a normal state, and therefore it may be determined that the point Pb is not normally abnormal. On the other hand, the point Pr does not follow the correlation established in the normal state, and therefore it may be determined to be abnormal. However, since the relationship between data is not taken into account in LOF, a large numerical value indicating the degree of abnormality is output for both the point Pb and the point Pr existing at points where the density of white circles is low. That is, there is a problem that it is determined to be abnormal at the point Pb that should not be determined to be abnormal.

  On the other hand, in the method of analyzing whether or not the correlation between a plurality of types of data is maintained and finding an abnormality from the result, there are a plurality of relationships between the data as shown in FIG. 14 and FIG. There is a problem that it is difficult to detect an abnormality when there is a significant relationship.

  For example, the white circles in FIG. 14 are 200 points (data group Rb) according to a two-dimensional normal distribution with an average vector of (3, 3) and a covariance matrix of [[1, 0.99], [0.99, 1]]. ) And a point (data group Rb ′) obtained by rotating these counterclockwise by (π / 5). In the example of FIG. 14, there are two relationships between data, and it is difficult to determine which correlation the detected data should be compared with. And in this FIG. 14, the correlation coefficient of the point displayed with the white circle is as small as “0.07”. Therefore, even in the method of detecting an abnormality because the relationship is broken, in the example of FIG. 14, it is determined that no correlation is found even if the data is in a normal state. difficult.

  FIG. 15 shows an image in a case where the normal state constitutes three separate groups as viewed in data. White circles in FIG. 15 are data in a normal state. The point Pg is data for abnormality determination. The straight line Lb is a single regression line (specifically, having a relationship of “Y = 0.9175X + 0.1081”) calculated from only normal white circles and plotted in FIG. In FIG. 15, the white circle group at the lower left (data group Rc) plots 100 points that are random numbers according to a normal distribution in which X and Y each have an average of “0.5” and a standard deviation of “0.1”. It is a thing. The group on the right straight line Lb (data group Rd) is a random number according to a normal distribution in which X is an average “3”, standard deviation “0.1”, Y is an average “4”, and standard deviation “0.1”. The 20 points taken are plotted. The group below the right straight line Lb (data group Re) is a random number according to a normal distribution with X being an average of “4” and standard deviation of “0.1”, and Y being an average of “3” and standard deviation of “0.1”. The 20 points taken are plotted. Note that the white circle has a high correlation coefficient of “0.9231”.

  Here, the point Pg shown in FIG. 15 is clearly far from the three data groups Rc to Rd of white circles. However, since the position is close to the straight line Lb of the single regression line, it is not possible to determine that the point Pg is broken from the correlation corresponding to the straight line Lb only by comparing the point Pg with the straight line Lb. That is, the point Pg is present at a location distant from the normal state data, but in the method of detecting an abnormality because the relationship is broken, the point Pg is detected as abnormal. I can't.

  The present invention has been made in view of the above, and provides an anomaly detection apparatus, an anomaly detection method, and an anomaly detection program capable of accurately detecting anomaly of detection target data based on the relationship between data. The purpose is to do.

  In order to solve the above-described problems and achieve the object, the abnormality detection device according to the present invention represents a deviation of the detection data, which is a detection target, from the relationship between the data based on the relationship between the data. Deviation value vector calculation unit for calculating deviation value vector and deviation value vector representing deviation from relationship between comparison data of comparison data to be compared, and deviation of comparison data between deviation value vector of detection data An abnormality degree calculation unit that calculates a discrete degree from a set of value vectors as a degree indicating abnormality, and an abnormality determination part that determines that the detected data is abnormal when the discrete degree exceeds a predetermined threshold value. Have.

  According to the present invention, it is possible to accurately detect abnormality of detection target data based on the relationship between data.

FIG. 1 is a block diagram illustrating an example of the configuration of the abnormality detection device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a data configuration to be processed in the first embodiment. FIG. 3 is a diagram illustrating an example of a deviation value vector calculation process performed by the deviation value vector calculation unit illustrated in FIG. 1. FIG. 4 is a diagram illustrating another example of the deviation value vector calculation process performed by the deviation value vector calculation unit illustrated in FIG. 1. FIG. 5 is a diagram for explaining the degree of abnormality calculation processing by the degree of abnormality calculation unit shown in FIG. FIG. 6 is a diagram for explaining another example of the abnormality degree calculation processing by the abnormality degree calculation unit shown in FIG. FIG. 7 is a flowchart showing a processing procedure of abnormality detection processing executed by the abnormality detection device shown in FIG. FIG. 8 is a diagram for explaining the abnormality detection process of the first embodiment. FIG. 9 is a diagram for explaining abnormality degree calculation processing according to the third embodiment. FIG. 10 is a diagram for explaining the abnormality degree calculation processing according to the third embodiment. FIG. 11 is a diagram for explaining an abnormality degree calculation process and an abnormality determination process according to the third embodiment. FIG. 12 is a diagram illustrating an example of a computer in which an abnormality detection apparatus is realized by executing a program. FIG. 13 is a diagram for explaining an abnormality detection method according to the related art. FIG. 14 is a diagram for explaining an abnormality detection method according to the related art. FIG. 15 is a diagram for explaining the abnormality detection method according to the related art.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited by this embodiment. Moreover, in description of drawing, the same code | symbol is attached | subjected and shown to the same part.

[Embodiment 1]
First, the first embodiment will be described. In the following embodiments, the configuration of the abnormality detection device according to the first embodiment and the flow of processing by the abnormality detection device will be described.

[Configuration of anomaly detection device]
FIG. 1 is a block diagram illustrating an example of the configuration of the abnormality detection apparatus 10 according to the first embodiment. As illustrated in FIG. 1, the abnormality detection device 10 according to the first embodiment includes a communication processing unit 11, a control unit 12, and a storage unit 13.

  The communication processing unit 11 controls communication related to various types of information exchanged with the connected terminal device 20. For example, the communication processing unit 11 receives from the terminal device 20 a request for abnormality detection processing for data to be compared, data to be detected, and data to be detected. For example, the communication processing unit 11 transmits the processing result of the abnormality detection process to the terminal device 20.

  The control unit 12 has an internal memory for storing a program that defines various processing procedures and the necessary data, and executes various processes using these programs. For example, the control unit 12 is an electronic circuit such as a CPU or MPU (Micro Processing Unit). The control unit 12 includes a relationship estimation unit 121, a divergence value vector calculation unit 122, an abnormality degree calculation unit 123, and an abnormality determination unit 124.

  The relationship estimation unit 121 estimates the relationship between data and calculates a parameter indicating the relationship between data. For example, for the given data, the parameter is estimated when the relationship between the data is given as an equation, but the data corresponding to the parameter is undetermined. Specifically, it is given that the relationship between the data X and the data Y is a single regression of “Y = aX + b”, but the relationship estimation is performed when “a” and “b” are unknown. The unit 121 estimates “a” and “b” based on the given data. In this case, it is desirable that the relationship estimation unit 121 uses data in which the device or the like that outputs the data is in a normal state, in other words, data that does not include data in an abnormal state for parameter estimation.

  The divergence value vector calculation unit 122 calculates a divergence value vector representing the divergence from the relationship based on the relationship between the data. In the first embodiment, the divergence value vector calculation unit 122 calculates a divergence value vector representing the divergence of the detection data that is the detection target from the relationship between the data based on the relationship between the data. Then, the divergence value vector calculation unit 122 calculates a divergence value vector representing the divergence from the relationship between the data of the comparison data to be compared based on the relationship between the data. That is, the divergence value vector calculation unit 122 calculates a value indicating a divergence from the relationship for the detection data and the comparison data when some relationship is found between the data. The divergence value vector calculation unit 122 may store a divergence value vector between data in a divergence value vector storage unit 131 of the storage unit 13 (described later). Further, the divergence value vector calculation unit 122 applies the parameter indicating the relationship between the data calculated by the relationship estimation unit 121 to the relationship between the data, and calculates a divergence value vector between the data. The relationship between the data may be given in advance.

  The abnormality degree calculation unit 123 calculates the degree of discreteness of the deviation value vector of the detection data from the set of deviation value vectors of the comparison data. Specifically, the degree-of-abnormality calculation unit 123 calculates a discrete degree based on a spatial distance, a density, and the like of a deviation value vector of detection data from a set of deviation value vectors of comparison data. The discrete degree calculated by the abnormality degree calculation unit 123 is used in the determination in the abnormality determination unit 124 (described later) as the abnormality degree indicating abnormality. Note that the degree of abnormality calculation unit 123 may calculate the degree of discreteness using a deviation value vector stored in a deviation value vector storage unit 131 (described later).

  The abnormality determination unit 124 determines that the detected data is abnormal when the discrete degree calculated by the abnormality degree calculation unit 123 exceeds a predetermined threshold. The abnormality determination unit 124 determines that the detection data is normal when the discrete degree is equal to or less than a predetermined threshold. The determination result by the abnormality determination unit 124 is output to the terminal device 20 through the communication processing unit 11 as an abnormality detection result, for example.

  The storage unit 13 is realized by a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk, and a processing program for operating the abnormality detection device 10 or a processing program The data used during the execution of is stored. The storage unit 13 includes a deviation value vector storage unit 131 that stores the deviation value vector calculated by the deviation value vector calculation unit 122.

[Example of data to be processed]
Next, an example of data to be processed in the abnormality detection apparatus 10 will be described. FIG. 2 is a diagram illustrating an example of a data structure to be processed in the abnormality detection apparatus 10.

As shown in FIG. 2, for example, it is assumed that N types of data “X ¹ ” to “X ^N ” are given as detection data. In FIG. 2, “x ⁿ _m ” is an observed value at m of the n-th data. The subscript “m” means an observed point or time point. For example, when “X ¹ ” to “X ^N ” indicate the user “N” from the user “1” and the data element indicates the presence / absence of purchase of the product “m”, the data with the same subscript is the purchase of the same product It means presence or absence. Alternatively, when “X ¹ ” to “X ^N ” indicate the CPU usage rate of the server “1” to the server “N” and the data element indicates the CPU usage rate at the time “m”, the data with the same subscript is It means that observation time is equal.

[Relationship Estimator Processing]
The relationship estimating unit 121 estimates the relationship that holds between the "X ^1" - "X ^N", calculates a parameter indicating the relationship between "X ^1" - "X ^N".

  In this case, it is desirable that the relationship estimation unit 121 use data for the parameter calculation using data in which the device or the like that outputs the data is in a normal state, in other words, data that does not include data in an abnormal state. Whether the data is in a normal state may be determined from the fact that the device is normal. Further, whether or not the data is in a normal state may be determined visually by human eyes based on the fact that the data does not include data corresponding to abnormal values. This is because an indicator that “differs from normal” is regarded as “abnormal” in the abnormality detection in the abnormality determination unit 124 is used. Furthermore, the relationship estimation unit 121 can be expected to improve the sensitivity of abnormality detection by estimating a relational expression that holds normal data using only normal data.

  For example, if the relationship between the data is given as an equation, but the one corresponding to the parameter is undetermined, the relationship estimating unit 121 estimates this parameter. Specifically, it is given that the relationship between the data X and the data Y is a single regression of “Y = aX + b”, but the relationship estimation is performed when “a” and “b” are unknown. The unit 121 estimates “a” and “b” based on the given data.

In addition, when the relationship estimation unit 121 is given a multiple regression equation with any of “X ¹ ” to “X ^N ” as an objective variable and the rest as explanatory variables, for example, Find the regression parameters. Further, the relationship estimation unit 121 selects two sets of “X ¹ ” to “X ^N ” as the relationship between the data, and sets one of the two sets as an objective variable and the other as an explanatory variable. When a single regression equation is given, parameters of this single regression equation are obtained.

Alternatively, as the relationship between data, each of the “X ^N ” is series data, and the relationship estimation unit 121 provides this self-regression formula or vector auto-regression formula for predicting the future from past data. You may obtain | require the parameter of a regression type or a vector autoregressive type. The subscript “1, 2,..., M,...” Has an order, for example, an example in which the CPU usage rate of the server is data.

  Further, the relationship between data may be modeled by a mixed distribution model, or may be expressed by an expression representing a more complicated nonlinear relationship. For example, the relationship between data may be expressed using some probability distribution. For example, when the relationship between the data is expressed by a mixed distribution model having “K” clusters, the relationship estimation unit 121 obtains a parameter of the relationship expressed by equation (1).

Note that in the present embodiment, when the relationship that holds between “X ¹ ” to “X ^N ” is given in advance, the calculation processing of the relationship estimation unit 121 can be omitted.

[Processing of deviation value vector calculation unit]
Subsequently, the divergence value vector calculation unit 122 calculates a divergence value vector from the relationship established between “X ¹ ” to “X ^N ” and “X ¹ ” to “X ^N ”. FIG. 3 is a diagram illustrating an example of a deviation value vector calculation process performed by the deviation value vector calculation unit 122.

In the example illustrated in FIG. 3, the divergence value vector is calculated for each subscript “m”. Further, in this example, data “X ¹ ” to “X ^N ” shown in FIG. 3 are given, and the relationship established between these “X ¹ ” to “X ^N ” is expressed by “F (X ¹ , X ^2). ,..., X ^N ) = 0 ”. In this example, it is assumed that the relational expression is a multiple regression expression in which any one of “X ¹ ” to “X ^N ” is an objective variable and the rest is an explanatory variable.

Specifically, the “divergence value of X ¹ ” in FIG. 3 is estimated using the relationship from the observed value “x ¹ _m ” at the subscript “m” and the data observed at “m”. And the value of “X ¹ ” “F (x ¹ _m , x ² _m ,..., X ^N _m )”. Further, "divergence values of X ^2" is the observed value of the subscript "m" and "x ² _m" from the observed data in the "m", the value of "X ^2" which is estimated using the relationship " F (x ¹ _m , x ² _m ,..., X ^N _m ) ”. The divergence value vector calculation unit 122 calculates each divergence value of “X ¹ ” to “X ^N ” by performing the calculation process illustrated in FIG. 3. Then, as shown in FIG. 3, the divergence value vector calculation unit 122 outputs the divergence value calculated from a plurality of types of data having a common subscript “m” as a divergence value vector.

In addition, the relationship between data is not only when one relationship is given to all data, such as “F (X ¹ , X ² ,..., X ^N ) = 0”, but also data In some cases. For example, there is a case where two sets are selected from “X ¹ ” to “X ^N ”, and relevance is given by a single regression equation in which one set is an objective variable and the other is an explanatory variable. Therefore, the divergence value vector calculation process in this case will be described with reference to FIG. FIG. 4 is a diagram illustrating another example of a divergence value vector calculation process performed by the divergence value vector calculation unit 122.

As shown in FIG. 4, for example, “the divergence value of X ¹ as viewed from X ² ” is obtained from the observed value “x ¹ _m ” at the subscript “m” and the data observed at “m” as “X ¹ ”And the value“ F ¹² (x ¹ _m , x ² _m ) ”of“ X ¹ ”estimated using the relationship“ F ¹² ”between“ X ² ”and the difference value. Divergence value vector calculation unit 122, by performing a calculation process shown in FIG. 4, calculates a "divergence values viewed from X ² of X ^1" - "divergence values viewed from X ^N-1 of X ^N". Then, as shown in FIG. 4, the divergence value vector calculation unit 122 outputs divergence values calculated from a plurality of types of data having a common subscript “m” as divergence value vectors.

Although not particularly specified in FIG. 4, it may be considered that the relationship between data holds only for some combinations. For example, the correlation coefficient is large between “X ¹ ” and “X ² ”, whereas the correlation coefficient is small between “X ¹ ” and “X ³ ”. This is a case where it is not allowed. In such a case, the divergence value vector calculation unit 122 may calculate the divergence value only for those for which a relationship is recognized.

Further, when the relationship between the data is expressed by the mixed distribution model shown in the equation (1), the divergence value vector calculation unit 122 assigns the degree of belonging to each cluster defined by the following equation (2) “ m _k "is calculated. It can be said that the greater the degree of belonging, the higher the degree of belonging to the cluster, that is, closer to the center point of the cluster. In other words, it can be said that the greater the degree of attribution, the smaller the degree of deviation from the cluster.

In this case, the divergence value vector calculation unit 122 uses “(m ₁ , m ₂ ,..., M _K )” as the divergence value vector for the degree of membership “m _k ” calculated using the equation (2). Ask for. Alternatively, the divergence value vector calculation unit 122 has a negative value such as “−logΣπ ^k P (x | θ ^k )” as the divergence value vector with respect to the degree of membership “m _k ” calculated using the equation (2). The log likelihood may be calculated. In this case, the divergence value vector is one-dimensional.

  The divergence value vector calculation unit 122 calculates the divergence value vector representing the divergence from the relationship between the detection data and the comparison data based on the relationship between the data by performing the calculation process as described above. To do. In general, there are a plurality of comparison data. Of course, the comparison data may be one.

[Processing of abnormality level calculation part]
Next, the processing of the abnormality degree calculation unit 123 will be described. The degree-of-abnormality calculation unit 123 calculates a discrete degree using the divergence value vector calculated from the detection data by the divergence value vector calculation unit 122 and the divergence value vector calculated from the comparison data.

  Specifically, the degree-of-abnormality calculation unit 123 calculates how far the deviation value vector of the detection data is spatially separated from the set of deviation value vectors of the comparison data. In this case, the degree-of-abnormality calculation unit 123 uses, for example, a k-NN (k-nearest neighbor method) method to determine how far the deviation value vector of the detection data is spatially separated from the set of deviation value vectors of the comparison data. To calculate. With reference to FIG. 5, the case where the degree-of-abnormality calculation unit 123 calculates the discrete degree using the k-NN method will be described.

  FIG. 5 is a diagram for explaining the abnormality degree calculation processing by the abnormality degree calculation unit 123. FIG. 5 is a diagram in which each divergence value vector calculated by the divergence value vector calculation unit 122 from the comparison data and the detection data is plotted with respect to the first to third divergence value vectors. In FIG. 5, a white circle data group R1 located in the vicinity of the origin corresponds to a deviation value vector calculated from the comparison data. The point P1 corresponds to the deviation value vector calculated from the detection data (see (1) in FIG. 5). In the k-NN method, the distance to the point Pk closest to the kth point as viewed from the point P1 corresponding to the deviation value of the detection data is calculated as the degree of abnormality (discrete degree) (see (2) in FIG. 5). Here, “k” is a parameter and is set using heuristics.

  In addition, the degree of abnormality calculation unit 123 may calculate how spatially sparse the divergence value vectors of the detection data exist from the set of divergence value vectors of the comparison data. In this case, the degree-of-abnormality calculation unit 123 calculates, for example, how far the divergence value vector of the detection data is spatially separated from the set of divergence value vectors of the comparison data using LOF.

  FIG. 6 is a diagram for explaining another example of the abnormality degree calculation processing by the abnormality degree calculation unit 123. LOF is a technique for calculating local density in space. The white circle data group R1 in FIG. 6 is a collection of points corresponding to the deviation value vector of the comparison data, and the point P1 is a point corresponding to the deviation value vector calculated from the detected data ((1) in FIG. 6). reference).

  Specifically, the average of the distances to the points Pk closest to the kth, as viewed from the divergence value (point P1) of the detection data, is obtained from the kth point Pk to the point Pm closest to the mth. The value divided by the average of the distance is calculated as the degree of abnormality (discrete degree) (see (2) in FIG. 6). For example, when there is a point P1 corresponding to the deviation value vector of the detected data at a high density position in the data group R1, the average of the distances from the point P1 to the kth closest point Pk becomes small, and the point Pk Since the distance from to point Pm is also small, the degree of discreteness is small. On the other hand, when the point P1 is located at a low density position in the data group R1, the average of the distance from the point P1 to the kth closest point Pk is large, so the degree of discreteness is small. “K” and “m” are parameters and are set using heuristics.

Further, by limiting the comparison data to data in a normal state, it is possible to improve the abnormality detection accuracy of the abnormality determination unit 124 described later. The definition of “normal state data” is as described above. In addition, when only normal data is used as comparison data, it should be noted that the divergence value vectors are spatially concentrated locally. For example, when the divergence value vector is calculated using the method described with reference to FIGS. 5 and 6, the divergence value vector is spatially concentrated near the origin. In addition, when the divergence value vector is defined as (m ₁ , m ₂ ,..., M _K ) with respect to the degree of membership “m _k ” shown in equation (2), the divergence value vector is spatially Concentrate on K clusters.

  Further, the detection data and the comparison data may be given separately. For example, the CPU usage rates of a plurality of servers for a specific past day reach the abnormality detection device 10 as comparison data, and the detection data sequentially arrives when the abnormality detection device 10 is operated. In such a case, it is not efficient in terms of calculation resources to recalculate the divergence value vector of the comparison data each time the detection target data arrives. Therefore, the divergence value vector storage unit 131 of the storage unit 13 stores the divergence value vector of the comparison data so that it is not necessary to recalculate the divergence value of the comparison data in such a case. When the deviation value vector storage unit 131 is used, the abnormality degree calculation unit 123 compares the deviation value vector of the detected data with the deviation value vector stored in the deviation value vector storage unit 131.

  Even when the divergence value vector storage unit 131 is used, the abnormality detection accuracy can be improved by storing only the divergence value vector calculated from the normal state data as the comparison data.

[Abnormality judgment unit processing]
The abnormality determination unit 124 determines that the detected data is abnormal when the discrete degree calculated by the abnormality degree calculation unit 123 exceeds a predetermined threshold. The abnormality determination unit 124 determines that the detection data is normal when the discrete degree is equal to or less than a predetermined threshold.

  Here, the threshold value used as the criterion for determination is set in advance. Alternatively, when there is test data, the degree of abnormality in specific data in the test data, that is, data when an abnormality occurs may be set as a threshold value. Alternatively, the degree of abnormality in the test data may be considered to follow an appropriate probability distribution, and a value such as the top 5% or the top 1% may be set as the threshold value. The determination result by the abnormality determination unit 124 is output to the terminal device 20 through the communication processing unit 11 as an abnormality detection result, for example.

[Flow of error detection processing]
Next, the abnormality detection process executed by the abnormality detection device 10 will be described. FIG. 7 is a flowchart showing a processing procedure of an abnormality detection process executed by the abnormality detection device 10.

  First, in the abnormality detection apparatus 10, the relationship estimation unit 121 performs a relationship estimation process for estimating the relationship between data with respect to input data and calculating a parameter indicating the relationship between data ( Step S1). The relationship estimation unit 121 uses data in which the device or the like that outputs the data is in a normal state, in other words, data that does not include data in an abnormal state for parameter estimation. When the relationship that holds between the data is given in advance, this step S1 can be omitted.

  Then, the divergence value vector calculation unit 122 executes divergence value vector calculation processing for calculating a divergence value vector between the data in the set of detection data to be detected and the set of comparison data based on the relationship between the data. (Step S2). Here, when the comparison data is given in advance, the divergence value vector calculation unit 122 calculates a divergence value vector between data in the set of comparison data, and stores it in the divergence value vector storage unit 131.

  Subsequently, the abnormality degree calculation unit 123 performs an abnormality degree calculation process for calculating the degree of discreteness of the deviation value vector of the detection data from the set of deviation value vectors of the comparison data as an abnormality degree (step S3). When the deviation value vector of the comparison data is calculated in advance and stored in the deviation value vector storage unit 131, the abnormality degree calculation unit 123 reads the deviation value vector of the comparison data from the deviation value vector storage unit 131, and A set of deviation value vectors of comparison data is acquired.

  And the abnormality determination part 124 performs the abnormality determination process which determines whether detection data are abnormal based on the discrete degree which the abnormality degree calculation part 123 calculated (step S4). In this case, the abnormality determination unit 124 determines that the detected data is abnormal when the discrete degree calculated by the abnormality degree calculation unit 123 exceeds a predetermined threshold. On the other hand, the abnormality determination unit 124 determines that the detection data is normal when the discrete degree is equal to or less than a predetermined threshold. The abnormality determination unit 124 outputs the determination result as an abnormality detection result to the terminal device 20 via the communication processing unit 11, and ends the abnormality detection process.

[Specific examples of abnormality detection processing]
FIG. 8 is a diagram for explaining the abnormality detection process of the first embodiment. FIG. 8 plots the set on the coordinate plane assuming that a set of X and Y is given as data. White circles in FIG. 8 correspond to comparison data in a normal state. Moreover, the point Pb is an example in the case of taking a value that did not exist until then while maintaining the correlation. The point Pr is an example when the correlation is broken. Further, based on normal comparison data (white circles in FIG. 8), a single regression “Y = aX + b” indicated by a straight line Lt is given as the relationship between X and Y.

  Since the point Pb shown in FIG. 8 is located on the straight line Lt and maintains the correlation that is established when it is normal, it is assumed to be normal. Here, in the conventional LOF, the relationship between data is not taken into consideration, and the points Pb and Pr existing at the point where the density of white circles is low are detected as abnormal.

  On the other hand, in the first embodiment, based on the relationship between the data, the divergence value vector between the data in the set of detection data and the divergence value vector between the data in the set of comparison data are calculated, Anomaly determination is performed with the degree of discreteness of the deviation value vector of the detection data from the set of deviation value vectors of the comparison data as the degree of abnormality.

  For example, a case where the point Pb is detection data is taken as an example. Since the point Pb is located on the straight line Lt, it can be said that “X, Y” indicated by the point Pb has a relationship of “Y = aX + b” indicated by the straight line Lt. Therefore, a divergence value vector with respect to “Y = aX + b” indicated by the straight line Lt is calculated for “X, Y” indicated by this point Pb, and the divergence value vector of normal comparison data (white circle) of the divergence value vector. When the degree of discreteness from the set is calculated, it becomes almost 0, and it can be detected that the point Pb is normal.

  On the other hand, a case where the point Pr is detection data will be described. Since this point Pr is away from the straight line Lt, it can be said that “X, Y” shown at the point Pr does not have a relationship of “Y = aX + b”. Therefore, the divergence value vector of “X, Y” indicated by the point Pr with respect to “Y = aX + b” indicated by the straight line Lt is calculated, and the divergence value vector of the normal comparison data (white circle) of the divergence value vector is calculated. When the degree of discreteness from the set is calculated, the value becomes large, and it can be detected that Pr is abnormal.

  As described above, the abnormality detection apparatus 10 introduces the concept of the divergence value vector, and is based on the spatial distance and density between the divergence value vector of the detection target data and the set of divergence value vectors of normal comparison data. The degree of discreteness (abnormality) is calculated, and the presence or absence of abnormality in the detected data is determined. Therefore, when there is a correlation between the data, the abnormality detection device 10 is normal for data that is on the correlation but can be assumed to be normal, for example, the point Pb that is out of the set of comparison data. Can be detected.

  Moreover, even when a complicated relationship that is not just a correlation between data is seen as shown in FIG. 14 and FIG. 15, anomaly detection is accurately performed by the concept of a deviation value vector from the relationship between data. Can perform well.

[Effect of Embodiment 1]
As described above, the first embodiment introduces the concept of divergence value vectors, and is a set of divergence value vectors of detected data and divergence value vectors of normal comparison data calculated based on the relationship between the data. In order to calculate the degree of anomaly (abnormality) based on the spatial distance and density of the data and determine the presence or absence of abnormality in the detected data, the abnormality detection of the detection target data based on the relationship between the data is accurately executed be able to.

[Embodiment 2]
Next, a second embodiment will be described. In the second embodiment, the Mahalanobis distance based on the deviation value vector of the detection data and the comparison data is calculated as the discrete degree, and the presence / absence of an abnormality is determined. Note that the abnormality detection device according to the second embodiment has a configuration equivalent to that of the abnormality detection device 10 shown in FIG.

  In the second embodiment, as in the first embodiment, the divergence value vector calculation unit 122 calculates a divergence value vector between data in a set of detection data to be detected based on the relationship between data. Then, the divergence value vector calculation unit 122 calculates a divergence value vector between data in the set of comparison data. As in the first embodiment, the divergence value vector calculation unit 122 calculates the divergence value vector between data in the set of comparison data when the comparison data is given in advance, and the divergence value vector storage unit 131 may be stored.

  Then, the degree-of-abnormality calculation unit 123 assumes that the divergence value vectors of the comparison data follow a multidimensional normal distribution, and calculates an average of these divergence value vectors and a covariance matrix. Subsequently, the abnormality degree calculation unit 123 calculates the Mahalanobis distance defined by the equation (3), and outputs the Mahalanobis distance as a discrete degree (abnormal degree).

The abnormality determination unit 124 determines that the detected data is abnormal when the Mahalanobis distance calculated by the abnormality degree calculation unit 123 exceeds a certain threshold. On the other hand, the abnormality determination unit 124 determines that the detected data is normal when the Mahalanobis distance calculated by the abnormality degree calculation unit 123 is equal to or less than a certain threshold value. In the second embodiment, it is assumed that the divergence value vector of the comparison data follows a multidimensional normal distribution. In this case, since the Mahalanobis distance of the divergence value vector approximately follows an x-square distribution, A threshold can be determined based on this. Such a method is referred to as T ² test of Hotelling (see "Kei Takeuchi, statistically dictionary P112, Toyo Keizai newspaper, 1989").

[Effect of Embodiment 2]
As described above, in the second embodiment, the Mahalanobis distance is calculated as the discrete degree, and the presence or absence of abnormality is determined based on the comparison result between the calculated Mahalanobis distance and the predetermined threshold value. Since the Mahalanobis distance is calculated based on the deviation value vector of the detection data and the comparison data based on the relationship between the data, the second embodiment is similar to the first embodiment in the relationship between the data. It is possible to accurately detect abnormality of detection target data based on the above.

[Embodiment 3]
Next, Embodiment 3 will be described. The abnormality detection device according to Embodiment 3 has a configuration equivalent to that of abnormality detection device 10 shown in FIG.

  Similarly to the first embodiment, the divergence value vector calculation unit 122 calculates a divergence value vector between data in a set of detection data to be detected based on the relationship between data. Then, the divergence value vector calculation unit 122 calculates a divergence value vector between data in the set of comparison data. As in the first embodiment, the divergence value vector calculation unit 122 calculates the divergence value vector between data in the set of comparison data when the comparison data is given in advance, and the divergence value vector storage unit 131 may be stored. Then, next, the process of the abnormality degree calculation part 123 is demonstrated.

[Processing of abnormality level calculation part]
In the third embodiment, the abnormality degree calculation unit 123 is further abbreviated as “One-class Support Vector Machine” (hereinafter, “One-class SVM”. For details, refer to “B. Scholkopf, JC Platt, J. Shawe-Taylor, AJ” Smola, and RC Williamson, “Estimating the Support of a High-Dimensional Distribution”, Neural Computation, 13 (7): 1443-1471, 2001.)) The region including is estimated.

  Specifically, with reference to FIG. 9, a process for obtaining the degree of discreteness (abnormality) will be described. FIG. 9 is a diagram for explaining abnormality degree calculation processing according to the third embodiment. FIG. 9 is a map of data divergence value vectors in a predetermined high-dimensional space.

  The degree-of-abnormality calculation unit 123 maps the deviation value vector of the comparison data, which is normal data, to a high-dimensional space (dimensions 1 and 2 in φ in FIG. 9) based on the One-class SVM. Then, the degree-of-abnormality calculation unit 123 obtains a plane (hyperplane) that maximizes the distance (margin) from the origin of the point corresponding to the divergence value vector of the mapped comparison data. This plane is shown as a hyperplane Le in the example of FIG. This hyperplane Le corresponds to the boundary of the set of divergence value vectors of normal comparison data, and actually, the point indicating the divergence value vector of the mapped comparison data is the origin side of the hyperplane Le. Located on the side that is not.

  Subsequently, the degree-of-abnormality calculation unit 123 maps the deviation value vector of the detection target data to the same high-dimensional space as the high-dimensional space mapped to the comparison data. For example, as shown in FIG. 9, each point indicating the deviation value vector of the mapped detection data is divided into a group R2 on the origin side and a group R3 not on the origin side when viewed from the hyperplane Le. The degree of abnormality calculation unit 123 calculates the degree of discreteness (degree of abnormality) based on whether or not the point corresponding to the divergence value vector of the mapped detection data is on the origin side when viewed from the hyperplane Le.

Therefore, the calculation process in the abnormality degree calculation unit 123 will be described with reference to FIG. FIG. 10 is a diagram for explaining the abnormality degree calculation processing according to the third embodiment. First, let divergence value vectors of comparison data be “e ₁ , e ₂ ,..., E _M ”. The minimization problem of minimizing the equation G shown in FIG. 10 (see equation (A)) under the conditions shown in equations (B) and (C) with respect to the deviation value vector of the comparison data is solved. The symbol “<,>” represents an inner product.

In this problem, an attempt is made to obtain a hyperplane that maximizes the smallest d _m when the distance (“φ (vector e _m ))” of each data is “d _m ”. The hyperplane parameters “vector w” and “ρ” that maximize the distance to the data closest to the hyperplane are obtained.

  This minimization problem can be solved by Lagrange's undetermined multiplier method for minimizing “L” shown in the following equation (4). The first line of the expression (4) is G itself, and the second line of the expression (4) reflects the constraints shown in the expression (B) of FIG. 10, and the third line of the expression (4). For, the constraints shown in the equation (C) of FIG. 10 are reflected.

FIG. 11 is a diagram for explaining an abnormality degree calculation process and an abnormality determination process according to the third embodiment. When the deviation value vector of abnormal data is “e ′”, the abnormality degree calculation unit 123 calculates the degree of abnormality with respect to the deviation value vector “e ′” of the detected data using the equation f (e ′) shown in FIG. . Expression f (e ′) applies the parameter obtained by Expression (4), and maps the divergence value vector “e _m ” of the comparison data to the high-dimensional space and the divergence value vector “e ′” of the detection data. The degree of anomaly is calculated based on the distance from a point mapped to a high-dimensional space. The degree-of-abnormality calculation unit 123 performs the calculation using the formula f (e ′), thereby determining whether or not the point corresponding to the divergence value vector of the mapped detection data is on the origin side when viewed from the hyperplane Le. Can be obtained.

  As described above, the degree-of-abnormality calculation unit 123 determines whether the point indicating the divergence value vector of the mapped detection data is on the origin side when viewed from the plane (for example, the hyperplane Le) according to the above-described One-class SVM, or Whether it is not on the origin side when viewed from the plane is calculated using the equation f (e ′).

[Abnormality judgment unit processing]
In the third embodiment, the abnormality determination unit 124 determines that the detection data is abnormal when the point indicating the deviation value vector of the mapped detection data is on the origin side when viewed from the plane. On the other hand, the abnormality determination unit 124 determines that the point indicating the divergence value vector of the mapped detection data is normal when it is not on the origin side when viewed from the plane. For example, the abnormality determination unit 124 determines that the detection data is abnormal for the group R2 on the origin side when viewed from the hyperplane Le among the points indicating the divergence value vector of the mapped detection data illustrated in FIG. judge. On the other hand, the abnormality determination unit 124 determines that the detection data is normal for the group R3 that is not on the origin side when viewed from the hyperplane Le (see the frame B1 in FIG. 9).

  Here, the abnormality determination unit 124 calculates the degree of abnormality obtained from the detected data by the equation (e ′) and the parameter (ρ tilde) corresponding to the hyperplane obtained by the undetermined multiplier method (equation (4)). Are compared to determine whether the detected data is abnormal. That is, as shown in FIG. 11, when the abnormality degree f (e ′) for the detection data is smaller than the parameter (ρ tilde) from the equation (4), the abnormality determination unit 124 determines that the detection data is hyperplane. The detection data is determined to be abnormal because it is determined to be closer to the origin than Le. On the other hand, the abnormality determination unit 124 determines that the detection data is not closer to the origin than the hyperplane Le when the abnormality degree f (e ′) for the detection data is greater than the parameter (ρ tilde). Determined to be normal.

[Effect of Embodiment 3]
As described above, in the third embodiment, the hyperplane where the distance from the origin is maximized is obtained by mapping the divergence value vector of the comparison data into the high-dimensional space. In the third embodiment, when the deviation value vector of the detection data is mapped to the high-dimensional space, the point corresponding to the mapped deviation value vector is on the origin side when viewed from the hyperplane. The degree of abnormality is calculated to determine whether there is an abnormality. That is, also in the third embodiment, as in the first embodiment, the distance between the divergence value vector of the detected data and the set of divergence value vectors of the normal comparison data calculated based on the relationship between the data. Therefore, the presence / absence of abnormality in the detected data is determined, so that the abnormality detection of the detection target data based on the relationship between the data can be accurately performed.

[System configuration of the embodiment]
Each component of the abnormality detection apparatus 10 illustrated in FIG. 1 is functionally conceptual, and does not necessarily need to be physically configured as illustrated. That is, the specific form of distribution and integration of the functions of the abnormality detection device 10 is not limited to the illustrated one, and all or a part thereof can be functionally or physically processed in arbitrary units according to various loads and usage conditions. Can be distributed or integrated.

  In addition, each or all of the processes performed in the abnormality detection apparatus 10 may be realized by a CPU (Central Processing Unit) and a program that is analyzed and executed by the CPU. Moreover, each process performed in the abnormality detection apparatus 10 may be implement | achieved as hardware by a wired logic.

  In addition, among the processes described in the embodiment, all or a part of the processes described as being automatically performed can be manually performed. Alternatively, all or part of the processing described as being performed manually can be automatically performed by a known method. In addition, the above-described and illustrated processing procedures, control procedures, specific names, and information including various data and parameters can be changed as appropriate unless otherwise specified.

[program]
FIG. 12 is a diagram illustrating an example of a computer in which the abnormality detection apparatus 10 is realized by executing a program. The computer 1000 includes a memory 1010 and a CPU 1020, for example. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to the display 1130, for example.

  The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the abnormality detection apparatus 10 is implemented as a program module 1093 in which a code executable by the computer 1000 is described. The program module 1093 is stored in the hard disk drive 1090, for example. For example, a program module 1093 for executing processing similar to the functional configuration in the abnormality detection device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

  The setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 and executes them as necessary.

  The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN, WAN, etc.). Then, the program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

  Although the embodiment to which the invention made by the present inventor is applied has been described above, the present invention is not limited by the description and the drawings that form part of the disclosure of the present invention according to this embodiment. That is, other embodiments, examples, operation techniques, and the like made by those skilled in the art based on the present embodiment are all included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Abnormality detection apparatus 11 Communication processing part 12 Control part 13 Storage part 121 Relationship estimation part 122 Deviation value vector calculation part 123 Abnormality degree calculation part 124 Abnormality determination part 131 Deviation value vector storage part

Claims (8)

Based on the relationship between the data, it represents the divergence from the relationship between the data of the divergence value vector that represents the divergence from the relationship between the data of the detection data that is the detection target and the comparison data that is the comparison target. A divergence value vector, a divergence value vector calculation unit for calculating,
An abnormality degree calculation unit for calculating a discrete degree of the deviation value vector of the detection data from the set of deviation value vectors of the comparison data as a degree indicating abnormality;
An abnormality determination unit that determines that the detection data is abnormal when the discrete degree exceeds a predetermined threshold;
An abnormality detection device characterized by comprising: A deviation value vector storage unit for storing the deviation value vector calculated by the deviation value vector calculation unit;
The abnormality detection device according to claim 1, wherein the abnormality degree calculation unit calculates the discrete degree using a deviation value vector stored in the deviation value vector storage unit. Further comprising a relationship estimation unit for estimating a relationship between the data and calculating a parameter indicating the relationship between the data;
The divergence value vector calculation unit applies the parameter indicating the relationship between the data calculated by the relationship estimation unit to the relationship between the data, and calculates the divergence value vector. The abnormality detection device according to 1 or 2.   The anomaly degree calculation unit calculates the discrete degree based on a spatial distance, a density, and the like of a deviation value vector of the detection data from a set of deviation value vectors of the comparison data. Item 4. The abnormality detection device according to any one of Items 1 to 3.   The degree of abnormality calculation unit calculates an average of the divergence value vector of the comparison data and a covariance matrix of the divergence value vector of the comparison data, and the divergence value vector of the detection data and the divergence value vector of the comparison data The Mahalanobis distance calculated | required based on the average of the above and the covariance matrix of the divergence value vector of the comparison data is calculated as the discrete degree. Anomaly detection device.   The anomaly calculation unit obtains a plane that maximizes the distance from the origin at a point corresponding to the mapped deviation value vector when the deviation value vector of the comparison data is mapped to a high-dimensional space, and When the deviation value vector is mapped to the high-dimensional space, the discrete degree is calculated based on whether or not a point corresponding to the mapped deviation value vector is on the origin side when viewed from the plane. The abnormality detection device according to any one of claims 1 to 3. An anomaly detection method executed by an anomaly detection device that detects the presence or absence of an anomaly with respect to a set of detection data to be detected,
Calculating a divergence value vector between data in the set of detection data and a divergence value vector between data in the set of comparison data to be compared based on the relationship between the data;
Calculating a discrete degree of the deviation value vector of the detection data from the set of deviation value vectors of the comparison data as a degree indicating abnormality;
Determining that the detection data is abnormal when the discrete degree exceeds a predetermined threshold;
An abnormality detection method characterized by including Calculating a divergence value vector between data in a set of detection data to be detected and a divergence value vector between data in a set of comparison data to be compared based on the relationship between the data;
Calculating a discrete degree of the deviation value vector of the detection data from a set distribution of the deviation value vector of the comparison data as a degree indicating abnormality;
Determining that the detection data is abnormal when the discrete degree exceeds a predetermined threshold;
An abnormality detection program that causes a computer to execute.

Images