JP6823501B2 - Anomaly detection device, anomaly detection method and program - Google Patents

Description

The present invention relates to an abnormality detection device, an abnormality detection method and a program.

In real-time abnormality detection, various data are regularly observed, and "abnormality" is detected when the data shows a tendency different from the normal state. Here, the anomaly detection algorithm learns by using the data of the "learning period" defined in advance as normal as the teacher data, and in the "test period" in which the abnormality is detected, the observed test data and the learning The tendency of the teacher data is compared. As such an abnormality detection algorithm, an algorithm has been proposed that learns the correlation of various data in the normal state and determines "abnormal" when the correlation of the learned data is broken during the test period. (For example, Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3).

Such an algorithm has an advantage that abnormality detection can be performed using only normal data without using abnormal data, which is difficult to determine whether or not it is abnormal.

Hodge, Victoria J., and Jim Austin. "A survey of outlier detection methodologies." Artificial intelligence review 22.2 (2004): 85-126. Mayu Sakurada, Takehisa Yairi, "Anomaly Detection of Spacecraft by Dimensionality Reduction Using Autoencoder", Proceedings of the Japanese Society for Artificial Intelligence National Convention 28, 1-3, 2014 Ringberg, Haakon, et al. "Sensitivity of PCA for traffic anomaly detection." ACM SIGMETRICS Performance Evaluation Review 35.1 (2007): 109-120.

However, when a large amount of data with low correlation is included in the input, the pattern of the state in which the data can be obtained at normal times also increases in combination, so that the teacher data required for learning increases and there is not enough teacher data. Accurate abnormality detection becomes difficult. In particular, when the types of data to be observed increase, the number of data with low correlation increases, so that such a problem becomes more remarkable.

The present invention has been made in view of the above points, and an object of the present invention is to suppress an increase in data required for learning to detect an abnormality.

Therefore, in order to solve the above problem, the abnormality detection device determines the correlation between the data elements of a plurality of types of data obtained from the detection target when the abnormality detection target is normal, and the correlation between the data elements. For each unit classified based on height, a learning unit that learns using a plurality of learning devices generated for the unit and outputs the learning result, and a plurality of types obtained from the detection target at a plurality of timings. For each of the data element groups of the data of, based on the learning result related to the unit, the degree of anomaly indicating the degree of collapse of the correlation of the data element group classified into the unit is calculated, and for each of the units. It has a detection unit that detects an abnormality of the detection target based on the degree of abnormality of.

It is possible to suppress an increase in data required for learning to detect an abnormality.

It is a figure which shows the system configuration example in 1st Embodiment. It is a figure which shows the hardware configuration example of the abnormality detection apparatus 10 in 1st Embodiment. It is a figure which shows the functional configuration example of the abnormality detection apparatus 10 in the 1st Embodiment. It is a flowchart for demonstrating an example of the processing procedure of the learning process in 1st Embodiment. It is a flowchart for demonstrating an example of the processing procedure of the detection process in 1st Embodiment. It is a figure for demonstrating an autoencoder. 6 is a flowchart for explaining a processing procedure additionally executed by the preprocessing unit 13 in the sixth embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a system configuration according to the first embodiment. In FIG. 1, the network N1 is a network for which an abnormality is detected. The network N1 is configured by connecting a plurality of nodes such as routers and server devices to each other, and packets are transmitted and received between arbitrary nodes in order to provide a predetermined service.

Measuring devices 20 are arranged at a plurality of locations in the network N1. The measuring device 20 collects observation data obtained by monitoring the arrangement location at a plurality of timings. Examples of collected observation data include MIB (Management Information Base) data, NetFlow flow data, CPU usage rate, and the like.

MIB is a common policy among manufacturers for monitoring network devices. The MIB data is aggregated in units of 5 minutes, for example, and includes "time, host name, interface (IF) name, input data amount (ibps), output data amount (obps)", and the like.

NetFlow is a technology that monitors the network in units of flows, and information about the flow is output when communication is completed. A flow is a unit for grasping "where" and "where" are performing "what kind of communication" and "how much", and is an IP address (srcIP) on the sender side of communication. , Port number on the sender side (srcport), IP address on the receiver side (dstIP), port number on the receiver side (dstport), and communication protocol (proto). The flow data includes "flow start time, srcIP, srcport, dstIP, dstport, proto, flow duration, total number of transmitted packets, total number of transmitted bytes" and the like.

The CPU usage rate is, for example, the usage rate of a CPU such as a server device or a router included in the network N1.

The observation data collected by the measuring device 20 is collected by the abnormality detecting device 10. The abnormality detection device 10 learns the characteristics at the normal time from the collected observation data, and detects the occurrence of an abnormality in the observation data input thereafter based on the learning result (determines the presence or absence of the abnormality). It is a computer. The process in which the characteristics in the normal state are learned is called "learning process". The process in which an abnormality is detected based on the result learned in the learning process is called "test process".

FIG. 2 is a diagram showing a hardware configuration example of the abnormality detection device 10 according to the first embodiment. The abnormality detection device 10 of FIG. 2 has a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, and the like, which are connected to each other by a bus B, respectively.

The program that realizes the processing in the abnormality detection device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed in the auxiliary storage device 102 from the recording medium 101 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, and may be downloaded from another computer via the network. The auxiliary storage device 102 stores the installed program and also stores necessary files, data, and the like.

The memory device 103 reads and stores the program from the auxiliary storage device 102 when the program is instructed to start. The CPU 104 executes the function related to the abnormality detection device 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

FIG. 3 is a diagram showing a functional configuration example of the abnormality detection device 10 according to the first embodiment. In FIG. 3, the abnormality detection device 10 includes a receiving unit 11, a learning processing control unit 12, a preprocessing unit 13, a learning unit 14, a detection processing control unit 15, a detection unit 16, and the like. Each of these parts is realized by a process of causing the CPU 104 to execute one or more programs installed in the abnormality detection device 10. The abnormality detection device 10 also uses the teacher data storage unit 121, the parameter storage unit 122, the observation data storage unit 123, the learning result storage unit 124, the learning data storage unit 125, and the like. Each of these storage units can be realized by using, for example, a storage device that can be connected to the auxiliary storage device 102 or the abnormality detection device 10 via a network.

The teacher data storage unit 121 stores observation data that has been confirmed to have been collected in the normal state in advance as teacher data. However, the teacher data may be artificially created instead of being selected from the observation data.

The receiving unit 11 receives the observation data from the measuring device 20. The received observation data is stored in the observation data storage unit 123. The learning processing control unit 12 controls the learning process.

The pre-processing unit 13 executes pre-processing on a set of teacher data, a set of observation data, or a set of learning data stored in the learning data storage unit 125. The pre-processing is processing such as extraction of feature quantities for each unit time from a data set and normalization of the extracted feature quantities. Features are expressed in the form of numerical vectors. At the time of the first learning, the teacher data group stored in the teacher data storage unit 121 is targeted for preprocessing. When the reception unit 11 starts receiving the observation data, the observation data group is subject to preprocessing. Further, when the detection unit 16 starts detecting an abnormality, determines that it is normal, and reaches a predetermined number of observation data stored in the learning data storage unit 125 as learning data, the learning data group is subject to preprocessing. It is said that.

The preprocessing unit 13 also generates or generates a parameter for normalizing the observation data or the training data (hereinafter, referred to as “normalization parameter”) when executing the preprocessing for the teacher data group or the training data group. The updated and generated or updated normalized parameters are stored in the parameter storage unit 122.

The learning unit 14 executes learning based on the teacher data or the learning data. The learning result by the learning unit 14 is stored in the learning result storage unit 124.

The detection process control unit 15 controls the detection process.

The detection unit 16 uses a numerical vector generated by preprocessing the observation data stored in the observation data storage unit 123 by the preprocessing unit 13 and a learning result stored in the learning result storage unit 124. Detect the occurrence of anomalies based on. Specifically, the detection unit 16 calculates the difference from the learning result of the preprocessed numerical vector as the degree of abnormality, and detects the occurrence of the abnormality by comparing the degree of abnormality with the threshold value. The value before normalization of the numerical vector in which no abnormality is detected is stored in the learning data storage unit 125 as learning data.

Hereinafter, the processing procedure executed by the abnormality detection device 10 will be described. FIG. 4 is a flowchart for explaining an example of the processing procedure of the learning process according to the first embodiment. In the following, for convenience, an example in which the flow data is the processing target will be shown.

When the learning process is started, the learning process control unit 12 acquires the teacher data group from the teacher data storage unit 121 and inputs the teacher data group to the preprocessing unit 13 (S101).

Subsequently, the preprocessing unit 13 divides the input teacher data group into a set for each unit time (S102). It is assumed that the teacher data storage unit 121 stores teacher data for a period of unit time × U (hereinafter, referred to as “learning period”). Therefore, the teacher data group is divided into U sets.

Subsequently, the preprocessing unit 13 extracts a feature amount according to a purpose for each divided set, and generates a multidimensional numerical vector having the extracted feature amount as an element of each dimension (S103).

For example, suppose that the unit time is 1 minute and the preprocessing unit 13 extracts the feature amount for each minute. Further, it is assumed that the feature amount is the total number of transmitted bytes of each protocol (TCP, UDP). In this case, assuming that the flow start time of the first teacher data is 12:00:00, the preprocessing unit 13 has the flow start time t of 11:59:00 <= t <12: of all the teacher data. For a set of teacher data (flow data) such as 00:00, the total number of transmitted bytes of all flows whose protocol is TCP, the total number of transmitted bytes of all flows whose protocol is UDP, etc. are calculated, and their characteristics. Generate a two-dimensional numerical vector with a quantity as an element of each dimension. Similarly, numerical vectors are generated for (U-1) other sets.

The feature amount attribute can also be specified as a combination such as "TCP and transmission port number is 80". Further, if it is considered that each flow has a value such as "number of flows: 1", the total number of flows of the flows having each attribute can be calculated in the same manner and regarded as a feature amount.

Subsequently, the preprocessing unit 13 calculates the maximum value xmax_i of each metric i (each dimension i) in each numerical vector, and stores the calculated xmax_1 in the parameter storage unit 122 (S104). That is, in the first embodiment, the maximum value xmax_i of each metric i is a normalization parameter.

Here, U = 3. Further, it is assumed that the numerical vectors generated in step S103 are {{80,20}, {90,35}, {100,50}}. This is because the total number of transmitted bytes of TCP and the total number of transmitted bytes of UDP in a certain 3 minutes are "TCP: 80 bytes, UDP: 20 bytes", "TCP: 90 bytes, UDP: 35 bytes", "TCP: 100 bytes, UDP:", respectively. It shows that it was "50 bytes". In this case, the maximum value xmax_i of each metric of these numerical vectors is {100,50} (that is, xmax_1 = 100, xmax_2 = 50).

Subsequently, the preprocessing unit 13 normalizes each numerical vector based on the normalization parameter (S105). Normalization is performed by dividing the value of the metric i of each numerical vector by the maximum value xmax_i. Therefore, the normalized numerical vectors are {{0.8, 0.4}, {0.9, 0.7}, {1,1}}.

Subsequently, the learning unit 14 learns the numerical vector using the learner (S106). The learning result is stored in the learning result storage unit 124.

Subsequently, the learning processing control unit 12 waits for the learning data storage unit 125 to store (accumulate) the learning data for the learning period (S107). That is, the standby continues until U unnormalized numerical vectors are stored in the learning data storage unit 125. The learning data storage unit 125 stores a numerical vector determined to be normal (no abnormality has occurred) by the detection unit 16.

When the numerical vectors for the learning period are stored in the learning data storage unit 125 (Yes in S107), the learning processing control unit 12 acquires the numerical vector group from the learning data storage unit 125 and preprocesses the numerical vector group. Input to unit 13 (S108). The acquired numerical vector group is deleted from the learning data storage unit 125. Subsequently, steps S104 and subsequent steps are executed for the numerical vector group. Therefore, in the next step S105, normalization is performed based on the newly calculated xmax_i.

FIG. 5 is a flowchart for explaining an example of the processing procedure of the detection process according to the first embodiment. The processing procedure of FIG. 5 may be started at any time after step S106 of FIG. 4 has been executed at least once. That is, the processing procedure of FIG. 5 is executed in parallel with the processing procedure of FIG.

In step S201, the detection processing control unit 15 waits for the elapse of the unit time. The unit time is the same time length as the unit time in the description of FIG. During this standby, the observation data collected in real time and received by the receiving unit 11 is stored in the observation data storage unit 123. When the unit time elapses (Yes in S201), the detection processing control unit 15 has the latest unit. The observation data group for the time is acquired from the observation data storage unit 123, and the observation data group is input to the preprocessing unit 13 (S202).

Subsequently, the preprocessing unit 13 extracts a feature amount according to the purpose from the observation data group, and generates a multidimensional numerical vector having the extracted feature amount as an element of each dimension (S203). For example, the total number of transmitted bytes of all flows whose protocol is TCP and the total number of transmitted bytes of all flows whose protocol is UDP are extracted, and a two-dimensional numerical vector having these as elements of each dimension is generated. Here, one numerical vector is generated.

Subsequently, the preprocessing unit 13 normalizes the generated numerical vector based on the maximum value xmax_i stored in the parameter storage unit 122 (S204). That is, each metric i of the numerical vector is divided by the maximum value xmax_i.

For example, when step S104 of FIG. 4 is executed only once based on the above teacher data, the maximum value xmax_i is {100,50}. Therefore, when the numerical vector is {60,40}, the numerical vector is normalized to {0.6,0.8}.

Subsequently, the detection unit 16 executes the abnormality determination process (S205). In the abnormality determination process, the presence or absence of an abnormality in the network N1 is determined based on the normalized numerical vector and the latest learning result stored in the learning result storage unit 124.

When it is determined that there is no abnormality (Yes in S206), the detection processing control unit 15 stores the numerical vector before normalization of the numerical vector in the learning data storage unit 125 as learning data (S207). When it is determined that there is an abnormality (No in S206), the numerical vector before normalization of the numerical vector is not stored in the learning data storage unit 125. Therefore, only the numerical vector at the normal time is stored in the learning data storage unit 125.

Subsequently, steps S201 and subsequent steps are repeated. In the process of repeating step S201 and subsequent steps, the normalization parameters used in step S204 are updated at any time in step S104 of FIG. 4 which is executed in parallel. As a result, the numerical vector can be normalized in consideration of the trend of the input observation data.

For example, when U = 3, step S207 is executed three times, and {{60,40}, {45,20}, {30,30}} are stored in the learning data storage unit 125. In this case, it is updated to xmax_1 = 60 and xmax_2 = 40, and the update result is reflected in the parameter storage unit 122.

Although the example in which the observation data is the flow data has been described above, the flow data, the MIB data, and the CPU usage rate may be received as the observation data in parallel. In this case, each step of the processing procedure of FIGS. 4 and 5 may be executed for each data type (flow data, MIB data, and CPU usage rate).

For MIB data given in a format such as {hostID, interfaceID, ibps, obps}, "ibps of host IDa in unit time", "obps of host IDa in unit time", and "host IDb in unit time". "Ibps", "Obps of host IDb in unit time" ... "Ibps of interfaceIDx in unit time", "Obps of interfaceIDx in unit time", "IBps of interfaceIDy in unit time", "obps of interfaceIDy in unit time" It is possible to extract a numerical vector as in.

Subsequently, an example of step S106 of FIG. 4 and step S205 of FIG. 5 will be described. In steps S106 and S205, the numerical vector group to which the data type is assigned as a label is input to the learning unit 14 or the detecting unit 16. In this embodiment, the label is one of "flow data", "MIB data", and "CPU usage". The label is attached to the teacher data and the observation data by, for example, the measuring device 20 or the receiving unit 11. That is, the label to be attached to the observation data can be specified based on the source of the observation data. The label is taken over by the numerical vector generated by the preprocessing unit 13.

In step S106 of FIG. 4, the learning unit 14 generates a learning device for each data type. The learning unit 14 classifies the numerical vector based on the label given to the input numerical vector, and inputs the numerical vector to the learning device corresponding to the classification result. In the present embodiment, a "flow data learner", a "MIB data learner", and a "CPU usage rate learner" are generated. As the learner, an autoencoder (Non-Patent Document 2) or a principal component analysis (Non-Patent Document 3) that detects anomalies by learning the correlation between numerical vector metrics can be used. In this embodiment, an example in which an autoencoder is used as the learning device will be described.

FIG. 6 is a diagram for explaining an autoencoder. The autoencoder is an anomaly detection algorithm based on deep learning. The autoencoder utilizes the fact that normal input data has a correlation between metrics and can be compressed to a low dimension. In the event of an abnormality, the correlation between the input data is broken, so compression is not performed correctly and the difference between the input data and the output data becomes large.

As shown in (1) of FIG. 6, the learner (autoencoder) generated by the learning unit 14 learns so that the output layer (Layer L ₃ ) is close to the input layer (Layer L ₁ ). Specifically, the learning unit 14 duplicates the numerical vector into two, applies one to the input layer, applies the other to the output layer, performs learning, and outputs the learning result. The learning result is stored in the learning result storage unit 124. The learning result is a group of parameters for the learning device. Since the learning device is generated for each data type, the learning result is also output for each data type and stored in the learning result storage unit 124.

On the other hand, the detection unit 16 also generates a learning device for each data type, similarly to the learning unit 14. As the learning device, a method corresponding to the learning device generated by the learning unit 14 among the autoencoder, the principal component analysis, and the like can be used as in the learning device generated by the learning unit 14.

In step S205 of FIG. 5, the detection unit 16 learns the “flow data learner”, the “MIB data learner”, and the “CPU usage rate” based on the learning result stored in the learning result storage unit 124. Generate a vessel. That is, the learning device generated by the detection unit 16 is the same as the learning device generated by the learning unit 14 at the time of outputting the learning result. As shown in (2) of FIG. 6, the detection unit 16 inputs the numerical vector for each data type input in step S205 to the learning device corresponding to the data type of the numerical vector, and the input data to the learning device. The distance between the output data and the output data (an index showing the degree of collapse of the correlation between the metrics) is calculated as the degree of abnormality. In the present embodiment, the mean squared error (MSE: Mean Squared Error), which is the distance between the input layer and the output layer of the autoencoder, is calculated as the degree of abnormality. The calculation formula of MSE is as follows.

本実施の形態では、フローデータのＭＳＥ、ＭＩＢデータのＭＳＥ、ＣＰＵ使用率のＭＳＥの３種のＭＳＥが得られる。検知部１６は、得られたＭＳＥの平均を、最終的な異常度として計算し、最終的な異常度が予め定められた閾値を超えていた場合に異常であると判定する。そうでない場合、検知部１６は、正常とであると判定する。

In the present embodiment, three types of MSEs are obtained: flow data MSE, MIB data MSE, and CPU usage rate MSE. The detection unit 16 calculates the average of the obtained MSE as the final degree of abnormality, and determines that the degree of abnormality is abnormal when the final degree of abnormality exceeds a predetermined threshold value. If not, the detection unit 16 determines that it is normal.

As described above, according to the first embodiment, a learning device is generated for each type of data, and learning and abnormality detection are performed. Here, it is presumed that the metrics (data elements) belonging to the same data type have a high correlation. Therefore, it is possible to reduce the possibility that data with low correlation is input to the same learner. As a result, it is possible to suppress an increase in data required for learning to detect an abnormality.

Next, the second embodiment will be described. The second embodiment will explain the differences from the first embodiment. The points not particularly mentioned in the second embodiment may be the same as those in the first embodiment.

In the second embodiment, the detection unit 16 calculates the weighted average of the abnormalities output from each learning device as the final abnormalities. At this time, the average value of MSE when the numerical vector group based on the teacher data or the learning data is input to the learner is used as the weight.

Therefore, in the second embodiment, each time the learning unit 14 executes step S106 of FIG. 4, the learning result output from the learning device for each data type based on the teacher data or the numerical vector group of the learning data is obtained. When storing in the learning result storage unit 124, for each data type, the numerical vector for each data type in which each numerical vector is input to the learner based on the learning result is input. By doing so, the learning unit 14 calculates the degree of abnormality for each data type and for each numerical vector, and further calculates the average degree of abnormality for each data type. For example, if U = 3, three abnormalities are calculated for each data type, and the average of the abnormalities is calculated for each data type. The average degree of abnormality for each data type is stored in the learning result storage unit 124 together with the learning result. Therefore, "MSE average of flow data", "MSE average of MIB data", and "MSE average of CPU usage rate" are stored. Hereinafter, each will be referred to as β _ {train, 1}, β _ {train, 2}, β _ {train, 3}.

In the detection process, when calculating the average of MSE obtained by inputting the numerical vector for each data type based on the observation data to each learner, the larger the average of MSE based on the teacher data or learning data, the larger the data type. It is possible that the MSE based on the observation data will also increase. Therefore, the detection unit 16 calculates the weighted average for the degree of abnormality for each data type, using the average of the MSE based on the teacher data or the learning data stored in the learning result storage unit 124 as a weight.

Specifically, the MSE when the numerical vector based on the flow data, the MIB data, and the observation data of the CPU usage rate is input to the learner based on the learning result is β_ {test, 1}, β_ {test, 2, respectively. }, β_ {test, 3}, the detection unit 16 calculates the final degree of abnormality β based on the following formula.
β = (β_ {test, 1} / β'_ {train, 1} + β_ {test, 2} / β'_ {train, 2} + β_ {test, 3} / β'_ {train, 3}) / (1 / β'_ {train, 1} + 1 / β'_ {train, 2} + 1 / β'_ {train, 3})
This is because the weighting coefficient is the reciprocal of the average of MSE based on the teacher data or training data (1 / β'_ {train, i}), and the larger the MSE based on the teacher data or training data, the more the observation data. It is shown that the weight of MSE based on is reduced.

As described above, according to the second embodiment, the final degree of abnormality can be calculated in the detection process in consideration of the difference in the magnitude of the degree of abnormality between the data types in the normal state.

Next, a third embodiment will be described. A difference from the first embodiment will be described in the third embodiment. The points not particularly mentioned in the third embodiment may be the same as those in the first embodiment.

In the third embodiment, the detection unit 16 determines whether or not there is an abnormality for each learner (that is, for each data type), and when it is determined that there is an abnormality for at least one of the data types, the final determination is made. Judgment result is "abnormal".

In the detection process, for each data type, the MSE obtained when a numerical vector based on the observation data is input to the learner related to the data type is β_ {test, 1}, β_ {test, 2}, β_ {test. , 3}. Here, the threshold value is assumed to be predetermined for each data type, and is expressed as θ_1, θ_2, and θ_3, respectively. In this case, the detection unit 16 determines that there is an abnormality when β_ {test, i} ≧ θ_i for each learning device i, and that there is no abnormality when it is not. In the present embodiment, the presence or absence of an abnormality is determined for each of the three types of learners, "flow data", "MIB data", and "CPU usage rate", and at least one of them is determined to be "abnormal". If this is the case, the final judgment of the presence or absence of an abnormality is determined to be "abnormal", and if not, "no abnormality" is determined.

As described above, according to the third embodiment, the same effect as that of the first embodiment can be obtained.

Next, a fourth embodiment will be described. The fourth embodiment will be described as different from the third embodiment. The points not particularly mentioned in the fourth embodiment may be the same as those in the third embodiment.

In the fourth embodiment, the detection unit 16 determines the presence or absence of an abnormality for each learning device of each data type, and then makes a final determination only when it is determined that all the learning devices have an abnormality. The result is "abnormal". For example, the final judgment result is "abnormal" only when it is judged as "abnormal" for all three types of learners of "flow data", "MIB data", and "CPU usage rate". Otherwise, the final judgment result will be "no abnormality".

As described above, according to the fourth embodiment, the same effect as that of the first embodiment can be obtained.

Next, a fifth embodiment will be described. The fifth embodiment will explain the differences from the third embodiment. The points not particularly mentioned in the fifth embodiment may be the same as those in the third embodiment.

In the fifth embodiment, the detection unit 16 determines the presence or absence of an abnormality for each learning device of each data type, and then determines the number of learning devices that are determined to be "abnormal" and the learning that is determined to be "no abnormality". The final determination of the presence or absence of an abnormality is made by a majority vote with the number of vessels. For example, if two or more of the three types of learners, "flow data", "MIB data", and "CPU usage rate", are judged to be "abnormal", the final judgment result is "abnormal". "Yes", otherwise the final judgment result is "No abnormality". When the number of learners is even, the handling when the number of "abnormal" and the number of "no abnormality" match is "abnormal", "no abnormality", or randomly. Whether to decide or not is decided in advance.

As described above, according to the fifth embodiment, the same effect as that of the first embodiment can be obtained.

Next, the sixth embodiment will be described. In the sixth embodiment, the points different from each of the above embodiments will be described. The points not particularly mentioned in the sixth embodiment may be the same as those in the above-described embodiments.

In the sixth embodiment, an example in which a learner is generated for each cluster based on the correlation between the metrics of the numerical vector for each data type, not for each data type, will be described. That is, in each of the above embodiments, the data type has a high correlation between data elements (metrics) based on the estimation that each data element (metric) belonging to the same data type will have a high correlation. It was used as a unit to be classified based on. On the other hand, in the sixth embodiment, the data element group is a plurality of sets (the following clusters) based on the height of the correlation between each data element (each metric), not based on such estimation. ), And the set is a unit to be classified based on the height of correlation between data elements (between metrics).

First, in the sixth embodiment, in step S103 of FIG. 4 and step S203 of FIG. 5, one numerical vector x is generated for each unit time, not for each data type. For example, one numerical vector x including each metric of the numerical vector of the flow data, each metric of the numerical vector of the MIB data, and each metric of the CPU usage rate is generated. The numerical vector x in the unit time t is expressed as x_ {i, t} (i = 1, ..., N, t = 1, ..., U).

Further, the preprocessing unit 13 executes the processing procedure shown in FIG. 7 following step S103 in FIG.

FIG. 7 is a flowchart for explaining a processing procedure additionally executed by the preprocessing unit 13 in the sixth embodiment.

In step S301, the preprocessing unit 13 assigns an independent ID to each metric of the numerical vector x.

Subsequently, the preprocessing unit 13 calculates the Pearson correlation coefficient for each set of the two metrics (S302). That is, the correlation coefficients α_ {i, j} between the metrics i and j are (x_ {i, 1}, ..., x_ {i, T}) and (x_ {j, 1} ,. It is calculated by the Pearson correlation coefficient with., {J, U}) (i = 1, ..., N, j = 1, ..., N, i <j).

Subsequently, the preprocessing unit 13 clusters the IDs of each metric into a predetermined number of groups K using a multidimensional scaling method based on the Pearson correlation coefficient α_ {i, j} (S303). Subsequently, the preprocessing unit 13 stores the correspondence information between the ID and the cluster, which indicates which cluster each ID is classified into, in the learning result storage unit 124 (S304).

The processing procedure of FIG. 7 may be executed once. That is, it does not have to be executed following step S108 of FIG.

In other cases, the data type in each of the above embodiments may be replaced with a cluster. For example, in step S106 of FIG. 4, the learning unit 14 generates a learning device for each cluster based on the correspondence information stored in the learning result storage unit 124, and performs learning. Among the normalized numerical vectors, the metric corresponding to the ID classified into the cluster to which the learner corresponds is input to each learner. The learning result is stored in the learning result storage unit 124 for each cluster.

Further, in step S205 of FIG. 5, the detection unit 16 generates a learning device for each cluster based on the learning result for each cluster stored in the learning result storage unit 124. The detection unit 16 inputs to each learner a metric corresponding to an ID classified into the cluster to which the learner corresponds among the normalized numerical vectors. In the detection process (FIG. 5), for example, following step S203, the preprocessing unit 13 may assign an independent ID to each of the numerical vectors, as in step S301 of FIG.

As described above, according to the sixth embodiment, the learner can be generated for each metric group having a higher correlation.

It should be noted that each of the above embodiments may be applied to data collected from other than the network. For example, each of the above embodiments may be applied to data collected from a computer system.

In each of the above embodiments, the pretreatment unit 13 is an example of a classification unit.

Although the examples of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications are made within the scope of the gist of the present invention described in the claims.・ Can be changed.

10 Anomaly detection device 11 Receiving unit 12 Learning processing control unit 13 Preprocessing unit 14 Learning unit 15 Detection processing control unit 16 Detection unit 20 Measuring device 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 CPU
105 Interface device 121 Teacher data storage unit 122 Parameter storage unit 123 Observation data storage unit 124 Learning result storage unit 125 Learning data storage unit B bus N1 network

Claims (8)

When the abnormality detection target is normal, the correlation between the data elements of a plurality of types of data obtained from the detection target is classified for each unit classified based on the height of the correlation between the data elements. A learning unit that learns using multiple learning devices generated for each unit and outputs the learning results,
Regarding the data element group of a plurality of types of data obtained from the detection target at a plurality of timings, the correlation of the data element group classified into the unit is broken for each unit based on the learning result related to the unit. A detection unit that calculates the degree of abnormality indicating the degree of the above and detects the abnormality of the detection target based on the degree of abnormality for each unit.
Have a,
The data elements that make up each of the units are different from each other.
Anomaly detection device characterized by this. The unit is a unit for each type.
The abnormality detection device according to claim 1, wherein the abnormality detection device is characterized. It has a classification unit that classifies data element groups of a plurality of types of data obtained from the detection target when the abnormality detection target is normal into a plurality of sets based on the height of correlation.
The unit is a unit for each set.
The abnormality detection device according to claim 1, wherein the abnormality detection device is characterized. The detector is
The average degree of anomaly for each unit,
Alternatively, a weighted average of the anomaly degree for each unit, which weights the anomaly degree calculated based on the learning result for each unit of the data element group when the abnormality detection target is normal.
Or, the degree of abnormality of at least one of the above units exceeds the threshold value.
Or, whether the degree of abnormality related to all the units exceeds the threshold value
Or, the number of the units whose degree of abnormality exceeds the threshold value,
Detects the abnormality of the detection target based on
The abnormality detection device according to any one of claims 1 to 3, wherein the abnormality detection device is characterized. When the abnormality detection target is normal, the correlation between the data elements of a plurality of types of data obtained from the detection target is classified for each unit classified based on the height of the correlation between the data elements. A learning procedure that learns using multiple learning devices generated for each unit and outputs the learning results,
Regarding the data element group of a plurality of types of data obtained from the detection target at a plurality of timings, the correlation of the data element group classified into the unit is broken for each unit based on the learning result related to the unit. A detection procedure that calculates the degree of abnormality indicating the degree of the above and detects the abnormality of the detection target based on the degree of abnormality for each unit.
The computer runs ,
The data elements that make up each of the units are different from each other.
Anomaly detection method characterized by this. The unit is a unit for each type.
The abnormality detection method according to claim 5, wherein the abnormality is detected. It has a classification procedure for classifying data elements of a plurality of types of data obtained from the detection target when the abnormality detection target is normal into a plurality of sets based on the height of correlation.
The unit is a unit for each set.
The abnormality detection method according to claim 5, wherein the abnormality is detected. A program for operating a computer as each part according to any one of claims 1 to 4.

Images