JP2018148350A - Threshold determination device, threshold level determination method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a threshold determination device capable of automatically determining a threshold level used for abnormality detection.SOLUTION: The threshold determination device includes: a detection part that calculates the abnormality degree based on a learning result, which is obtained by learning the data collected from the abnormality detection targets, and the data, which is observed from the abnormality detection targets in a test period, to detect an occurrence of abnormality by comparing the abnormality degree to the threshold level; and a threshold level determination part that determines the threshold level based on the existence of missed anomalies and erroneous detection. When any erroneous detection occurs, the threshold level determination part may increase the threshold level, and when there is any missed anomaly, the threshold level determination part may lower the threshold level.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a threshold determination device, a threshold determination method, and a program.

  As a method for detecting an abnormality in a computer system in real time, an algorithm that regularly observes various data on the system and detects the abnormality when the data shows a tendency different from the normal time can be considered.

  For example, in the “test period” in which data of “learning period” defined in advance as normal time is used as learning data and learned, and abnormality detection is performed, the tendency of the observed test data and the learned teacher data An anomaly detection algorithm that is compared with the trend is considered.

  Various methods have been proposed as anomaly detection algorithms. In many cases, the output of the algorithm is an anomaly degree defined by the distance from the normal state of the observed test data. For this reason, when performing abnormality detection, it is necessary to determine an abnormality threshold value for determining the presence or absence of abnormality (see, for example, Non-Patent Document 1 and Non-Patent Document 2).

Mitsutoshi Kawasaki, Daihiro Yoshikawa, Takeshi Furuhashi, "Examination of availability of reconstruction error in deep learning for the purpose of detecting anomalies in time series data", The 29th Annual Conference of the Japanese Society for Artificial Intelligence, 2015 Chandola, Varun, Arindam Banerjee, and Vipin Kumar. "Anomaly detection: A survey." ACM computing surveys (CSUR) 41.3 (2009): 15.

  When the threshold value used for abnormality detection is low, many abnormalities can be detected. On the other hand, false detections that determine “abnormal” when there is no actual abnormality increase. On the other hand, when the threshold value is set high, false detections are reduced. However, when the threshold value is actually abnormal, there is an increase in oversight that cannot be determined as “abnormal”. For this reason, it is necessary to appropriately determine the threshold value according to the purpose. However, setting the threshold value using a hand is a factor for increasing the operation of the operator, and a device for automatically determining the threshold value is required.

  The present invention has been made in view of the above points, and an object thereof is to automatically determine a threshold value used for abnormality detection.

  Therefore, in order to solve the above problem, the threshold value determination device calculates the degree of abnormality based on the learning result obtained by learning the data collected from the abnormality detection target and the data observed from the abnormality detection target during the test period. And a detection unit that detects the occurrence of an abnormality by comparing the degree of abnormality with a threshold value, and a threshold value determination unit that determines the threshold value based on the presence or absence of an abnormality and the presence or absence of erroneous detection.

  A threshold value used for abnormality detection can be automatically determined.

It is a figure which shows the system configuration example in 1st Embodiment. It is a figure which shows the hardware structural example of the abnormality detection apparatus 10 in 1st Embodiment. It is a figure which shows the function structural example of the abnormality detection apparatus 10 in 1st Embodiment. It is a flowchart for demonstrating an example of the process sequence of the learning process in 1st Embodiment. It is a flowchart for demonstrating an example of the process sequence of the detection process in 1st Embodiment. It is a figure for demonstrating an auto encoder. It is a flowchart for demonstrating an example of the process sequence of the threshold value determination process in 1st Embodiment. It is a flowchart for demonstrating an example of the process sequence of the threshold value determination process in 2nd Embodiment. It is a flowchart for demonstrating an example of the process sequence of the threshold value determination process in 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of a system configuration in the first embodiment. In FIG. 1, a network N1 is a network that is an abnormality detection target. The network N1 is configured by connecting a plurality of nodes such as routers and server devices to each other, and packets are transmitted and received between arbitrary nodes in order to provide a predetermined service.

  Measuring devices 20 are arranged at a plurality of locations in the network N1. The measuring device 20 collects observation data obtained by monitoring the arrangement location at a plurality of timings. Examples of collected observation data include MIB (Management Information Base) data, NetFlow flow data, CPU usage rate, and the like.

  MIB is a common policy among manufacturers for monitoring network devices. The MIB data is aggregated in units of, for example, 5 minutes, and includes “time, host name, interface (IF) name, input data amount (ibps), output data amount (obsps)”, and the like.

  NetFlow is a technology that performs network monitoring in units of flows, and information about the flows is output when communication is completed. The flow is a unit for grasping “where” and “where” “what kind of communication” “how much” is performed, and the IP address (srcIP) on the sender side of the communication , The sender side port number (srcport), the receiver side IP address (dstIP), the receiver side port number (dstport), and the communication protocol (proto). The flow data includes “flow start time, srcIP, srcport, dstIP, dstport, protocol, flow duration, total number of transmitted packets, total number of transmitted bytes”, and the like.

  The CPU usage rate is, for example, the usage rate of a CPU such as a server device or a router included in the network N1.

  Observation data collected by the measurement device 20 is collected by the abnormality detection device 10. The abnormality detection device 10 learns the characteristics at the normal time from the collected observation data, and detects the occurrence of abnormality in the observation data input thereafter based on the learning result (determines whether there is an abnormality). It is a computer. Note that a process in which normal feature learning is performed is referred to as a “learning process”, and a period in which the learning process is performed is referred to as a “learning period”. A process in which an abnormality is detected based on a result learned in the learning process is referred to as a “test process”, and a period during which the test process is performed is referred to as a “test period”.

  An abnormality recording device 30 for recording information when an abnormality actually occurs is connected to the abnormality detection device 10. The abnormality recording device 30 is a computer for a network operator or the like to record failure information of the network N1. For example, the abnormality recording device 30 can use project management software called redmin. The abnormality recording device 30 records information input time, failure information, log samples, and the like. The information input time is the time when the network operator or the like inputs the failure information, the failure information is the content of the information input by the network operator, and the log sample is a syslog or the like acquired from the measuring device 20 by the network operator or the like. Message. Syslog is a standard for the measurement device 20 to transfer a log message on the network. The log message transferred by the syslog includes “time, message, host” and the like. Is the time output by. Here, the time in the log sample is treated as an abnormality occurrence time when an abnormality has occurred.

  FIG. 2 is a diagram illustrating a hardware configuration example of the abnormality detection apparatus 10 according to the first embodiment. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, and the like, which are mutually connected by a bus B.

  A program that realizes processing in the abnormality detection apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 executes functions related to the abnormality detection device 10 according to a program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

  FIG. 3 is a diagram illustrating a functional configuration example of the abnormality detection device 10 according to the first embodiment. In FIG. 3, the abnormality detection device 10 includes a reception unit 11, a learning processing control unit 12, a preprocessing unit 13, a learning unit 14, a detection processing control unit 15, a detection unit 16, an abnormality occurrence information acquisition unit 17, and a threshold determination unit 18. Etc. Each of these units is realized by processing that one or more programs installed in the abnormality detection apparatus 10 cause the CPU 104 to execute. The anomaly detection apparatus 10 also includes a teacher data storage unit 121, a parameter storage unit 122, an observation data storage unit 123, a learning result storage unit 124, a learning data storage unit 125, a detection result storage unit 126, an abnormality occurrence information storage unit 127, and the like. Is used. Each of these storage units can be realized using, for example, a storage device that can be connected to the auxiliary storage device 102 or the abnormality detection device 10 via a network.

  In the teacher data storage unit 121, observation data that has been confirmed in advance to be collected in a normal state is stored as teacher data. However, the teacher data may be artificially created instead of being selected from the observation data.

The receiving unit 11 receives observation data from the measurement device 20. The received observation data is stored in the observation data storage unit 123. The learning process control unit 12 controls the learning process.

  The preprocessing unit 13 performs preprocessing on a set of teacher data, a set of observation data, or a set of learning data stored in the learning data storage unit 125. Pre-processing is processing such as extraction of feature amounts per unit time from a data set and normalization of extracted feature amounts. The feature amount is expressed in the form of a numerical vector. In the first learning, a teacher data group stored in the teacher data storage unit 121 is a target of preprocessing. When reception of observation data is started by the receiving unit 11, the observation data group is subjected to preprocessing. Furthermore, when the detection of the abnormality by the detection unit 16 is started and it is determined that the detection data is normal and the observation data stored in the learning data storage unit 125 as learning data reaches a predetermined number, the learning data group is subjected to preprocessing. It may be said.

  The pre-processing unit 13 also generates or normalizes parameters for normalizing observation data or learning data (hereinafter referred to as “normalization parameters”) when performing pre-processing on the teacher data group or learning data group. The updated normalization parameter is stored in the parameter storage unit 122.

  The learning unit 14 performs learning based on teacher data or learning data. The learning result by the learning unit 14 is stored in the learning result storage unit 124.

  The detection process control unit 15 controls the detection process.

  The detection unit 16 uses a numerical vector generated when the observation data stored in the observation data storage unit 123 is preprocessed by the preprocessing unit 13 and a learning result stored in the learning result storage unit 124. The occurrence of abnormality is detected based on this. Specifically, the detection unit 16 detects the occurrence of an abnormality by calculating a difference from the learning result as a degree of abnormality for the preprocessed numerical vector and comparing the degree of abnormality with a threshold value. Although the threshold value is set in advance, the threshold value determination unit 18 updates the threshold value to an appropriate value as described below. The value before normalization of the numerical vector in which no abnormality is detected may be stored in the learning data storage unit 125 as learning data. Further, the detection unit 16 stores the detection presence / absence for each unit time, the calculated degree of abnormality, and the like in the detection result storage unit 126.

  The abnormality occurrence information acquisition unit 17 acquires the abnormality occurrence time when the abnormality has occurred from the abnormality recording device 30 and stores it in the abnormality occurrence information storage unit 127. The abnormality occurrence time may be acquired periodically (for example, once a day).

  The threshold value determination unit 18 considers that an error has actually occurred in a certain time zone before and after the abnormality occurrence time stored in the abnormality occurrence information storage unit 127 and stores the abnormality in the detection result storage unit 126. The presence / absence of an abnormality and the presence / absence of a false detection are determined by comparing with the presence or absence of detection. As a result, the threshold determination unit 18 determines a threshold used for comparison with the degree of abnormality in the detection unit 16.

  Hereinafter, a processing procedure executed by the abnormality detection apparatus 10 will be described. FIG. 4 is a flowchart for explaining an example of the processing procedure of the learning process in the first embodiment. In the following, for the sake of convenience, an example in which flow data is a processing target will be described.

  When the learning process is started, the learning process control unit 12 acquires a teacher data group from the teacher data storage unit 121, and inputs the teacher data group to the preprocessing unit 13 (S101).

  Subsequently, the preprocessing unit 13 divides the input teacher data group into sets for each unit time (S102). The teacher data storage unit 121 stores teacher data for a unit time × U period (learning period). Therefore, the teacher data group is divided into U sets.

  Subsequently, the preprocessing unit 13 extracts a feature amount according to the purpose for each divided set, and generates a multidimensional numerical vector having the extracted feature amount as an element of each dimension (S103).

  For example, it is assumed that the unit time is 1 minute, and the preprocessing unit 13 extracts the feature amount every minute. Further, the feature amount is assumed to be the total number of transmitted bytes of each protocol (TCP, UDP). In this case, assuming that the flow start time of the first teacher data is 12:00, the pre-processing unit 13 sets the flow start time t of all the teacher data to 11:55:00 <= t <12: For a set of teacher data (flow data) such as 00:00, the total number of transmitted bytes of all flows whose protocol is TCP, the total number of transmitted bytes of all flows whose protocol is UDP, and the like are calculated. A two-dimensional numerical vector having a quantity as an element of each dimension is generated. Similarly, numerical vectors are generated for (U-1) other sets.

  Note that the attribute of the feature amount can be specified as a combination such as “TCP and transmission port number 80”. Further, if each flow is considered to have a value such as “number of flows: 1”, the total number of flows of each attribute can be calculated in the same manner and regarded as a feature amount.

  Subsequently, the preprocessing unit 13 calculates the maximum value xmax_i of each metric i (each dimension i) in each numerical vector, and stores the calculated xmax_1 in the parameter storage unit 122 (S104). That is, in the first embodiment, the maximum value xmax_i of each metric i is a normalization parameter.

  Here, U = 3. Further, it is assumed that the numerical vector generated in step S103 is {{80, 20}, {90, 35}, {100, 50}}. This is because the total number of TCP transmission bytes and the total number of UDP transmission bytes in a certain three minutes are “TCP: 80 bytes, UDP: 20 bytes”, “TCP: 90 bytes, UDP: 35 bytes”, and “TCP: 100 bytes, UDP: "50 bytes". In this case, the maximum value xmax_i of each metric of these numerical vectors is {100, 50} (that is, xmax_1 = 100, xmax_2 = 50).

  Subsequently, the preprocessing unit 13 normalizes each numerical vector based on the normalization parameter (S105). Normalization is performed by dividing the value of the metric i of each numerical vector by the maximum value xmax_i. Therefore, the normalized numerical vector is {{0.8, 0.4}, {0.9, 0.7}, {1, 1}}.

  Subsequently, the learning unit 14 learns the numerical vector using a learning device (S106). The learning result is stored in the learning result storage unit 124.

  Subsequently, the learning process control unit 12 waits for the learning data for the learning period to be stored (accumulated) in the learning data storage unit 125 (S107). That is, the standby is continued until U number of numerical vectors before normalization are stored in the learning data storage unit 125. The learning data storage unit 125 stores a numerical vector determined to be normal (no abnormality has occurred) by the detection unit 16.

  When the numerical vectors for the learning period are stored in the learning data storage unit 125 (Yes in S107), the learning processing control unit 12 acquires the numerical vector group from the learning data storage unit 125, and preprocesses the numerical vector group. Input to the unit 13 (S108). The acquired numerical vector group is deleted from the learning data storage unit 125. Subsequently, step S104 and subsequent steps are executed for the numerical vector group. Therefore, in the next step S105, normalization is performed based on the newly calculated xmax_i.

  FIG. 5 is a flowchart for explaining an example of the processing procedure of the detection processing in the first embodiment. The processing procedure in FIG. 5 may be started any time after step S106 in FIG. 4 has been executed at least once. That is, the processing procedure of FIG. 5 is executed in parallel with the processing procedure of FIG.

In step S201, the detection processing control unit 15 waits for the unit time to elapse. The unit time is the same length as the unit time in the description of FIG. During this standby, the observation data collected in real time and received by the receiving unit 11 is stored in the observation data storage unit 123. When the unit time has elapsed (Yes in S201), the detection processing control unit 15 The observation data group for the time is acquired from the observation data storage unit 123, and the observation data group is input to the preprocessing unit 13 (S202).

  Subsequently, the preprocessing unit 13 extracts a feature amount according to the purpose from the observation data group, and generates a multidimensional numerical vector having the extracted feature amount as an element of each dimension (S203). For example, the total number of transmitted bytes of all flows whose protocol is TCP and the total number of transmitted bytes of all flows whose protocol is UDP are extracted, and a two-dimensional numerical vector having these as elements of each dimension is generated. Here, one numerical vector is generated.

  Subsequently, the preprocessing unit 13 normalizes the generated numerical vector based on the maximum value xmax_i stored in the parameter storage unit 122 (S204). That is, each metric i of the numeric vector is divided by the maximum value xmax_i.

  For example, when step S104 of FIG. 4 is executed only once based on the teacher data, the maximum value xmax_i is {100, 50}. Therefore, when the numerical vector is {60, 40}, the numerical vector is normalized to {0.6, 0.8}.

  Subsequently, the detection unit 16 executes an abnormality determination process (S205). In the abnormality determination process, the difference between the normalized numerical vector and the latest learning result stored in the learning result storage unit 124 is calculated as the degree of abnormality. By comparing the degree of abnormality with a threshold value, it is determined whether there is an abnormality in the network N1. The detection unit 16 stores the determined presence / absence of abnormality (that is, detection presence / absence) and the calculated degree of abnormality in the detection result storage unit 126.

  When it is determined that there is no abnormality (Yes in S206), the detection processing control unit 15 stores the numerical vector before normalization of the numerical vector in the learning data storage unit 125 as learning data (S207). When it is determined that there is an abnormality (No in S206), the numerical vector before normalization of the numerical vector is not stored in the learning data storage unit 125. Therefore, the learning data storage unit 125 stores only normal numerical vectors.

  Subsequently, step S201 and subsequent steps are repeated. Note that, in the process of repeating step S201 and subsequent steps, the normalization parameter used in step S204 is updated as needed in step S104 of FIG. 4 being executed in parallel. As a result, the numerical vector can be normalized in consideration of the trend of the input observation data.

  For example, when U = 3, step S207 is executed three times, and {{60, 40}, {45, 20}, {30, 30}} is stored in the learning data storage unit 125. In this case, it is updated to xmax_1 = 60 and xmax_2 = 40, and the update result is reflected in the parameter storage unit 122.

  In the above description, an example in which the observation data is flow data has been described. However, flow data, MIB data, and a CPU usage rate may be received in parallel as observation data. In this case, each step of the processing procedure of FIGS. 4 and 5 may be executed for each data type (flow data, MIB data, and CPU usage rate).

  For example, for MIB data given in a format such as {hostID, interfaceID, ibps, obsps}, “host IDa ibps in unit time”, “host IDa obsps in unit time”, “host IDb in unit time” Ibps of host IDb in unit time "..." interfaces IDx ibs in unit time "," interfaceIDx obsps in unit time "," interfaceIDy ibps in unit time "," interfaceIDy obsps in unit time " It is possible to extract a numerical vector like

  Next, an example of step S106 in FIG. 4 and step S205 in FIG. 5 will be described. In steps S <b> 106 and S <b> 205, a numerical vector group assigned with the data type as a label is input to the learning unit 14 or the detection unit 16. In the present embodiment, the label is any one of “flow data”, “MIB data”, and “CPU usage rate”. For example, the label is given to the teacher data and the observation data by the measuring device 20 or the receiving unit 11. That is, the label to be given to the observation data can be specified based on the observation data collection source. The label is inherited by a numerical vector generated by the preprocessing unit 13.

  In step S106 of FIG. 4, the learning unit 14 generates a learning device for each data type. The learning unit 14 classifies the numerical vector based on the label given to the input numerical vector, and inputs the numerical vector to the learning device corresponding to the classification result. In the present embodiment, a “flow data learning device”, a “MIB data learning device”, and a “CPU usage rate learning device” are generated. As a learning device, it is possible to use an auto encoder, principal component analysis, or the like that performs abnormality detection by learning a correlation between numerical vector metrics. For more information on auto-encoders, refer to “Mayumi Kajita and Takehisa Yairi,“ Detection of spacecraft anomalies by dimension reduction using auto-encoders ”, Proceedings of the Japanese Society for Artificial Intelligence 28, 1-3, 2014”. The principal component analysis is detailed in, for example, “Ringberg, Haakon, et al.“ Sensitivity of PCA for traffic anomaly detection. ”ACM SIGMETRICS Performance Evaluation Review 35.1 (2007): 109-120. In this embodiment, an example in which an auto encoder is used as a learning device will be described.

  FIG. 6 is a diagram for explaining the auto encoder. The auto encoder is an anomaly detection algorithm based on deep learning. The auto encoder utilizes the fact that the input data at normal time has a correlation between metrics and can be compressed to a low dimension. Since the correlation between the input data is lost at the time of abnormality, the compression is not performed correctly, and the difference between the input data and the output data becomes large.

As shown in (1) of FIG. 6, the learning device (auto encoder) generated by the learning unit 14 performs learning so that the output layer (Layer L ₃ ) is close to the input layer (Layer L ₁ ). Specifically, the learning unit 14 duplicates the numerical vector, applies one to the input layer, applies the other to the output layer, performs learning, and outputs a learning result. The learning result is stored in the learning result storage unit 124. The learning result is a parameter group for the learning device. Since learning devices are generated for each data type, learning results are also output for each data type and stored in the learning result storage unit 124.

  On the other hand, similarly to the learning unit 14, the detection unit 16 also generates a learning device for each data type. As the learning device, a method corresponding to the learning device generated by the learning unit 14 in the auto encoder, the principal component analysis, or the like can be used as in the learning device generated by the learning unit 14.

  In step S205 in FIG. 5, the detection unit 16 performs “flow data learning device”, “MIB data learning device”, and “CPU usage rate learning” based on the learning result stored in the learning result storage unit 124. Is created. That is, the learning device generated by the detection unit 16 is the same as the learning device generated by the learning unit 14 when the learning result is output. As shown in (2) of FIG. 6, the detection unit 16 inputs the numerical vector for each data type input in step S205 to a learning device corresponding to the data type of the numerical vector, and inputs data to the learning device. The distance between the output data and the output data (an index indicating the degree of collapse of the correlation between metrics) is calculated as the degree of abnormality. In the present embodiment, a mean squared error (MSE) that is a distance between the input layer and the output layer of the auto encoder is calculated as the degree of abnormality. The calculation formula of MSE is as follows.

本実施の形態では、フローデータのＭＳＥ、ＭＩＢデータのＭＳＥ、ＣＰＵ使用率のＭＳＥの３種のＭＳＥが得られる。検知部１６は、得られたＭＳＥの平均を、最終的な異常度として計算し、最終的な異常度が予め定められた閾値を超えていた場合に異常であると判定する。そうでない場合、検知部１６は、正常とであると判定する。

In the present embodiment, three types of MSEs are obtained: MSE for flow data, MSE for MIB data, and MSE for CPU usage. The detection unit 16 calculates the average of the obtained MSEs as the final abnormality degree, and determines that the abnormality is abnormal when the final abnormality degree exceeds a predetermined threshold. Otherwise, the detection unit 16 determines that it is normal.

  FIG. 7 is a flowchart for explaining an example of the processing procedure of the threshold value determination process in the first embodiment. In the processing procedure of FIG. 7, the abnormality occurrence time acquired from the abnormality recording device 30 by the abnormality occurrence information acquisition unit 17 is stored in the abnormality occurrence information storage unit 127, and step S205 of FIG. 5 is executed at least once. It can be started at any time later. That is, the processing procedure of FIG. 7 is executed in parallel with the processing procedure of FIG.

The threshold value determination unit 18 totals the abnormality occurrence time stored in the abnormality occurrence information storage unit 127 and the presence / absence of detection stored in the detection result storage unit 126 for each unit time (S301). For example, if there is no abnormality occurrence time at 11:55:00 <= t <12:00, no abnormality is detected, and no abnormality is detected at 11:55:00 <= t <12:00 If there is no detection, the presence / absence of abnormality and the presence / absence of detection are totaled as follows.
Time Abnormality presence / absence Detection presence / absence 12:00 0 0
12:01 1 0
12:02 0 1
12:03 1 1
...
Here, 1 indicates that there is an abnormality and detection is present, and 0 indicates that there is no abnormality and no detection. In the above example, since the detection unit 16 could not detect the occurrence of the abnormality even though an abnormality occurred at 12:01, it can be determined that it is overlooked. Moreover, since the detection part 16 has detected the occurrence of an abnormality even though no abnormality has occurred at 12:02, it can be determined that it is a false detection. Since an oversight occurs because the threshold is high and an erroneous detection occurs because the threshold is low, the threshold value determination unit 18 increases the threshold value when an erroneous detection occurs, and decreases the threshold value when an oversight occurs. Let

  Specifically, the threshold value determination unit 18 calculates the number of false detections n_1 and the number of misses n_2 from the total results of the presence / absence of abnormality and the presence / absence of detection (S302). In the aggregation results from 12: 00 to 12: 03, the calculation is performed as n_1 = 1 and n_2 = 1. Here, the greater the number of false detections, the greater the increase in the threshold value, and the greater the number of misses, the smaller the decrease in the threshold value. For example, if the current threshold value is θ_0, the newly applied threshold value θ is determined as follows.

θ = θ_0 + n_1 * Δθ−n_2 * Δθ
Here, Δθ is a parameter for obtaining the increase / decrease amount of the threshold (n_1 * Δθ−n_2 * Δθ). If this parameter Δθ is too large, the threshold θ may fluctuate greatly, and the possibility of false detection or oversight may increase.

  Therefore, when the calculation of the number of false detections and the number of misses for a certain period is completed (Yes in S303), the threshold determination unit 18 standard deviation σ_1 of the number of false detections n_1 and the standard deviation of the number of misses n_2. σ_2 is calculated (S304).

  Assuming that a parameter for obtaining a predetermined increase / decrease amount is Δθ_0, a newly applied parameter Δθ is calculated as follows based on σ_1 and σ_2 (S305).

Δθ = Δθ — 0 * (1 / σ_1) * (1 / σ_2)
Note that Δθ may be a fixed value determined in advance, a fixed value obtained through trial and error, or the like, without dynamically changing as described above.

  The threshold value determination unit 18 calculates an increase / decrease amount of the threshold value (n_1 * Δθ−n_2 * Δθ) using Δθ, n_1, and n_2, and determines a threshold value θ to be newly applied (S306).

  As described above, according to the first embodiment, the initial threshold value is automatically updated to an appropriate threshold value. In addition, an appropriate threshold increase / decrease amount is determined based on the number of false detections and the number of misses, thereby reducing false detections and misses.

  Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be described. Points that are not particularly mentioned in the second embodiment may be the same as those in the first embodiment.

  In the second embodiment, the threshold value θ is determined from the distribution of the degree of abnormality calculated by the detection unit 16.

  FIG. 8 is a flowchart for explaining an example of a processing procedure of threshold determination processing according to the second embodiment. In the processing procedure of FIG. 8, the abnormality occurrence time acquired from the abnormality recording device 30 by the abnormality occurrence information acquisition unit 17 is stored in the abnormality occurrence information storage unit 127, and step S205 of FIG. 5 is executed at least once. It can be started at any time later. That is, the processing procedure of FIG. 8 is executed in parallel with the processing procedure of FIG.

  The threshold value determination unit 18 refers to the abnormality occurrence time stored in the abnormality occurrence information storage unit 127, and the abnormality degree x_1,... Stored in the detection result storage unit 126 in the time zone when no abnormality has occurred. ..., X_N (N: the number of abnormalities calculated in the normal state) is acquired (S401). That is, the threshold value determination unit 18 acquires the abnormalities x_1,..., X_N calculated at the normal time. The abnormalities x_1,..., X_N at normal time are obtained by inputting the teacher data stored in the teacher data storage unit 121 or the learning data stored in the learning data storage unit 125 to the detection unit 16. It may be sought.

  The threshold value determination unit 18 calculates the average μ and the standard deviation σ of the abnormalities x_1,..., X_N (S402).

  The threshold value determination unit 18 defines the degree of deviation from the distribution of the degree of abnormality at normal time, and determines the threshold θ based on the degree of deviation (S403). For example, the detection unit 16 calculates when observation data is input. When the degree of abnormality exceeds the normal average of + 3σ, the observation data is regarded as abnormal, and the threshold θ of the degree of abnormality is determined as θ = μ + 3σ.

  As described above, according to the second embodiment, the threshold value is automatically determined from the normal data. Note that the threshold value θ determined in the second embodiment can be used as the initial value θ_0 of the threshold value in the first embodiment.

  In the third embodiment, the threshold θ is determined so that the detection accuracy of the abnormality detected by the detection unit 16 satisfies the target value. As the abnormality detection accuracy, there are a precision, a recall, an F value, and the like. “Precision” is a ratio of those actually detected out of those detected as abnormal, and corresponds to a small number of false detections. “recall” is a ratio of those which are abnormal in nature and detected as abnormal, and corresponds to a small amount of oversight. The F value is a harmonic average of the precision and the recall.

  FIG. 9 is a flowchart for explaining an example of a processing procedure of threshold value determination processing according to the third embodiment. In the processing procedure of FIG. 9, the abnormality occurrence time acquired from the abnormality recording device 30 by the abnormality occurrence information acquisition unit 17 is stored in the abnormality occurrence information storage unit 127, and step S205 of FIG. 5 is executed at least once. It can be started at any time later. That is, the processing procedure of FIG. 9 is executed in parallel with the processing procedure of FIG.

  The threshold value determination unit 18 acquires time series x_1,..., X_T of the degree of abnormality for each unit time stored in the detection result storage unit 126, and the abnormality occurrence time stored in the abnormality occurrence information storage unit 127. Is acquired (S501). As in step S301, when there is an abnormality at time t, b_t = 1 is set, and when there is no abnormality, b_t = 0 is set, and time series b_1,.

  Let us consider a case where the metric having the target accuracy is set to “recall” and the maximum threshold value at which “recall” satisfies the target value “r” is adopted. In this case, the number m of abnormalities that need to be detected at time t = 1,..., T can be calculated as m = r * Σ_ {t = 1,..., T} b_t (S502).

  Therefore, the threshold value determination unit 18 determines the m-th value when the abnormality levels {x_t | b_t = 1 for t = 1,..., T} at the time when the abnormality occurs are arranged in descending order as the threshold value θ. (S503).

  When the target accuracy metric is a precision or F value, the initial value of the threshold is set to 0, the precision or F value is calculated while gradually increasing the threshold, and the calculated precision or F value is the target. A threshold value when the value is satisfied can be adopted.

  As described above, according to the third embodiment, the threshold value that satisfies the target value of the abnormality detection data is automatically determined. Note that the threshold value θ determined in the third embodiment can also be used as the initial value θ_0 of the threshold value in the first embodiment.

  Each of the above embodiments may be applied to data collected from other than the network. For example, the above-described embodiments may be applied to data collected from a computer system.

  In each of the above embodiments, the abnormality detection device 10 is an example of a threshold value determination device. The abnormality occurrence information acquisition unit 17 is an example of an acquisition unit.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

DESCRIPTION OF SYMBOLS 10 Abnormality detection apparatus 11 Receiving part 12 Learning process control part 13 Preprocessing part 14 Learning part 15 Detection process control part 16 Detection part 17 Abnormality generation information acquisition part 18 Threshold value determination part 20 Measuring apparatus 30 Abnormal recording apparatus 100 Drive apparatus 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 CPU
105 interface device 121 teacher data storage unit 122 parameter storage unit 123 observation data storage unit 124 learning result storage unit 125 learning data storage unit 126 detection result storage unit 127 abnormality occurrence information storage unit B bus N1 network

Claims (8)

Calculate the degree of abnormality based on the learning result of learning the data collected from the abnormality detection target and the data observed from the abnormality detection target during the test period, and compare the abnormality degree with the threshold value. A detection unit for detecting occurrence;
A threshold value determination unit that determines the threshold value based on whether there is an oversight of an abnormality and a false detection;
A threshold value determination device.   The threshold value determination device according to claim 1, wherein the threshold value determination unit increases the threshold value when an erroneous detection occurs and decreases the threshold value when an oversight occurs.   The threshold value determination unit calculates the increase / decrease amount of the threshold value based on the number of false detections, the standard deviation of the number of false detections, and the standard deviation of the number of misses and the number of misses. Threshold determination device.   4. The threshold value determination unit according to claim 1, wherein the threshold value determination unit determines the threshold value based on a degree of deviation from the distribution of the degree of abnormality calculated by the detection unit when no abnormality is detected. 5. Threshold determination device. It further has an acquisition unit for acquiring an abnormality occurrence time when an abnormality has occurred in an abnormality detection target,
4. The threshold value determination unit according to claim 1, wherein the threshold value determination unit determines the threshold value based on the abnormality occurrence time so that the detection accuracy of the abnormality detected by the detection unit satisfies a target value. 5. The threshold value determination device according to item. It further has an acquisition unit for acquiring an abnormality occurrence time when an abnormality has occurred in an abnormality detection target,
The threshold determination unit determines that the detection unit is overlooked when the detection unit does not detect the occurrence of an abnormality in the time zone of the abnormality occurrence time, and the detection unit is in an abnormal state other than the time zone of the abnormality occurrence time. The threshold value determination apparatus according to any one of claims 1 to 4, wherein when the occurrence is detected, it is determined that the detection is a false detection. Calculate the degree of abnormality based on the learning result of learning the data collected from the abnormality detection target and the data observed from the abnormality detection target during the test period, and compare the abnormality degree with the threshold value. A detection procedure to detect the occurrence;
A threshold value determination procedure for determining the threshold value based on whether there is an oversight of an abnormality and a false detection;
A threshold determination method in which the computer executes.   The program for functioning a computer as each part as described in any one of Claims 1 thru | or 6.

Images