JP2008154010A - Data processor, and data processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To smooth a feature place of time series data preferentially. <P>SOLUTION: A data input/processing part 420 accumulates input data 410 in each unit time, a feature analysis part 430 classifies the accumulated data into prescribed areas and performs main component analysis in each area to calculate a feature vector, a protruding point determining part 440 arranges the feature vector of each area in a two-dimensional plane and determines a protruding rate from a distribution of feature vectors, a smoothing coefficient calculating part 450 gives the protruding rate of an area corresponding to each accumulated data from the data input/processing part 420, calculates a relative evaluation value in the area of each data, calculates a smoothing coefficient in which the relative evaluation value of each data, the protruding rate of the area of each data and the number of data to be a target of a moving average calculation by a smoothing part 460 become a proportional relation, and the smoothing part 460 calculates the number of target data of a moving average on the basis of the smoothing coefficient and smoothes each accumulated data from the data inputting/processing part 420 by the moving average calculation. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for smoothing time-series data.

In unauthorized access detection, there is a method of detecting anomalies by analyzing time-series data generated from collected packet logs.
In this method, time series data and learning data are compared. Learning data is a reference for measuring the amount of change in time-series data.

FIG. 1 shows a configuration example of an unauthorized access analysis system 100 described in Non-Patent Document 1, for example.
The unauthorized access analysis system 100 shown in FIG. 1 targets a network belonging to a specific organization such as a company as a monitoring target, for example, as shown in FIG. A firewall (F / W), an S-NIDS (Signature based Network IDS) (Intrusion Detection System), and a packet log (fixed point observation data) from the packet collection device are input to the unauthorized access analysis system 100 and analyzed in real time.

In FIG. 1, the information collection unit 6 periodically collects F / W, S-NIDS, and packet logs of the packet collection device.
The log information totaling unit 5 totals packet information necessary for detecting unauthorized access from the packet logs collected by the information collecting unit 6. For example, the number of packets per source IP address per unit time, the number of packets per destination port, or the packet length, etc. are aggregated.
The abnormality detection unit 4 detects abnormal network traffic based on the data aggregated by the log information aggregation unit 5 and outputs an early alert.
The unauthorized access determination unit 3 is a function that determines that an unauthorized access is caused when an abnormal state of traffic is detected by the abnormality detection unit 4. The log information aggregation unit 5 performs aggregation from a plurality of analysis viewpoints, comprehensively determines the detection results of the abnormality detection unit 4 for each of them, and determines that unauthorized access is the cause. Further, known vulnerability information stored in a security information database (not shown) is also used for determination. For example, if an abnormality is detected in the analysis result of a packet to a specific service (port) in the abnormality detection unit 4 and the vulnerability of the service has been disclosed recently, an unauthorized use of the vulnerability It can be determined that there is a possibility of access.
If it is determined that there is a false detection, the information is in a normal state and fed back to the abnormality detection unit 4.
The security information database is, for example, a database that manages the latest vulnerability information / patch information of software.
The countermeasure unit 2 is a function of outputting countermeasure guidelines such as an instruction for restricting access to a specific port, applying a patch, etc. when unauthorized access is determined by the unauthorized access determination unit 3. The network administrator takes measures against this output.
A GUI (Graphical User Interface) 1 displays an early alert, the cause of unauthorized access, countermeasure information, and the like.

Next, as an example of a conventional technique for analyzing time series data, an example of the abnormality detection unit 4 when principal component analysis is used will be described.
In the method using the principal component analysis, the variation occurring in the time series data is determined. The determination method follows the following procedure.
FIG. 26 shows details of the abnormality detection unit 4.
The data acquisition unit 43 inputs time-series data (input data 42) and defines the learning data 41.
The analysis unit 44 calculates the feature amount of the time series data.
The determination unit 45 determines an abnormal value of the time series data.

The data acquisition unit 43 inputs time-series data for measuring abnormality and defines learning data.
As described above, the learning data is a reference for measuring the amount of change in the time series data. There are a method of using a part of input time series data as learning data and a method of creating based on some modeling.
In the case of the example in FIG. 26, the learning data is defined as a continuous range of a range in the time series data.

The analysis unit 44 analyzes the time series data.
Here, a feature amount related to time series data is calculated.
As an example of the analysis method, the time series data obtained from the data acquisition unit 43 is decomposed into a certain size every unit time.
Each decomposed data is analyzed and converted into a small number of features.
As a result, multi-dimensional time-series information generated in a certain period is compressed into smaller-dimensional information. As a result, it becomes possible to analyze the abnormality at a higher speed.

The determination unit 45 compares the feature amount of the time series data obtained by the analysis unit 44 with the previously defined learning data 41.
As a result of the comparison, if the input data 42 is different from the learning data 41, it is determined that the input data 42 is abnormal.
In the comparison method, a space for feature amounts is defined, and the input data 42 analyzed and converted into feature amounts is placed in this space. After that, there is a method in which the distribution of the input data 42 is examined, and the data that deviates from the distribution group by a certain amount is regarded as abnormal.

  The above procedure is repeated. When new time series data is analyzed, the training data is also defined again.

Moreover, as a prior art which performs smoothing processing and analyzes time-series data, for example, there are techniques described in Patent Document 1 and Non-Patent Document 2.
These techniques are techniques for detecting change points in time series data.
In these techniques, smoothing processing using moving average processing is performed on time-series data, but smoothing is performed on all regions of time-series data.
That is, smoothing is performed on the entire target time-series data.
Japanese Patent Laid-Open No. 2000-213948 Hiroyuki Sugawara, Seiji Fujii, Shigeki Kitazawa, Norio Hirai, Rika Kashima, Shinsuke Higashi, “Proposal of Unauthorized Access Analysis System by Fixed-Point Observation”, IPSJ 68th National Convention, IPSJ, 2006. Junichi Takeuchi and Kenji Yamanishi, “Unified treatment of outlier detection and change point detection using forgetting learning algorithm”, 2000 Information theory learning theory workshop, 2002.

The conventional method for smoothing time-series data is to perform smoothing on the entire time-series data.
For this reason, when the conventional method is used for detecting an abnormality in the network, the necessary information is smoothed, and the detection performance is deteriorated.
In the case of the time series data abnormality detection method as described above, there may be a problem in detection performance depending on the method of defining the learning data.
An example is when the learning data contains noise.
When a part of the learning data includes a prominent value having a different tendency from the others, it greatly affects the determination of abnormality. FIG. 26 shows an example in which detection is performed using learning data in the prior art.
In the example of FIG. 26, a method of comparing the Mahalanobis generalized distance value and the distribution of the learning data area is used as a method of determining whether a large variation has occurred in the time series data.
In the conventional abnormality judgment method. Learning data (stationary area data) is used as a target for comparing anomalies.
In the determination process, information in the learning data is analyzed to determine a threshold value for abnormality determination. When information having a different tendency such as noise exists in the learning data, the conventional method has a problem that the abnormality determination threshold includes noise and delays the abnormality determination.

Conventionally, the following threshold values are used as a determination method that time-series data fluctuation has occurred.
Data value at the time of detection> a times the maximum value in the learning data (a is a constant)
In this method, when learning data includes data having a value larger than that of other data, it is strongly influenced by the learning data. Therefore, when the data value at the time of detection changes smaller than the threshold value, there is a high possibility of overlooking the abnormality.

  The main object of the present invention is to solve such problems, and to preferentially smooth characteristic portions of the learning data and to improve the accuracy of the subsequent abnormality detection processing. With a purpose.

The data processing apparatus according to the present invention
A divergence value setting unit that performs analysis on a plurality of data each having a data value, and sets, for each data, the degree of deviation of the data value of each data from the data value of other data, as a divergence value;
A smoothing coefficient calculating unit that calculates a smoothing coefficient for smoothing for each data, reflecting the deviation value of each data set by the deviation value setting unit;
And a smoothing unit that performs smoothing of each data using the smoothing coefficient of each data calculated by the smoothing coefficient calculating unit.

  According to the present invention, since the degree of smoothing is increased with respect to data whose data value is different from other data, noise can be removed and the accuracy of abnormality detection can be improved.

Embodiment 1 FIG.
In this embodiment, time series data used as learning data is smoothed to remove information that hinders abnormality detection. At this time, smoothing is performed more strongly on regions where the tendency of data is characteristic.

FIG. 3 shows a configuration example of the abnormality detection unit 4 (data processing apparatus) according to the present embodiment.
The abnormality detection unit 4 according to the present embodiment is a part of the unauthorized access analysis system 100 shown in FIG. Details of the other elements of the unauthorized access analysis system 100 are as described above, and a description thereof will be omitted.
The unauthorized access analysis system 100 may be realized by a single computer as a whole, or each element included in the unauthorized access analysis system 100 is realized by a different computer, and each computer is connected via a network for unauthorized access. It may be a form in which the analysis system is realized.

  In addition, as described above, the unauthorized access analysis system 100 including the anomaly detection unit 4 according to the present embodiment monitors a network belonging to a specific organization such as a company as shown in FIG. . Packet logs (fixed point observation data) from the firewall (F / W), S-NIDS, and packet collection device are input to the unauthorized access analysis system 100 and analyzed in real time.

In FIG. 3, the data input / processing unit 420 stores the total number of input data 410 that is time series data per unit time. This input data 410 is time series data used as learning data. Hereinafter, the input data 410 is also referred to as time series data.
The feature quantity analysis unit 430 calculates principal component scores from the time series data aggregated by the data input / processing unit 420 and collects them into a group of feature regions.
The protruding point determination unit 440 (deviation value setting unit) investigates the group of feature regions obtained by the feature amount analysis unit 430, and scores the region compared to other regions. In other words, the protrusion point determination unit 440 performs analysis on a plurality of data each having a data value, and the degree of protrusion of the data value of each data from the data value of other data is determined for each data. Value). Specifically, as will be described later, the protruding point determination unit 440 groups each data for each predetermined region, and analyzes the degree of deviation for each region to determine the protruding rate.
The smoothing coefficient calculation unit 450 determines a parameter relating to smoothing according to the score of the feature region in the protruding point determination unit 440. That is, the smoothing coefficient calculation unit 450 reflects the protrusion rate determined by the protrusion point determination unit 440 and calculates a smoothing coefficient for smoothing for each data.
The smoothing unit 460 smoothes the time series data according to the parameters in the smoothing coefficient calculation unit 450. Although details will be described later, the smoothing unit 460 performs smoothing by performing a moving average calculation using an arbitrary number of data on the data to be smoothed.

Here, with reference to the flowchart of FIG. 24, the operation example (data processing method) of the abnormality detection unit 4 (data processing apparatus) according to the present embodiment will be outlined.
In this embodiment, a part of the time-series data to be inspected is used as learning data, and when the time-series data to be inspected is input, the processing shown in the flowchart of FIG. Data is smoothed.

First, the data input / processing unit 420 inputs input data 410 that is time-series data to be smoothed (S2401). As described above, since part of the time series data that is the target of abnormality detection is used as learning data, the data input / processing unit 420 inputs part of the time series data that is the target of abnormality detection as input data 410. To do.
Then, the data input / processing unit 420 aggregates the input data 410 every predetermined unit time (S2402).
Thereafter, the data input / processing unit 420 outputs the aggregated data to the feature amount analysis unit 430, the protruding point determination unit 440, and the smoothing coefficient calculation unit 450, respectively.

Next, the feature amount analysis unit 430 inputs the data output from the data input / processing unit 420, divides the input data into predetermined regions, and calculates a feature amount for each region (S2403).
The data from the data input / processing unit 420 is arranged according to a predetermined order. The data is grouped into a plurality of areas (groups) according to this order, and the principal component analysis of the data values of the data included in each area is performed. Thus, the feature amount of each group is calculated.
Then, the feature amount analysis unit 430 outputs data indicating the feature amount for each region to the protruding point determination unit 440.

The protruding point determination unit 440 arranges the feature amounts of each region on a two-dimensional plane, and determines a protruding rate (deviation value) from the distribution of the feature amounts (S2404) (deviation value setting step).
That is, the protruding point determination unit 440 sets, as the protruding rate, the degree that the feature amount of each region deviates from the feature amount of the other region for each region grouped by the feature amount analysis unit 430. Details of the protrusion rate will be described later.
Thereafter, the protrusion point determination unit 440 outputs data indicating the protrusion ratio of each region to the smoothing coefficient calculation unit 450.

The smoothing coefficient calculation unit 450 inputs data aggregated for each unit time by the data input / processing unit 420 and inputs data indicating the protrusion rate of each region from the protrusion point determination unit 440.
Then, the smoothing coefficient calculation unit 450 assigns the protrusion rate of the corresponding region to each data from the data input / processing unit 420, calculates the relative evaluation value in the region of each data, The smoothing coefficient for each data is calculated by reflecting the evaluation value and the protrusion rate of the area to which each data belongs (S2405) (smoothing coefficient calculating step).
Here, the relative evaluation value is an evaluation value indicating how a certain data value is compared with other data included in the same region. Details of the relative evaluation value will be described later.
The smoothing coefficient calculation unit 450 calculates a smoothing coefficient in which the relative evaluation value of each data and the protrusion rate of the region to which each data belongs and the degree of smoothing by the smoothing unit 460 are in a proportional relationship.
Specifically, a smoothing coefficient is calculated in which the relative evaluation value and the protrusion ratio are proportional to the number of data to be subjected to moving average calculation by the smoothing unit 460.
As described above, for data having a large relative evaluation value or protrusion ratio, the degree of smoothing is increased by increasing the number of data targeted for moving average.

Finally, the smoothing unit 460 inputs the data aggregated per unit time by the data input / processing unit 420, and also inputs the smoothing coefficient of each data from the smoothing coefficient calculation unit 450, and according to the smoothing coefficient Each data is smoothed (S2406) (smoothing step).
The smoothing unit 460 determines the number of data to be subjected to moving average calculation according to the smoothing coefficient, and performs the moving average calculation using data of the determined number of data to smooth the data.
In the example shown in this embodiment, moving average calculation is performed using the same number of data as the smoothing coefficients.

  Next, the operation of the abnormality detection unit 4 according to the present embodiment will be described in detail.

The data input / processing unit 420 aggregates the input data 410 to be analyzed every unit time. The parameters for initial setting are as follows.
Aggregation unit time: Unit time for aggregation of time series data to be observed

The format of the input data 410 is shown in FIG.
Note that the serial numbers shown in FIG. 4 represent each data and are described for explanation, and do not exist in actual data.
The input data 410 is, for example, data of the number of packets for each source IP address. Usually, such input data 410 is generated irregularly. Therefore, the data input / processing unit 420 performs the aggregation unit time specified in advance. Summarize the data.
In FIG. 4, the event occurrence date and time (pre-aggregation event occurrence date and time) are irregular time intervals.

FIG. 5 is an example of input data after aggregation.
In FIG. 5, the event occurrence date / time (post-aggregation event occurrence date / time) is the first time when the aggregation is started per unit time. Further, the number of event occurrences (the number of event occurrences after aggregation) is the total number of event occurrences before aggregation that occurred per unit time.
When the unit time of the input data is divided into {T ₁ , T ₂ , T ₃ }, {T ₄ , T ₅ }, {T ₆ , T ₇ }, the total result is three types of information. The result of totaling the data of the unit time {T ₁ , T ₂ , T ₃ } is the serial number a ₁ . The post-aggregation event occurrence date / time is T ₁ , and the post-aggregation event occurrence number is the sum of C ₁ to C ₃ .
As in FIG. 4, the serial numbers in FIG. 5 are added for the sake of explanation and do not exist in actual data.
Further, as shown in FIG. 3, the data in FIG. 5 is output to each of the feature amount analysis unit 430, the protruding point determination unit 440, and the smoothing coefficient calculation unit 450.

FIG. 14 shows an example when the input data 410 is tabulated at intervals of 5 minutes.
The top eight events of the input data 410 are aggregated into five events.
Of the input data, 2006/07/01 0:00:20 and 2006/07/01 0:01:13, 2006/07/01 0:03:04 are events that occurred within the first 5 minutes, so one event And
At that time, the event occurrence date and time uses the information (2006/07/01 0:00:20) that appears earlier, and the event occurrence number is the total number 17 (4 + 8 + 5) of both.
Similarly, the event occurrence dates are 2006/07/01 0:10:33 and 2006/07/01 0:11:30, 2006/07/01 0:16:22 and 2006/07/01 0:19: 54 things are put together.
If the event occurs only once within the totaling time of the event (2006/07/01 0:22:43), the event generation time is held as it is. In the case of 2006/07/01 0:05:00), the event occurrence number is set to 0.

The feature quantity analysis unit 430 calculates principal component scores from the time series data aggregated by the data input / processing unit 420, and then converts the principal component scores into time series. The parameters for initial setting are as follows.
Principal component target dimensions: Number of dimensions for calculating principal component analysis

The number of principal component target dimensions is the number of time series data received from the data input / processing unit 420 that is the number of columns of the principal component target matrix when calculating the principal component analysis.
The feature amount analysis unit 430 extracts the data of the number of principal component target dimensions from the beginning of the time series data, and applies it to the principal component analysis.
An example of input data of the feature quantity analysis unit 430 is shown in FIG.
The data of FIG. 6 that is input data of the feature quantity analysis unit 430 and the data of FIG. 5 that is output data of the data input / processing unit 420 are the same.
In FIG. 5 and FIG. 6, the notation method is different for the convenience of the following description. However, the post-aggregation event occurrence date / time T ₁ of serial number a _{1 and} the post-aggregation event occurrence number C ₁ + C ₂ + C ₃ Corresponding to the event occurrence date / time T ₁ and the event occurrence number C ₁ of the serial number d _{1 in} FIG. 6, the post-aggregation event occurrence date / time T ₄ and the post-aggregation event occurrence number C ₄ + C ₅ of the serial number a ₂ in FIG. event occurrence time _{T 2} of the serial number _{d 2} in a relationship corresponding to the event occurrence count _{C 2.} The same applies to the subsequent lines.

Here, assuming that the number of principal component target dimensions is k, k pieces are grouped together from the beginning of the time series data, and processing is performed for each group (for each region). In the case of the example in FIG. 6, a matrix of 1 row and k columns is created from the number of event occurrences from d ₁ to d _k , and principal component analysis is performed with elements included in this matrix as one group (region). The matrix to handle is as follows.
(C ₁ , C ₂ ,..., C _k )
Thereafter, the next k pieces are extracted from the time-series data, a matrix is similarly created, and principal component analysis is performed. This process is repeated sequentially.

As a result of the principal component analysis, a time series of principal component scores representing k pieces of time series data is obtained. The principal component score is a plurality of scores, such as first, second,..., And the first two are used in the subsequent steps.
FIG. 7 shows the relationship between the sequence created from the time series data and the feature quantity obtained by the principal component analysis.

In FIG. 7, PC _{1_1} and PC _{2_1} are feature amounts representing arrays (C ₁ , C ₂ ,..., C _k ) created from input time-series data. The same applies to the following sequences.

FIG. 15 is an example in which the above-described procedure by the feature amount analysis unit 430 is represented by time series data.
First, n partial time series (regions) are created by dividing time series data (time series data after aggregation by the data input / processing unit 420) by k elements from the top.
Next, principal component analysis is performed on each partial time series.
The concept of principal component analysis is shown in FIG.
As a result, two principal component scores were obtained per partial time series.
As an output of this step, the feature amount analysis unit 430 creates data shown in FIG. 8 describing the event occurrence time and the feature amount, and outputs the data to the protruding point determination unit 440.

The protruding point determination unit 440 inputs data as shown in FIG. 9, investigates the group of feature regions obtained by the feature amount analysis unit 430, and scores the region compared to other regions. In FIG. 9, serial numbers are given for convenience of explanation, but they are not in the actual data but are actually input in the same format as in FIG.
Specifically, the feature point survey by the protruding point determination unit 440 is performed by extracting the first feature amount and the second feature amount from the input from the feature amount analysis unit 430 and arranging them on a two-dimensional plane. As the arrangement method, for example, the first feature value is arranged at the Y-axis coordinate, and the second feature value is set as the X-axis coordinate.

FIG. 17 is a diagram in which input data (FIG. 9) from the feature amount analysis unit 430 is arranged in a two-dimensional feature amount space (principal component space).
It can be seen that among the feature quantities from serial numbers (a) to (f), the feature quantity of (c) deviates from the group.

Next, the protruding point determination unit 440 calculates the deviation from the group based on the distribution of the feature amount space (principal component space).
Here, the value indicating the deviation from the group is defined as the protrusion rate. The protrusion ratio takes a numerical value from 0 to 1 and indicates the degree of deviation from the center of gravity of the group.
There is a method for obtaining the population average as a method for obtaining the center of gravity of the group. The Mahalanobis generalized distance can be calculated to calculate the deviation from the group.
FIG. 10 shows output data of the protruding point determination unit 440. The protruding point determination unit 440 adds a protruding rate P corresponding to the event occurrence date and time in the time series data.

FIG. 18 shows a concept when setting the protrusion rate from the input data from the feature amount analysis unit 430.
When the feature quantity spatial distribution of each partial time series was investigated, it was found that the feature quantity in (c) was more dissimilar than the others. Therefore, the protruding point determination unit 440 intentionally sets the protruding rate higher than the other in the region (c) where the degree of deviation is large.

  Here, as a value indicating the degree of divergence in each region, the ratio of protrusion is used as a ratio. However, the ratio may not be a ratio as long as the degree of divergence can be indicated.

The smoothing coefficient calculation unit 450 calculates a coefficient to be used for the smoothing process with respect to information on each point of the time series data according to the score of the feature region in the protruding point determination unit 440. The parameters for initial setting are as follows.
Aggregation unit time: Unit time for aggregation of time series data to be observed

Data input from the protruding point determination unit 440 by the smoothing coefficient calculation unit 450 is as shown in FIG.
The portion of the area in FIG. 11 is for indicating the positional relationship of items, and does not exist in actual data. Therefore, the data input by the smoothing coefficient calculation unit 450 from the protruding point determination unit 440 is actually the same as in FIG. Moreover, the area | region of FIG. 11 has shown the partial time series demonstrated in FIG.

Further, as shown in FIG. 3, the smoothing coefficient calculation unit 450 inputs time series data per unit time from the data input / processing unit 420.
Data input from the data input / processing unit 420 by the smoothing coefficient calculation unit 450 is the same as that in FIG.
The smoothing coefficient calculation unit 450 sets the area allocated by the protruding point determination unit 440 for each piece of data in FIG. 6, as shown in FIG.
As described above, the smoothing coefficient calculation unit 450 sets the corresponding region for the data from the data input / processing unit 420 of FIG. 6, thereby giving the protrusion rate of the corresponding region to each data.

When the number of principal component target dimensions by the feature quantity analysis unit 430 is k, one area includes k time-series data. That is, in the example of FIG. 9, the region r ₁ includes k pieces of data from a pair of T ₁ and C _{1 to} a pair of T _k and C _k . Now, the projected rate for data i in case of a _{P i,} region _{r j (j = 1, ...} , m) smoothing factor for all data i of _M i (i = 1, ... , N) is as follows.

In Equation 1, max (r _j ) indicates a process for obtaining the maximum value in the event occurrence number C among the k pieces of data included in the region r _j .
Further, min (r _j ) indicates a process for acquiring the minimum value in the event occurrence count C among the k pieces of data included in the region r _j .
The third term on the right side, that is, (C _i −min (r _j )) / ((max (r _j ) −min (r _j ))) indicates that each event occurrence number C _i included in the region r _j This is a calculation as to how it is positioned in comparison with the number of other event occurrences included in the region r _j , and is a calculation for calculating the relative evaluation value of each event occurrence number C _i .
As described above, the smoothing coefficient M _i is a coefficient based on the relative evaluation value of each data and the protrusion rate of the region to which each data belongs, and as will be described later, the relative evaluation value and the protrusion rate, and the smoothing unit 460. There is a proportional relationship with the number of data that is the target of the moving average calculation.
The output data of the smoothing coefficient calculation unit 450 is as shown in FIG. 13, and the smoothing coefficient M _i is added to the data of FIG.

Next, the smoothing unit 460 smoothes the time series data based on the smoothing coefficient.
As shown in FIG. 3, the smoothing unit 460 inputs time series data per unit time from the data input / processing unit 420.
Data input from the data input / processing unit 420 by the smoothing unit 460 is the same as in FIG.
Then, the smoothing unit 460 smoothes the time series data per unit time from the data input / processing unit 420 by using the output data (FIG. 13) from the smoothing coefficient calculation unit 450.
For specific smoothing, a moving average method with variable sample values is used.
The following is the definition of the moving average method. Now, suppose that there are q time series before and after the following x _i at the center.

  The formula of the moving average calculation performed by the smoothing unit 460 according to the present embodiment is as follows.

In Equation 3, y _i indicates the value of the number of event occurrences C _i after smoothing.
In Equation 3, x _i means the number of event occurrences C _i .
Also referred to _{m i} and the moving average value. Moving average value m _i indicates the number of data to be subjected to the moving average calculation by the smoothing unit 460.
Moving average value _{m i} is the smoothing coefficient _{M i} and equivalence.
Normally, the moving average value is constant (in the above equation 2, q is fixed), but in this embodiment, the moving average value _mi is the smoothing coefficient M _i calculated by the smoothing coefficient calculation unit 450. It is linked to.
That is, the number of data to be subjected to the moving average calculation by the smoothing unit 460 varies depending on the value of the smoothing coefficient M _i .
The greater the smoothing coefficient M _i, in other words, if at least one of large projecting ratio P _i relative evaluation value and each of the event occurrence count C _i of each of the event occurrence count C _i, the moving average calculation The number of target data increases, and as a result, the value of many event occurrences is reflected and the degree of smoothing increases. Since the data having a large smoothing coefficient M _i is data that tends to be prominent compared to the preceding or subsequent data or region, the data is leveled by increasing the degree of smoothing.

As described above, in this embodiment, the value of each point in the steady region is obtained by a moving average. At that time, the value of the moving average value m _i is determined based on the smoothing factor M _i.
When a certain point in the stationary region is a region included in the group of feature amount spaces, the moving average value _mi is small, so the original information is retained. That is, the value does not change greatly.
In the case that a time series data is a region not included in the group of the feature space, the moving average value m _i increases, smoothes the protruding information.

FIG. 19 shows the concept of smoothing time-series data.
In the steps so far, the smoothing coefficient for each region is set so that the moving average value increases as the time-series value increases. In other words, noise removal works more strongly in areas where the values are more prominent. In FIG. 19, the region (b) is strong and smoothed.

  The time-series data after the smoothing process is performed as learning data, and the abnormality detection unit 4 performs abnormality detection using the learning data by means not shown in FIG.

  As described above, in the present embodiment, by performing principal component analysis and partial moving average processing on time series data in which noise information is mixed, it is effective for network abnormality detection processing with learning data. The series data can be smoothed.

In the above description, two types of feature amounts are used in the feature amount analysis. However, the number of feature amounts is not limited to two, and may be one or three or more.
In the above description, in order to speed up the calculation, the time series data is divided into regions, the feature amount is calculated for each region, and the protrusion rate of each region is determined based on the feature amount for each region. The protrusion rate may be determined for each piece of data based on the data value of each piece of data.

  In this embodiment, the data input / processing means for storing the number of time series data aggregated per unit time, and the principal component score is calculated from the time series data aggregated by the data input / processing means. Principal component analysis means for converting the score into a time series, a region editing processing means for collecting the principal component score time series obtained by the principal component analysis means into a group of feature areas from a head in a certain number, and the area editing A group of feature areas obtained by the processing means is investigated, and a projection point determination means for scoring the area in comparison with other areas, and parameters for smoothing are determined according to the score of the feature area in the projection point determination means. An abnormality detection unit (data processing apparatus) having a smoothing coefficient determination unit and a smoothing unit that performs smoothing of time-series data in accordance with parameters in the smoothing coefficient determination unit has been described.

Embodiment 2. FIG.
As a form of smoothing time-series data, it is possible to deal with time-series data with large fluctuations.
As shown in the upper part of FIG. 20, when the tendency of the time series data changes between the first half and the second half, the feature amount space (principal component space) obtained as a result of the principal component analysis is roughly divided into two groups.
In the example of FIG. 20, the regions (a) to (c) form one group, and the regions (e) to (f) form another group.
In such a case, the protrusion rate cannot be accurately determined, and it is difficult to determine the smoothing coefficient of the time series data.

In the present embodiment, the following function is added to the protruding point determination unit 440 in order to solve this problem.
The configuration of the abnormality detection unit 4 according to the present embodiment is the same as that shown in FIG. 3, and the process of each element is the same as that of the first embodiment except for the protruding point determination unit 440.

As shown in the previous stage of FIG. 20, the protruding point determination unit 440 according to the first embodiment targets all input data when calculating the protruding rate of time-series data. As a result, the regions (a) to (c) form one group, and the regions (e) to (f) form a main component space that forms another group.
In the present embodiment, the protruding point determination unit 440 responds to fluctuations in time-series data by calculating the protruding rate using only neighboring regions.

The lower part of FIG. 20 is a conceptual diagram illustrating a protrusion rate determination method of the protrusion point determination unit 440 according to the second embodiment.
When calculating the protrusion ratio regarding the region of the time series data (a), the regions (z) to (b) on both sides adjacent to each other are used as a basis. That is, the feature amounts of the regions (z) to (b) are arranged in the feature amount space, and the feature amounts are compared to determine the protrusion rate.
Similarly, calculation of the protrusion ratio regarding the region (b) uses the regions (a) to (c).

  The distinction between the protrusion rate determination method according to the first embodiment (upper part of FIG. 20) and the protrusion rate determination method according to the second embodiment (lower part of FIG. 20) is, for example, a constant amount of time-series data. If the amount is equal to or greater than the amount, the method according to the second embodiment can be considered to be operated according to the method according to the first embodiment if the amount is less than a certain amount.

As described above, in the present embodiment, the protruding point determination unit 440 performs the determination of each region (group) based on the relationship between the feature amount of each region (group) and the feature amount of an arbitrary number of regions close to each region. Set the protrusion rate (deviation value).
In the above description, only the regions adjacent to both sides of the region where the protrusion ratio is calculated are used, but not limited to both sides, any number of adjacent regions can be used. For example, the front and rear five areas may be used, or only the preceding five areas may be used.

  As described above, according to the present embodiment, when the time series data to be smoothed is fluctuating, the time series area used for obtaining the protrusion ratio is narrowed, thereby affecting the influence of the fluctuation of the time series data. Can be reduced. This makes it possible to detect a protruding portion in any part of the time series.

Embodiment 3 FIG.
As a derivation of the example shown in the second embodiment, the feature amount of time series data to be smoothed may be frequently distributed. In this case, depending on the number of feature amounts, when a group in the feature amount space is determined, the master-slave relationship of the group may be reversed and the protrusion rate may be erroneously determined.

FIG. 21 shows an example in which the protrusion rate is erroneously determined.
In FIG. 21, many regions ((a), (c) to (f)) protruding in the time series data appear.
When this data is subjected to principal component analysis and arranged in the feature amount space, the protruding region has a larger ratio as a group.
If the protrusion ratio is calculated from the size relationship of the areas constituting the group, the area deviating from the group is determined to be the area (b), and the relationship for calculating the protrusion ratio is reversed.
For this reason, the protrusion rate of the area (b) is high, and the protrusion ratios of the other areas are low, which is far from the actual situation.

In order to correct such a situation, the protruding point determination unit 440 according to the present embodiment grasps a specific value between regions.
When viewed from the distribution of the feature amount space, it can be seen that the regions (a) and (c) to (f) are the center.
However, when the average value of each region is obtained, it can be seen that the average value of the region (b) is smaller than the others, and the feature amount space in this case needs to obtain the deviation from the region (b).
As described above, the protrusion point determination unit 440 according to the present embodiment calculates the average value of the data values included in each region, and determines the protrusion rate in accordance with the actual situation.
The configuration of the abnormality detection unit 4 according to the present embodiment is the same as that shown in FIG. 3, and the process of each element is the same as that of the first embodiment except for the protruding point determination unit 440.

  As described above, the protruding point determination unit according to the present embodiment calculates the average value of the data in the region for each region (group), and calculates the average value of each region, the feature amount of each region, and the other region. Based on the relationship with the feature amount, the protrusion rate (deviation value) of each region is set.

  As described above, according to the present embodiment, when it is difficult to determine the protruding point of time series data viewed from the distribution of the feature amount space, it is possible to prevent erroneous determination by obtaining the average value of each region. .

Embodiment 4 FIG.
In the present embodiment, an example in which learning data is accumulated in the abnormality detection system will be described.
That is, in the present embodiment, learning data is smoothed prior to abnormality detection, the learning data after smoothing is accumulated, and learning data accumulated at the time of abnormality detection is used.

The left side of FIG. 22 shows an outline of the process flow in the conventional abnormality detection system.
The data obtained from the input data is analyzed by the abnormality detection system, and a warning is given if there is an abnormality.
Conventionally, in order to compare whether or not the input data is abnormal, a normal region from the input data is used as learning data. The learning data is newly defined every time the input data is updated. However, if the learning data is accumulated, the past results can be used and the detection accuracy is considered to be improved. However, since abnormal values are included in the input data, smoothing is performed using the procedure shown in the first embodiment before the learning data is accumulated.
The right side of FIG. 22 is an outline of the processing flow.

In the present embodiment, it is used for accumulation processing of learning data of the abnormality detection system.
Unlike normal smoothing, the characteristic part is preferentially smoothed, so that even when noise information is included in the input data, it can be used as learning data.

Embodiment 5. FIG.
In this embodiment, a method of using the time series information search system will be described.
FIG. 23 shows an outline of the processing flow of such a search system.
In the conventional system shown on the left side of FIG. 23, when time series information is input, the dictionary database refers to the dictionary data and selects a similar pattern.
However, since time-series information has various features, it is difficult to retrieve the same information.
In order to search for related similar information, a smoothing process is necessary. However, if the information is simply smoothed, the information of the input data is lost, and the search is difficult. If noise is mixed in the input data, the search cannot be performed as it is.
In this case as well, a technique for preferentially smoothing noise information as shown in the first embodiment is effective.

The right side of FIG. 23 shows an outline of the processing flow of the method according to the present embodiment.
The input data is not directly used as a search key, but is smoothed by the method shown in the first embodiment. This makes it possible to expand the search range and deal with noise.
That is, the data (detection pattern) to be collated with the dictionary data in the dictionary database is smoothed by the smoothing unit shown in the first embodiment, and the smoothed data (detection pattern) is stored in the dictionary database. Output to.
In the dictionary database, dictionary data matching the smoothed data (detection pattern) is searched, and the search result is returned.

In this embodiment, the smoothing process is used for the search process of the time-series information search system.
In this way, even if the noise information is removed, time series information other than the noise information is saved, so that the search accuracy can be improved.

  Finally, a hardware configuration example of the unauthorized access analysis system 100 and the abnormality detection unit 4 described in the first to fifth embodiments will be described.

  FIG. 25 is a diagram illustrating an example of hardware resources of the unauthorized access analysis system 100 and the abnormality detection unit 4 described in the first to fifth embodiments. The configuration of FIG. 25 is merely an example of the hardware configuration of the unauthorized access analysis system 100 and the abnormality detection unit 4, and the hardware configuration of the unauthorized access analysis system 100 and the abnormality detection unit 4 is described in FIG. It is not limited to this configuration, and other configurations may be used.

In FIG. 25, the unauthorized access analysis system 100 and the abnormality detection unit 4 include a CPU 911 (also referred to as a central processing unit, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, and a processor) that executes a program. . The CPU 911 is connected to, for example, a ROM (Read Only Memory) 913, a RAM (Random Access Memory) 914, a communication board 915, a display device 901, a keyboard 902, a mouse 903, and a magnetic disk device 920 via a bus 912. Control hardware devices. Further, the CPU 911 may be connected to an FDD 904 (Flexible Disk Drive), a compact disk device 905 (CDD), a printer device 906, and a scanner device 907. Further, instead of the magnetic disk device 920, a storage device such as an optical disk device or a memory card read / write device may be used.
The RAM 914 is an example of a volatile memory. The storage media of the ROM 913, the FDD 904, the CDD 905, and the magnetic disk device 920 are an example of a nonvolatile memory. These are examples of a storage device or a storage unit.
The communication board 915, the keyboard 902, the scanner device 907, the FDD 904, and the like are examples of an input unit and an input device.
Further, the communication board 915, the display device 901, the printer device 906, and the like are examples of an output unit and an output device.

The communication board 915 may be connected to a LAN (Local Area Network), the Internet, a WAN (Wide Area Network), etc., for example.
The magnetic disk device 920 stores an operating system 921 (OS), a window system 922, a program group 923, and a file group 924. The programs in the program group 923 are executed by the CPU 911, the operating system 921, and the window system 922.

The program group 923 stores programs for executing the functions described as “˜unit” and “˜means” in the description of the first to fifth embodiments. The program is read and executed by the CPU 911.
In the file group 924, in the description of the first to fifth embodiments, “determination of”, “calculation of”, “comparison of”, “evaluation of”, “determination of”, and “setting of” are set. ", Data, signal values, variable values, and parameters indicating the results of the processes described as""," totaling ", etc. are stored as items of" ~ file "and" ~ database ". The “˜file” and “˜database” are stored in a recording medium such as a disk or a memory. Information, data, signal values, variable values, and parameters stored in a storage medium such as a disk or memory are read out to the main memory or cache memory by the CPU 911 via a read / write circuit, and extracted, searched, referenced, compared, Used for CPU operations such as calculation, calculation, processing, editing, output, printing, and display. Information, data, signal values, variable values, and parameters are stored in the main memory, registers, cache memory, and buffers during the CPU operations of extraction, search, reference, comparison, calculation, processing, editing, output, printing, and display. It is temporarily stored in a memory or the like.
The arrows in the flowcharts described in the first to fifth embodiments mainly indicate input / output of data and signals. The data and signal values are the memory of the RAM 914, the flexible disk of the FDD904, the compact disk of the CDD905, and the magnetic disk device. It is recorded on a recording medium such as a 920 magnetic disk, other optical disks, minidisks, and DVDs. Data and signals are transmitted online via a bus 912, signal lines, cables, or other transmission media.

  Further, in the description of the first to fifth embodiments, what is described as “to part” and “to means” may be “to circuit”, “to device”, and “to device”. Also, “˜step”, “˜procedure”, and “˜processing” may be used. That is, what is described as “˜unit” and “˜means” may be realized by firmware stored in the ROM 913. Alternatively, it may be implemented only by software, or only by hardware such as elements, devices, substrates, and wirings, by a combination of software and hardware, or by a combination of firmware. Firmware and software are stored as programs in a recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD. The program is read by the CPU 911 and executed by the CPU 911. That is, the program causes the computer to function as “to part” and “to means” in the first to fifth embodiments. Alternatively, the computer executes the procedures and methods of “to part” and “to means” of the first embodiment shell.

  As described above, the unauthorized access analysis system 100 and the abnormality detection unit 4 shown in the first to fifth embodiments include a CPU as a processing device, a memory as a storage device, a magnetic disk, etc., a keyboard as an input device, a mouse, a communication board, etc. It is a computer equipped with a display device, a communication board, etc. as an output device, and implements the functions indicated as “to part” and “to means” by using these processing device, storage device, input device, and output device as described above. To do.

1 is a diagram illustrating a configuration example of an unauthorized access analysis system according to Embodiment 1. FIG. The figure which shows the relationship between the unauthorized access analysis system which concerns on Embodiment 1, and a monitoring object. FIG. 3 is a diagram illustrating a configuration example of an abnormality detection unit according to the first embodiment. The figure which shows the example of the input data before totaling of the data input and process part which concerns on Embodiment 1. FIG. The figure which shows the example of the input data after totaling of the data input and process part which concerns on Embodiment 1. FIG. FIG. 6 is a diagram illustrating an example of input data of a feature amount analysis unit according to the first embodiment. FIG. 6 is a diagram illustrating an example of a relationship between time series data and a feature amount in a feature amount analysis unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of an output format of a feature amount analysis unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of input data of a protruding point determination unit according to the first embodiment. FIG. 6 is a diagram illustrating an example of output data of a protruding point determination unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of input data from a protruding point determination unit of a smoothing coefficient calculation unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of input data from a data input / processing unit of a smoothing coefficient calculation unit according to the first embodiment. FIG. 6 is a diagram illustrating an example of output data of a smoothing coefficient calculation unit according to the first embodiment. FIG. 6 is a diagram showing a specific example of data totaling processing of the data input / processing unit according to the first embodiment. FIG. 10 is a diagram showing a specific example of the regionizing process of the feature amount analysis unit according to the first embodiment. FIG. 6 is a diagram illustrating a specific example of principal component analysis processing of a feature amount analysis unit according to the first embodiment. The figure which shows the specific example of the arrangement | positioning process to the principal component space of the protrusion point determination part which concerns on Embodiment 1. FIG. The figure which shows the specific example of the protrusion point determination process of the protrusion point determination part which concerns on Embodiment 1. FIG. FIG. 6 shows a specific example of the smoothing process of the smoothing unit according to the first embodiment. The figure which shows the specific example of the protrusion point determination process of the protrusion point determination part which concerns on Embodiment 2. FIG. FIG. 10 is a diagram illustrating a specific example of a protrusion point determination process of a protrusion point determination unit according to the third embodiment. FIG. 6 shows a method according to a fourth embodiment. FIG. 6 shows a method according to a fifth embodiment. FIG. 3 is a flowchart showing an operation example of an abnormality detection unit according to the first embodiment. The figure which shows the hardware structural example of the unauthorized access analysis system which concerns on Embodiment 1-5, and an abnormality detection part. The figure explaining a prior art.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 GUI, 2 Countermeasure part, 3 Unauthorized access determination part, 4 Abnormality detection part, 5 Log information totaling part, 6 Information collection part, 100 Unauthorized access analysis system, 410 Input data, 420 Data input and processing part, 430 Feature quantity analysis Part, 440 protrusion point determination part, 450 smoothing coefficient calculation part, 460 smoothing part.

Claims (13)

A divergence value setting unit that performs analysis on a plurality of data each having a data value, and sets, for each data, the degree of deviation of the data value of each data from the data value of other data, as a divergence value;
A smoothing coefficient calculating unit that calculates a smoothing coefficient for smoothing for each data, reflecting the deviation value of each data set by the deviation value setting unit;
A data processing apparatus comprising: a smoothing unit that smoothes each data using the smoothing coefficient of each data calculated by the smoothing coefficient calculating unit. The smoothing coefficient calculation unit includes:
The data processing apparatus according to claim 1, wherein a smoothing coefficient is calculated so that a degree of smoothing by the smoothing unit changes in accordance with a deviation value set by the deviation value setting unit. The deviation value setting unit
Analyzing a plurality of data arranged according to a predetermined order,
The smoothing unit
Perform smoothing by performing a moving average calculation using an arbitrary number of data on the data to be smoothed,
The smoothing coefficient calculation unit includes:
The smoothing coefficient in which the divergence value set by the divergence value setting unit and the number of data targeted for moving average calculation by the smoothing unit are proportional to each other is calculated. Data processing device. The data processing device further includes:
Feature quantity analysis that groups a plurality of data arranged according to a predetermined order into a plurality of groups according to the order, calculates a principal quantity of data values of data included in each group, and calculates a feature quantity of each group Part
The deviation value setting unit
For each group grouped by the feature quantity analysis unit, the degree of deviation of the feature quantity of each group from the feature quantity of the other group is set as a deviation value,
The smoothing coefficient calculation unit includes:
Calculate the relative evaluation value of the data value of each data in the group to which each data belongs, and reflect the relative evaluation value of each data and the deviation value of the group to which each data belongs to smooth each data The data processing apparatus according to claim 1, wherein a smoothing coefficient is calculated. The smoothing coefficient calculation unit includes:
The data processing apparatus according to claim 4, wherein a smoothing coefficient in which a relative evaluation value and a divergence value are proportional to a degree of smoothing by the smoothing unit is calculated. The smoothing unit
Perform smoothing by performing a moving average calculation using an arbitrary number of data on the data to be smoothed,
The number of data to be subjected to moving average calculation is determined according to the smoothing coefficient calculated by the smoothing coefficient calculation unit, and data is smoothed by performing moving average calculation using data of the determined number of data. The data processing apparatus according to claim 4, wherein: The deviation value setting unit
5. The data processing apparatus according to claim 4, wherein a divergence value of each group is set based on a relationship between a feature amount of each group and a feature amount of an arbitrary number of groups close to each group. The deviation value setting unit
For each group, calculate the average value of the data in the group, and set the divergence value of each group based on the average value of each group and the relationship between the feature value of each group and the feature value of other groups. 5. The data processing apparatus according to claim 4, wherein The data processing device includes:
A data processing device that generates learning data for detecting abnormality of predetermined inspection target data by smoothing data by the smoothing unit,
The data processing apparatus according to claim 1, wherein when the inspection target data is input, setting of a deviation value by the deviation value setting unit is started. The data processing device includes:
A data processing device that generates learning data for detecting abnormality of predetermined inspection target data by smoothing data by the smoothing unit,
2. The data processing apparatus according to claim 1, wherein smoothing of data by the smoothing unit is completed and generation of learning data is completed before inputting the inspection target data. The data processing device includes:
It is connected to a dictionary database that stores predetermined dictionary data,
2. The smoothing by the smoothing unit is performed on data to be collated with dictionary data in the dictionary database, and the smoothed data is output to the dictionary database. Data processing device. A divergence value setting step in which a computer analyzes a plurality of data each having a data value, and sets, for each data, the degree of deviation of the data value of each data from the data value of other data as a divergence value; ,
A smoothing coefficient calculating step in which a computer reflects a deviation value of each data set in the deviation value setting step and calculates a smoothing coefficient for smoothing for each data;
A data processing method, comprising: a smoothing step in which a computer smoothes each data using the smoothing coefficient of each data calculated in the smoothing coefficient calculating step. Analyzing a plurality of data each having a data value, and for each data, a divergence value setting process for setting the degree of divergence of the data value of each data from the data value of other data as a divergence value;
A smoothing coefficient calculation process for calculating a smoothing coefficient for smoothing for each data, reflecting a deviation value of each data set by the deviation value setting process;
A program for causing a computer to execute a smoothing process for smoothing each data using a smoothing coefficient of each data calculated by the smoothing coefficient calculating process.

Images