KR102090239B1 - Method for detecting anomality quickly by using layer convergence statistics information and system thereof - Google Patents

Abstract

According to an embodiment of the present invention, disclosed is a method for generating hierarchical convergence statistical information by using both global and regional statistical information, and detecting anomality of information output by a system at high speed based on the generated hierarchical convergence statistical information.

Description

High-speed anomaly detection method and system using hierarchical convergence statistics information {Method for detecting anomality quickly by using layer convergence statistics information and system thereof}

The present invention relates to a high-speed anomaly detection method using hierarchical convergence statistical information and a system thereof, and more specifically, to generate hierarchical convergence statistical information using both global and regional statistical information, and to generate hierarchical convergence statistics. It relates to a method and system for detecting anomalies of information output by a system at a high speed based on information.

For administrators who manage databases of large-scale numerical information, it is important to quickly find outliers that exhibit anomalies. This is because the faster the database detects the outliers, the easier it is to recover and modify the system.

As a methodology for finding outliers, there are statistical methods, machine learning methods, and methods using neural networks, but there is no method to guarantee superior performance in relation to accuracy and cost.

As an example, the method based on statistical information has a problem that the coverage of ideality is difficult to balance between global and regional, and it is also difficult to control the level of accuracy. The statistical method uses a statistical metric measurement formula to model the distribution of data, and accordingly determines how regular or irregular each data point is, and the ideality according to the distribution can be determined using Zscore. It is pointed out that the statistical method does not consider the occurrence context in the time or space of the event at all, and in particular, the outliers that have low frequency of occurrence but are actually only noise in the data showing abnormality, but are not significantly related to the ideality ( outlier) is also included, and according to statistical methods, it is difficult to distinguish such outliers from meaningful ideality data.

As another example, neural network-based technology can obtain the most sophisticated and flexible accuracy when it is elaborately designed, but it can act as a stumbling block in high cost. In particular, neural network-based technology is used for very large databases. Applying it can be considered unrealistic due to the enormous amount of computation and cost. As an example of a method using a neural network, an autoencoder or the like is used, but the method based on the neural network is capable of sufficiently fast data processing when applied to an application step after modeling is completed. When modeling is not completed in a situation where there are millions of given data, it is impossible to output results within a few seconds.

As another example, in the method of finding outliers using the machine learning method, the time of training is essential before training the model and producing the results, and the time delay required is necessary to process the result in real time for large data. There is a problem that cannot be. The machine learning method attempts a semantic analysis in consideration of the context in which an event showing an abnormality in the system occurs, and typically includes a support vector machine (SVM) and a principal component analysis (PCA).

In other words, it was almost impossible to quickly determine whether or not a database that outputs numerical data on a large scale outputs ideality data using only a known methodology, thus minimizing computational complexity and quickly finding outliers while guaranteeing accuracy over a certain level. It is necessary to introduce a methodology that can be issued.

Republic of Korea Registered Patent No. 10-1964412 (announced April 1, 2019)

The technical problem to be solved by the present invention is to provide a method for determining the ideality of database data and a system for implementing the method, which utilize statistical information, and which can derive very fast results through an algorithmic device. have.

The method according to an embodiment of the present invention for solving the above technical problem, receives raw data including a plurality of data from a database, and calculates overall statistics of the data included in the raw data Total statistics calculation step; A first group statistics calculating step of dividing the raw data into a first layer cluster including data of a predetermined division constant, and calculating a first group statistics for each of the divided first layer clusters; A first hierarchical statistical calculation step of calculating hierarchical convergence statistics for the first layer based on the calculated overall statistics and the calculated first cluster statistics; A second group statistics calculation step of dividing the divided first layer cluster into a second layer cluster including the data of the division constant, and calculating a second group statistics for each of the divided second layer clusters; A second layer statistics calculation step of calculating a layer convergence statistics for the second layer based on the hierarchical convergence statistics and the calculated second group statistics; A statistical value iterative calculation step of repeating the second group statistics calculation step and the second layer statistics calculation step until a predetermined condition is satisfied; And an abnormality determination step of determining anomalies of data belonging to the last layer cluster of the last layer based on the last layer convergence statistics calculated last in the statistical value iterative calculation step.

The system according to another embodiment of the present invention for solving the above technical problem, calculates overall statistics for receiving raw data including a plurality of data from a database and calculating overall statistics of the data included in the raw data part; A first group statistics calculation unit for dividing the raw data into a first layer cluster including data of a predetermined division constant, and calculating a first group statistics for each of the divided first layer clusters; A first hierarchical statistical calculation unit for calculating hierarchical convergence statistics for the first layer based on the calculated overall statistics and the calculated first group statistics; A second group statistics calculation unit for dividing the divided first layer cluster into a second layer cluster including the data set of the division constant, and calculating a second group statistics for each of the divided second layer clusters; A second layer statistics calculation unit calculating a layer convergence statistics for the second layer based on the hierarchical convergence statistics and the calculated second group statistics; A statistical value iterative calculation unit that repeats calculating cluster statistics and hierarchical convergence statistics until a predetermined condition is satisfied; And an ideality determination unit for determining anomalies of data belonging to the last layer cluster of the last layer based on the last layer convergence statistics calculated last in the statistical value repeat calculation unit.

One embodiment of the present invention can provide a computer-readable recording medium storing a program for executing the method.

The present invention has a feature that, while using a statistical method, and simultaneously fusion of global information and regional information, data representing ideality can be output as a result value at a very high speed. The above characteristics become distinct from those in which the existing statistical method reflects only global characteristics and outputs the result values. Due to the characteristics of the algorithm, the result values based on model learning and inference are currently distributed on a typical server. It is distinguished from neural network techniques or machine learning techniques that cannot be output within seconds.

1 is a block diagram showing an example of a system according to the present invention.
2 is a diagram schematically showing an example of a statistical value iterative calculation unit implemented by a recursive function.
3 is a view for explaining hierarchical convergence statistics and hierarchical cluster statistics.
4 is a diagram illustrating an example of a process for outputting a data set indicating ideality.
5 is a flowchart of an example of a high-speed anomaly detection method using hierarchical convergence statistical information according to the present invention.

The terminology used in the embodiments has been selected for general terms that are currently widely used while considering functions in the present invention, but this may vary depending on the intention or precedent of a person skilled in the art or the appearance of new technologies. In addition, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, their meanings will be described in detail in the description of the applicable invention. Therefore, the terms used in the present invention should be defined based on the meanings of the terms and the contents of the present invention, not simply the names of the terms.

When a part of the specification "includes" a certain component, this means that other components may be further included instead of excluding other components unless specifically stated otherwise. In addition, terms such as “… unit” and “… module” described in the specification mean a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a block diagram showing an example of a system according to the present invention.

Referring to FIG. 1, the system 100 according to the present invention includes a total statistics calculation unit 110, a first group statistics calculation unit 120, a first layer statistics calculation unit 130, and a second group statistics calculation unit ( 140), the second layer statistical calculation unit 150, the statistical value iterative calculation unit 160, it can be seen that includes the abnormality determination unit 170 and the abnormal number receiving unit 180. Depending on the embodiment, the abnormality receiving unit 180 may be omitted from the system 100.

The overall statistics calculating unit 110 receives raw data including a plurality of data sets from a database, and calculates overall statistics of data included in the raw data.

Depending on the embodiment, the database may be included in the system 100 according to the present invention or may be omitted. When the database is omitted from the system 100 according to the present invention, the overall statistics calculation unit 110 may receive raw data from the database through a wired cable or wireless communication method. The database stores raw data, and raw data refers to various numerical information itself organized according to unique standards defined in the database.

The raw data received from the database may be log data of a specific system. In addition, the raw data may be data stored by binary or hexadecimalization of data consisting of numbers or characters. The raw data includes a plurality of data, and each data refers to data output from different viewpoints or different modules. Each data may include not only essential data that is meaningful for actually calculating statistics, but also metadata for distinguishing different data sets or times when data was recorded. As a concept including metadata, the term data set can be used.

Table 1 shows an example of row data including a total of eight data sets. Referring to Table 1, the raw data includes 8 data sets, and each data set is numerical data output at the same time in a total of 8 different modules, or at 8 different time points in one module. It may be outputted numerical data.

The overall statistics calculating unit 110 receives the raw data, classifies and identifies the data set included in the raw data, and then calculates the overall statistics of the data. Here, the statistical value calculated by the total statistical calculation unit 110 based on the data may be at least one of an average, a variance, a standard deviation, and a z-score (Zscore). According to an embodiment, the statistics calculated by the overall statistics calculating unit 110 may be new statistical values calculated by a combination of the above-described statistical values.

The first group statistics calculation unit 120 divides the raw data into a first group cluster including data of a predetermined division constant, and calculates the first group statistics for each divided first group cluster.

The division constant is a value referenced by the first group statistics calculation unit 120 in order to divide the raw data into a plurality of data (or data sets), and is preset in the first group statistics calculation unit 120. For example, if the division constant is 2, the first group statistics calculation unit 120 divides the raw data into two first layer clusters. As another example, if the division constant is 2 and the data (or data set) included in the raw data is eight as shown in Table 1, the first cluster statistics calculator 120 generates a total of two first layer clusters. The two first hierarchical clusters may be a group including four data (or data sets), respectively. Here, the term “first layer” refers to a name for distinguishing the second layer, the third layer, and the like, which will be described later, and both clusters generated by the first group statistics calculation unit 120 are first layer clusters.

The first cluster statistics calculation unit 120 divides the data included in the raw data into the first layer clusters, and then calculates the first group statistics for each first layer cluster. The first group statistics mean statistical values calculated based on data belonging to each of the first layer clusters, and are calculated through the same method as the total statistics calculation unit 110 calculates the total statistics. As an example, 1, 2, 3, and 4 are included in the raw data, and the overall statistics are averaged, 2.5 is calculated, and the first hierarchical cluster is divided into {1, 2}, and {3, 4}, respectively. Assuming, the first group statistics can be calculated to be 1.5 and 3.5, respectively. Here, 1.5 and 3.5 are different values, but both are statistical values of clusters belonging to the first layer, and may be collectively referred to as a first cluster statistics in this specification. Also, the second group statistics and the third group statistics, which will be described later, may be understood in the same context as above.

The first hierarchical statistical calculation unit 130 is based on the overall statistics calculated by the overall statistical calculation unit 110 and the first group statistical calculation calculated by the first group statistical calculation unit 120, and the first hierarchical convergence statistics are calculated. Calculate. Here, the first layer convergence statistics refers to a value obtained by combining statistics of the first layer clusters in the first layer and statistics accumulated in the upper layer than the first layer.

The reason for obtaining the first layer convergence statistics in the present invention is to solve the problems of the conventional statistical methods, neural network techniques, and methods using machine learning.

First, the disadvantage of the statistical method was that it was difficult to reflect regional information because only global information was reflected.However, the method using neural network techniques and machine learning can learn regional regional patterns, but it can be used for large-scale data (data sets). There was a problem in that it was not possible to calculate a result value having a certain degree of accuracy within seconds. However, in order to solve the above problems, if statistical methods are used, but simply local information is infinitely extended, it is impossible to detect data with abnormality within a short time even with statistical methods.

The present invention uses a statistical method to some extent, and compares with global information as well as regional information, and calculates a statistical value and utilizes the calculated statistical value in comparison with a purely statistical method. It is characterized by detecting a data set indicating abnormality within seconds. The above features are regional information, primarily calculating cluster statistics for each layer, and as a principle of a recursive function, repeatedly calculating hierarchical convergence statistics by considering statistical values accumulated in a higher layer than the current layer. Thus, it may be implemented as a step of identifying data having the highest ideality through comparison between a data set belonging to a cluster in the final layer and the final layer convergence statistics. Detailed description of the above-described process will be described later with reference to FIGS. 2 to 5.

Next, the description of FIG. 1 will be continued.

The second cluster statistics calculating unit 140 divides the first divided first cluster into a second group cluster including a data set of division constants, and calculates a second group statistics for each second group cluster.

Here, the division constant referenced by the second group statistics calculation unit 140 to divide the first layer clusters is the same as the division constant referenced when dividing the raw data. For example, if the raw data is divided into four first layer clusters, each of the four first layer clusters may be further divided into four second layer clusters, and the total number of clusters belonging to the second layer is 16. . Each cluster contains the same number of data sets if the number of hierarchies created is the same. Subsequently, the second cluster statistics calculating unit 140 calculates statistical values of each of the second clusters belonging to the second layer, and these statistical values may be referred to as second cluster statistics.

The second hierarchical statistical calculation unit 150 is based on the hierarchical convergence statistics for the first hierarchical layer calculated by the first hierarchical statistical calculation unit 130 and the second group statistical statistics calculated by the second group statistical calculation unit 140. By doing so, the hierarchical convergence statistics for the second layer are calculated. If the hierarchical convergence statistics for the first layer are information incorporating statistical values for the reference layer and the first layer in which the raw data exist, the hierarchical convergence statistics for the second layer are the reference layer, the first layer, and the second layer. Refers to information that incorporates statistical values.

The statistical value repetition calculation unit 160 generates a hierarchical cluster until a predetermined condition is satisfied, calculates cluster statistics for the generated hierarchical cluster, and fuses the calculated cluster statistics with statistics of a higher layer to obtain hierarchical convergence statistics. Repeat the calculation.

For example, the statistical value iterative calculation unit 160 divides the second layer cluster into third layer clusters according to a preset division constant, calculates third group statistics for each third layer cluster, and By repeatedly calculating the hierarchical convergence statistics for the third layer based on the hierarchical convergence statistics for the second layer and the third group statistics, it is also possible to calculate the hierarchical convergence statistics for the 100th layer.

The statistical value repetition calculation unit 160 is logically implemented instead of a physical configuration, and the second group statistics calculation unit 140 and the second layer statistics calculation unit 150 are repeatedly operated as shown in FIG. 1. It can be implemented in such a way as to control. In this case, the second group statistics calculation unit 140 and the second layer statistics calculation unit 150 first calculate the hierarchical convergence statistics for the second group statistics and the second layer, and subsequently the third group. Cluster statistics and hierarchical convergence statistics for the third layer are performed, and data values necessary for calculating the result may be temporarily stored in a high-speed buffer and used in a manner.

The condition that the statistical value repeat calculation unit 160 stops the iterative process is preset in the statistical value repeat calculation unit 160.

As an example, the statistical value iterative calculation unit 160 may repeat hierarchical division and statistical value calculation until a data set belonging to a cluster of a specific layer is no longer divided. In other words, if only one data set belongs to the cluster, the statistical value iterative calculation unit 160 may consider the preset condition to be satisfied, and terminate the hierarchical division and statistical value calculation process.

As another example, the statistical value repetition calculation unit 160 may divide a layer when the number of data sets belonging to a cluster of the current layer is smaller than the number of preset data sets, or when it is equal to the number of preset data sets. And the statistical value calculation process. According to the present exemplary embodiment, if the number of data sets preset in the statistic repetition calculation unit 160 is 4, and the number of data sets belonging to each cluster in the current layer is equal to 4 or less than 4 , It is regarded that the preset condition is satisfied, and the statistical value repetition calculation unit 160 may end the process of hierarchical division and statistical value calculation.

As an optional embodiment, the statistical value repetition calculator 160 may be implemented as a recursive function based on the repeated operation. The statistic repetition calculation unit 160 implemented as a recursive function may be determined in a specific operation with a script written in a programming language, and various function parameters may be employed to operate the recursive function in this process.

2 is a diagram schematically showing an example of a statistical value iterative calculation unit implemented by a recursive function.

When the statistical value iterative calculation unit 160 is implemented with a recursive function as shown in FIG. 2, the input file name 210, the N value 220, the reference σ value 230, the number of samples 240, the global ratio 250, The regional ratio 260 may be requested as a function parameter. The statistical value repetition calculation unit 160 may be implemented as a script of repeatedly executing an execution file several times according to conditions.

First, the input file name 210 means the name of the file when raw data is converted into a file.

The N value 220 means the total number of data sets included in the raw data. According to FIG. 2, 300,000 data sets exist in raw data.

The reference σ value 230 is an initial reference value referred to determine whether an abnormality is present, and may be expressed as a multiple of σ preset according to an embodiment.

The sample number 240 is data selected for use in calculating a statistical value among data sets belonging to a cluster when calculating cluster statistics for each layer cluster and hierarchical convergence statistics for each layer in the recursive function. It means the number of three. For example, if 10 data sets exist in the second layer cluster, and the number of samples is 6, the statistical value of the second layer cluster includes six randomly selected data sets among the data sets belonging to the second layer cluster. Can be calculated on a basis. The smaller the number of samples 240, the greater the operating speed of the overall system 100 increases, and when the number of samples 240 is not specified as a specific value, the statistical value repeat calculation unit 160 performs data without sampling. The statistical values for the entire set are calculated.

The global ratio 250 and the regional ratio 260 are ratios in which the recursive function fuses the statistical values received from the upper level with the statistical values in the current layer, and should be 1.0. Here, the statistical value received from the upper level means hierarchical convergence statistics for the previous layer, and the statistical value at the current level means cluster statistics for the current layer. In addition, the manager of the system 100 may assign a higher weight to either the global consolidation ratio 250 or the regional ratio 260, either the convergence statistics for the immediately preceding layer or the statistical values in the current layer. The administrator of the system 100 of the present invention can individually obtain a result focused on global information or a result focused on regional information through weight adjustment as described above.

The recursive function shown in FIG. 2 is newly applied each time a cluster of each layer is divided by a division constant. In particular, when the number of samples 240 is designated as a constant value, statistical value calculation based on sampling is performed, and in this process, the statistical value repeat calculation unit 160 may randomly sample, but may decrease the computational speed. To prevent this, sampling may be performed using an equi distance method.

As described above, as the statistical repetition calculation unit 160 repeatedly accumulates statistics, as the layer deepens, global information gradually has less global characteristics, and eventually problems with traditional statistical methods (reflection of regional information) This difficulty) can be minimized. In addition, by adjusting the number of samples 240, the global ratio 250, and the regional ratio 260 in different ways, the administrator can control the overall processing speed and accuracy of the system 100 at his will.

3 is a view for explaining hierarchical convergence statistics and hierarchical cluster statistics.

3 may be implemented by the system 100 according to the present invention, it will be described with reference to FIG. 1, and descriptions overlapping with those described in FIGS. 1 and 2 will be omitted. In addition, although the division constant is set to 2 in FIG. 3 for convenience, it will be apparent to those skilled in the art that the number may be an integer greater than 2 according to an embodiment.

Referring to FIG. 3, a total of four layers are illustrated, and it can be seen that the data set included in the raw data is a total of eight sets from ds1 to ds8. In FIG. 3, the cluster statistics in the reference layer is G1, and G1 is a value determined by one or a combination of at least two or more of the mean, variance, standard deviation, and zscore for the data sets ds1 to ds8. The value calculated by) has already been described.

The first cluster statistical calculation unit 120 divides the raw data into two, which is a division constant, so that four data sets are included in the first hierarchy cluster. Referring to FIG. 3, it can be seen that two first layer clusters were generated according to the division constant, and the first group statistics of the two first layer clusters are G2 and G3, respectively.

The first hierarchical statistical calculation unit 130 fuses the first cluster statistics G2 and G3 of the two first hierarchical clusters with the previously calculated cluster statistics G1 to calculate hierarchical convergence statistics S1 for the first layer. Hierarchical convergence statistics S1 for the first layer may be set as a function value having G1, G2, and G3 as parameters, and the first hierarchical statistical calculation unit 130 calculates the hierarchical convergence statistics S1, and the preset global ratio and Since the regional ratio can be considered, it has already been described through FIG. 2 that various S1 values can be calculated.

The second cluster statistics calculating unit 140 divides the two first layer clusters generated in the first layer into a second layer cluster based on the division constant, and calculates the second group statistics. Referring to FIG. 3, a total of four second layer clusters were generated according to the number and division constant of the first layer cluster, and it can be seen that the second group statistics of the four second layer clusters are G4 to G7, respectively. In FIG. 3, for convenience of description, the order of the data sets is not mixed, but it is described as being included in the cluster. However, it is apparent that the data set may be randomly included in each layer cluster according to an embodiment.

The second layer statistics calculating unit 150 fuses the second group statistics G4 to G7 of the four second layer clusters with the previously calculated layer convergence statistics S1 to calculate the layer convergence statistics S2 for the second layer. Hierarchical convergence statistics S2 for the second layer may be set as a function value having S1, G4, G5, G6, and G7 as parameters, and the second hierarchical statistical calculation unit 150 calculates hierarchical convergence statistics S2 in advance. Since the set global ratio and the regional ratio can be considered, it has already been described in FIG. 2 that various S2 values can be calculated.

The statistical value repetition calculation unit 160 repeats the hierarchical division process and the hierarchical convergence statistics calculation process until a predetermined condition is satisfied, and FIG. 3 shows the third layer to be the final layer (last layer). In FIG. 3, for convenience, the third layer is set as the final layer, but at least one of the number of data sets included in the raw data, the division constant for dividing the hierarchical cluster, or the conditions previously set in the statistical value repeat calculation unit 160. According to the above, the process of hierarchical division and the process of calculating the convergence of statistics by the statistical value repetition calculation unit 160 may be increased or decreased.

The statistical value iterative calculation unit 160 determines that no further layer division can be performed as the data set is included in each of the third layer clusters in the third layer, and determines the last layer calculated by the ideality determination unit 170 Convergence statistics S2 is delivered.

Next, the configuration for FIG. 1 will be continued.

The ideality determining unit 170 determines the ideality of the data sets belonging to the last layer cluster of the last layer based on the last layer convergence statistics calculated by the statistical value repeat calculation unit 160 last.

Through the process in which the statistical value iterative calculation unit 160 repeatedly operates, clusters in the first layer are divided into clusters of the n-th layer (where n is an integer greater than 1), and n-1 from the reference layer. Hierarchical convergence statistics that combine statistical values up to the hierarchies can be calculated. Here, the last layer convergence statistics are hierarchical convergence statistics that combine the statistical values from the reference layer to the n-1 layer, and the last layer is the nth layer.

The ideality determining unit 170 receives the last layer convergence statistics from the statistical repetition calculation unit 160 and uses the received last layer convergence statistics to determine the ideality of the data set belonging to the last layer cluster. As an example, if the last layer convergence statistics is a value related to an average, a value having the greatest deviation from the average among the data sets belonging to the last layer cluster may be determined as the most ideal data set, and the last layer convergence statistics is the standard deviation. If it is a value of, a data set indicating the largest standard deviation among the data sets belonging to the last layer cluster may be selected by the ideality determining unit 170.

When interpreting the above results, the ideality determination unit 170 compares the data sets belonging to the cluster of the final layer and the final layer convergence statistics, which are global and regional statistical values, and the data sets belonging to the cluster of the final layer are You can see how to judge how ideal you are. If there is no preset value, the ideality determination unit 170 may specify only one data set showing the highest ideality, but according to an embodiment, the plurality of data sets may be specified in order of high ideality.

Table 2 shows an example of the results determined by the ideality determining unit 170 as a table.

In Table 2, Line Number is the order of raw data, position in the array is where the data set with the highest ideality is found, Data Value is the value of the data set with the highest ideality, and Zscore is the standard of the data set with the highest ideality. The standard score, Sampling Average is the average of the sampled data set values, Sampling Std. Is the standard deviation of the sampled data set values, and Depth is the depth value of the function call, which means the number of the last layer.

When analyzing the first row of Table 2, when analyzing the anomaly using the 72nd row data as input data, the data set located at 222520, which is in the 19th layer and has a value of 97, has the highest ideality. It was judged that, as a result of sampling to accelerate data processing, the average of the sampled data sets is 54.513008 and the standard deviation is 23.187880. In addition, in Table 2, as information about each data set, through the existence of Zscore in addition to the mean and standard deviation, it can be seen that statistical values calculated in various ways were used in each layer to determine the ideality of the data set. .

Next, the description of the configuration of FIG. 1 will be continued.

The abnormal number receiving unit 180 receives the target abnormal number from the user, and transmits it to the abnormality determination unit 170. The ideality determination unit 170 may sequentially extract the datasets having the highest ideality among the datasets belonging to the cluster of the last layer as many as the target abnormality number. The abnormal number receiving unit 180 may be omitted from the system 100 according to an embodiment.

4 is a diagram illustrating an example of a process for outputting a data set indicating ideality.

FIG. 4 shows a method in which the ideality determination unit 170 determines the data set for which the ideality is determined to be the highest through the same process as in FIG. 3 and then searches for the highest ideality value for each layer in reverse order.

First, in the final layer, each hierarchical cluster contains only one data set, and in step S410, the second hierarchical layer includes two data sets for each hierarchical cluster. After comparing and determining the abnormality between data sets belonging to the second layer cluster, the ideality determining unit 170 detects only a data set showing high abnormality. According to FIG. 4, ds1, ds4, ds5, and ds8 are ds2, ds3, It can be seen that each exhibits higher ideality than ds6 and ds7.

Subsequently, when step S430 is performed, the first layer includes four data sets for each layer cluster. The ideality determining unit 170 compares and determines the abnormality between data sets belonging to the first layer cluster, and then detects only the data set showing the highest ideality. According to FIG. 4, ds4 and ds5 are higher than ds1 and ds8, respectively. It can be seen that indicates.

Finally, through step S450, the data set included in the raw data becomes a reference layer forming a cluster, and the ideality determination unit 170 is a data set that has the highest ideality in the first layer. By comparing and analyzing ds4 and ds5, ds5 having the highest ideality can be detected.

In addition, by synthesizing the above process, the ideality determination unit 170 may detect a plurality of data sets representing the highest ideality by arranging the data sets showing the highest ideality for each layer and cluster. As an example, referring to FIG. 4, the ideality determining unit 170 determines ds5 as a data set showing the greatest ideality based on a result of comparing ds4 and ds5 in the first layer, and then ds4 , The ideality of the data set can be sorted in descending order by ds1 (ds8). Here, when the abnormality number receiving unit 180 receives 3 from the user, the system 100 may output ds5, ds4, and ds1 (ds8) as output values, and additionally through comparison of the ideality of ds1 and ds8, further You can also print out the exact results.

The process described in FIG. 4 is an example of a method of sorting a data set when the data set showing the highest ideality in FIG. 3 is identified. It is common knowledge in this field that there may be other examples in addition to the method described in FIG. 4. It will be obvious to those who have it.

5 is a flowchart of an example of a high-speed anomaly detection method using hierarchical convergence statistical information according to the present invention.

Since the method according to FIG. 5 can be implemented by the system according to FIG. 1, hereinafter, descriptions overlapping with those described in FIG. 1 will be omitted and will be described with reference to FIG. 1.

The overall statistics calculating unit 110 receives raw data from the database and calculates overall statistics of the data set (S510).

The first cluster statistics calculating unit 120 divides the raw data into a first layer cluster, and calculates a first group statistics for each first layer cluster (S520).

The first hierarchical statistical calculation unit 130 calculates hierarchical convergence statistics for the first hierarchy based on the overall statistics and the first cluster statistics (S530).

The second group statistics calculation unit 140 divides the first layer clusters into the second layer clusters and calculates the second group statistics for each second group cluster (S540). It has been already explained that the division constants used to divide the clusters in steps S520 and S540 remain the same, and the division constants can be arbitrarily adjusted for accuracy or processing speed.

The second layer statistics calculating unit 150 calculates the layer convergence statistics for the second layer based on the layer convergence statistics for the first layer and the second group statistics (S550).

The statistical value iterative calculation unit 160 repeatedly performs cluster division and hierarchical statistical calculation until a predetermined condition is satisfied (S560). When the predetermined condition is satisfied (S570), the statistical value iterative calculation unit 160 determines the ideality of the datasets belonging to the last layer cluster based on the last layer convergence statistics calculated in step S560 (S580).

In step S580, the process of determining and arranging the ideality of the data sets belonging to the last hierarchical cluster by the ideality determining unit 170 has been described with reference to FIGS. 3 to 4.

The present invention has a feature that, while using a statistical method, and simultaneously fusion of global information and regional information, data representing ideality can be output as a result value at a very high speed. This is a feature that distinguishes the existing statistical method from outputting the result value by reflecting only the global characteristics, and is also distinguished from the neural network technique or machine learning technique, which cannot output the result value within seconds due to the characteristics of the algorithm.

When implemented by the inventor script, the result was output within 3 to 5 seconds at the latest for 10 million data sets, and the partitioning constant (which is the basis of sampling rate, cluster division, and ideality determination in the present invention) When the reference σ value) is adjusted, a more useful and suitable result value can be output to a user who manages and operates the system 100.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program can be recorded on a computer-readable medium. At this time, the medium includes a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, a magneto-optical medium such as a floptical disk, and a ROM. , Hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention or may be known and available to those skilled in the computer software field. Examples of computer programs may include machine language codes such as those produced by a compiler, as well as high-level language codes that can be executed by a computer using an interpreter or the like.

The specific implementations described in the present invention are examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connecting members of the lines between the components shown in the drawings are illustrative examples of functional connections and / or physical or circuit connections, and in the actual device, alternative or additional various functional connections, physical It can be represented as a connection, or circuit connections. In addition, unless specifically mentioned, such as “essential”, “importantly”, etc., it may not be a necessary component for application of the present invention.

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar indication terms may be in both singular and plural. In addition, in the case where a range is described in the present invention, it includes the invention to which the individual values belonging to the range are applied (if there is no contrary description), and describes the individual values constituting the range in the detailed description of the invention. Same as Finally, unless there is a clear or contradictory description of the steps constituting the method according to the invention, the steps can be done in a suitable order. The present invention is not necessarily limited to the description order of the above steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is limited due to the examples or exemplary terms, unless it is defined by the claims. It does not work. In addition, those skilled in the art can recognize that various modifications, combinations, and changes can be configured according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Claims (15)

In the high-speed anomaly detection method using hierarchical convergence statistical information performed by the high-speed anomaly detection system,
An overall statistics calculating step of receiving raw data including a plurality of data from the database and calculating overall statistics of the data included in the raw data;
A first group statistics calculating step of dividing the raw data into a first layer cluster including the data of a predetermined division constant, and calculating a first group statistics for each of the divided first layer clusters;
A first hierarchical statistical calculation step of calculating a hierarchical convergence statistics for the first hierarchical layer based on the calculated overall statistics and the calculated first cluster statistics;
The second cluster statistics calculation unit calculates a second cluster statistics for dividing the divided first layer cluster into a second layer cluster containing the data of the division constant, and calculating a second group statistics for each of the divided second layer clusters. step;
A second layer statistics calculating step of calculating a layer convergence statistics for the second layer based on the hierarchical convergence statistics and the calculated second group statistics;
A statistical value repetition calculation step of repeating the second cluster statistics calculation step and the second hierarchical statistical calculation step until the statistical value repetition calculation unit satisfies a preset condition; And
Hierarchical convergence statistics including the ideality judgment step of determining the ideality of data belonging to the last layer cluster of the last layer based on the last layer convergence statistics calculated last in the statistical value iterative calculation step. High-speed anomaly detection method using information. According to claim 1,
The statistics calculated at each step constituting the high-speed anomaly detection method,
A high-speed anomaly detection method using hierarchical convergence statistical information, characterized in that at least one of the average, variance, standard deviation, and Zscore. According to claim 1,
The high-speed anomaly detection method,
The abnormal number receiving unit further includes an abnormal number receiving step of receiving a target abnormal number,
The ideality determination step,
A high-speed anomaly detection method using hierarchical convergence statistics information, characterized in that the data showing the highest abnormality among the data belonging to the cluster of the last layer is extracted by the target anomaly. According to claim 1,
The ideality determination step,
Based on the hierarchical convergence statistics calculated at each step constituting the high-speed anomaly detection method, high-speed anomalies using hierarchical convergence statistical information, characterized in that it detects the data having the highest abnormality among the data belonging to the raw data Detection method. According to claim 1,
The splitting constant is a high-speed anomaly detection method using hierarchical convergence statistical information, characterized in that 2. According to claim 1,
The division constant is a high-speed anomaly detection method using hierarchical convergence statistical information, characterized in that the integer is 3 or more. According to claim 1,
The statistics calculated at each step constituting the high-speed anomaly detection method,
A high-speed anomaly detection method using hierarchical convergence statistical information, characterized in that it is calculated from data sampled at a predetermined number. A computer-readable recording medium storing a program for executing the method according to claim 1. A total statistics calculation unit for receiving raw data including a plurality of data from a database and calculating overall statistics of the data included in the raw data;
A first group statistics calculation unit for dividing the raw data into a first layer cluster including data of a predetermined division constant, and calculating a first group statistics for each of the divided first layer clusters;
A first hierarchical statistical calculation unit for calculating hierarchical convergence statistics for the first layer based on the calculated overall statistics and the calculated first group statistics;
A second group statistics calculation unit for dividing the divided first layer cluster into a second layer cluster containing the data of the division constant, and calculating a second group statistics for each of the divided second layer clusters;
A second layer statistics calculation unit calculating a layer convergence statistics for the second layer based on the hierarchical convergence statistics and the calculated second group statistics;
A statistical value iterative calculation unit that repeats calculating cluster statistics and hierarchical convergence statistics until a predetermined condition is satisfied; And
Based on the last layer convergence statistics calculated at the statistical repetition calculation unit, hierarchical convergence statistical information including an abnormality determination unit that determines anomaly of data belonging to the last layer cluster of the last layer is utilized. High-speed anomaly detection system. The method of claim 9,
The overall statistics, cluster statistics and hierarchical convergence statistics,
A high-speed anomaly detection system using hierarchical convergence statistics, characterized by at least one of average, variance, standard deviation, and Zscore. The method of claim 9,
The high-speed anomaly detection system,
It further includes an abnormal number receiving unit for receiving a target abnormal number,
The ideality determination unit,
A high-speed anomaly detection system using hierarchical convergence statistical information, characterized in that the data showing the highest abnormality among the data belonging to the cluster of the last layer is extracted by the target anomaly. The method of claim 9,
The ideality determination unit,
A high-speed anomaly detection system using hierarchical convergence statistical information, characterized in that, based on the calculated hierarchical convergence statistics, the data having the highest abnormality among data belonging to the raw data is detected. The method of claim 9,
The splitting constant is high-speed anomaly detection system using hierarchical convergence statistical information, characterized in that 2. The method of claim 9,
The division constant is a high-speed anomaly detection system using hierarchical convergence statistical information, characterized in that the integer is 3 or more. The method of claim 9,
The overall statistics, cluster statistics and hierarchical convergence statistics,
A high-speed anomaly detection system using hierarchical convergence statistical information, characterized in that it is calculated from data sampled at a preset number.

Images