KR100776640B1 - System and method for finding the time sensitive frequent itemsets - Google Patents

Abstract

The present invention provides a system and method for searching a relatively frequent item using a time difference to search for a relatively frequent item using a time difference in a data mining system. The system of the present invention provides a predetermined transaction in a data stream input through an input module. And a search module for searching for frequent items through aggregation of items appearing within the network, and for searching for relatively frequent items using a time difference between items appearing during a predetermined transaction, and a storage module for storing frequent items found in the search module. The present invention can search for time-sensitive frequent items, increase the accuracy of frequent item search, and efficiently use a limited memory.

Data Mining, Transactions, Relative Frequencies, Time Differences, Navigation, Data Streams

Description

 System and method for searching relative frequent items using time difference {SYSTEM AND METHOD FOR FINDING THE TIME SENSITIVE FREQUENT ITEMSETS}

1 is a system configuration according to the present invention.

Figure 2 is a table showing the basic elements for searching relative frequent items according to the present invention.

Figure 3 is a view for explaining the relative frequent items according to the present invention.

4 is an algorithm of a method of searching for a relative frequent item using a time difference according to the present invention.

5 is a flow chart for FIG. 4.

Figure 6 is a graph showing the accuracy for the frequent items and frequent items as an experimental example of the present invention.

7 is a graph showing the average execution time according to the minimum support as an experimental example of the present invention.

8 is a graph showing FP-Tree memory usage as an experimental example of the present invention.

<Explanation of symbols for the main parts of the drawings>

10: input module 20: navigation module

30: storage module

The present invention relates to a data mining system, and more particularly, to a relatively frequent item searching system and method using a time difference to search for a relatively frequent item in consideration of a temporal aspect in a data stream.

Recently, due to the development of storage devices and the development of networks, a large amount of data is continuously increasing in a short time. For example, large amounts of data are generated in many applications such as network intrusion detection, ubiquitous, and e-commerce, and many efforts are being made in various fields to extract valuable information from such applications.

In general, in a data set that is the subject of data mining, all unit information that appears in an application field is defined as an item, and the unit information having semantic concurrency (that is, semantically occurring together) in the application field is defined. A meeting is defined as a transaction. A transaction has information of unit items having semantic concurrency, and a data set that is the subject of data mining analysis is defined as a set of transactions generated in a corresponding application field.

Extracting valuable information from data streams through data mining techniques is one of the main research areas, and data streams are characterized by continuous data growth in a very short time. Therefore, the following conditions are required to process data streams in data mining.

First, since it is impossible to store all of the growing data in a limited storage space in a very short time, there is a need for a method of efficiently storing data without loss of information by using the memory space flexibly.

Second, in the data stream, the data is generated very quickly, and the mining results at this point are important. To do this, each transaction in the data stream should be read only once as soon as it is created (Jang Joong-hyuk, Lee Won-seok., (2003). Searching for frequent items based on open data mining in the data stream. 10-D (3).)

The most basic problem in data streams is finding frequent items (Charikar, M., Chen, K., & Farach-Colton, M., (2002) .Finding Frequent Items in Data Streams.International Colloquium on Automata, Languages, and Programming, 508-515.).

Conventional data mining techniques use a lot of memory and require a long processing time because they make a candidate candidate with a single search for static transactions, and then find a frequent item with a higher support than a predetermined threshold.

In addition, the data stream cannot store all the unit items because a very large amount of data is constantly coming in, it is not suitable to apply the existing data mining techniques to the data stream. Therefore, new methods for finding frequent items in data streams are being studied (Jong Joong-hyuk, Lee Won-seok., (2003). Searching for frequent items based on open data mining in data streams. Journal of Information Processing Society, 10-D (3)., Manku, GS, & Motwani, R., (1994) .Approximate frequency counts over data streams, In Proc. Of the 28th Conference on Very Large Databases., Giannella, C., Han, J., Pei, J., Yan , X., & Yu, PS, (2003) .Mining Frequent Patterns in Data Streams at Multiple Time Granularities, Next Generation Data Mining, AAAI / MIT.

However, the method of searching for frequent items in the data streams up to now is searching for frequent items by simply counting frequent items or randomly setting a sliding window of a certain size, that is, searching for frequent items limited to a section. Since the frequent items are aggregated according to this flow, it is not possible to guarantee the accuracy of the frequent items search by overlooking the change of the frequent items and the relative frequent items over time.

Algorithms for frequent items, which have been studied previously, are mostly based on the Apriori principle (Rakesh Agrawal, & Ramakrishnan Srikant., (1994) .Fast Algorithms for Mining Association Rules.Proc. 20th Int.Conf.Very Large Data Bases, 487-499.).

The principle is that all subsets of frequent items must be frequent items. If the maximum length of frequent items is n, Apriori algorithm searches for up to n + 1 to create a candidate set, and since it searches for frequent items, it uses a lot of memory and it takes a lot of time to search for frequent items due to repeated database search. .

On the other hand, FP-growth using a divide-and-conquer does not produce candidate sets (Han, J., & Yin, Y., (2000). Mining frequent patterns without candidate generation. In Proc. IEEE Symposium on Foundations of Computer Science, 359-366.)

FP-growth shows that it is very efficient for mining long or short frequent items, is scalable, and is one step ahead in speed over the Apriori algorithm.

However, both of these techniques require multiple traversal of the data set and a full rescan when a new transaction occurs. In addition, if the data set continuously increases rapidly, performance may be lowered due to the limitation of available memory.

The Count Sketch algorithm also focuses on the frequency of unit items in the data stream (Charikar, M., Chen, K., & Farach-Colton, M., (2002) .Finding Frequent Items in Data Streams. International Colloquium on Automata, Languages, and Programming, 508-515.), The Lossy Counting algorithm finds a set of frequent items that occur in a data stream given the minimum support and maximum tolerance conditions (Manku, GS, & Motwani, R). , (1994) .Approximate frequency counts over data streams, In Proc. Of the 28th Conference on Very Large Databases.).

However, these algorithms focus on simply searching for frequent items without considering time, and thus, there is a disadvantage in that an accurate frequent item cannot be searched.

The present invention has been made in view of the above, and an object of the present invention is to accurately search for frequent frequent items that have not been found by using the time difference in which items appear during a certain transaction in consideration of time in a data stream. The present invention provides a system and method for searching for relatively frequent items using a time difference to improve the performance and to efficiently use and manage limited memory resources.

In order to achieve the object of the present invention, a relatively frequent item search system using a time difference according to the present invention comprises: an input module for processing an input data stream when searching for frequent items in a data mining system; A search module for searching for frequent items by aggregating items appearing within a predetermined transaction in a data stream input through the input module, and searching for a relatively frequent item using a time difference between items appearing during a predetermined transaction; And a storage module for storing the frequent items searched by the search module.

The relative frequent items have a relative occurrence frequency that is a value obtained by dividing the number of fluid transactions by the sum of the differences for the frequency of appearance intervals, and is larger than the relative occurrence frequency for the frequent items. The result of subtracting the value of the relative occurrence frequency of is characterized in that the time from less than zero to the time greater than zero.

In order to achieve the object of the present invention, a method for searching a relatively frequent item using a time difference according to the present invention, in the method for searching a frequent item in a data mining system, searches for a transaction newly added to a data stream and appears for each item. Updating the frequency and ID information of the transaction; Adding an item having a value smaller than a predetermined minimum support but greater than a maximum support error among the items appearing in the added transaction as a partial frequent item; Outputting information of an item having the highest total frequency and the support of at least the minimum support to date at the request of the user; And searching for a relatively frequent item using a time difference between items that appear during the transaction.

In addition, the relative frequent item search method using the time difference of the present invention using the FP-Tree algorithm, in the frequent item search method using the FP-Tree algorithm in the data mining system, by searching for a transaction newly added to the data stream Updating appearance frequencies and transaction ID information for each item when items appearing in a transaction exist in the node of the FP-Tree; If items appearing in the added transaction do not exist in the node present in the FP-Tree, inserting items having a value smaller than the minimum support but larger than the maximum support error into partial frequent items and inserting them into the nodes of the FP-Tree. ; Outputting information of an item having the highest total frequency and the support of at least the minimum support to date at the request of the user; And searching for a relatively frequent item using a time difference between items that appear during the transaction.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the following examples are merely to illustrate the invention is not limited to the contents of the present invention.

1 is a diagram illustrating a system configuration for implementing the present invention. An input module 10 for processing a large input data stream, a search module 20 for searching for frequent items in the data stream, and a searched frequent item It consists of a storage module 30 for storing the.

The search module 20 searches for frequent items through the aggregation of items that appear within a given transaction in the data stream input through the input module 10, and uses the time difference between the items that appear during the given transaction and the relative frequent items. The search for frequent items of the present invention is performed by the search module 20.

In the present invention, first, a data set to be searched for defining frequent items in a data stream is defined.

S _N = {T ₁ , T ₂ , T ₃ , ..., T _N } refers to the set of transactions that have occurred to date, that is, the entire data set, and the current transaction is T _t = {I ₁ , I ₂ , I ₃ , ..., I _N } (t = 1, .., n), and each transaction has an ID that is a unique identifier called TID. The set of unit items that are components of a transaction is represented by I = {i ₁ , i ₂ , i ₃ , ..., i _N }.

And FP-stream algorithms (Giannella, C., Han, J., Pei, J., Yan, X., & Yu, PS, (2003) .Mining Frequent Patterns in Data Streams at Multiple Time Granularities, Next Generation Data Mining , As in AAAI / MIT.), A single item is frequently used with a minimum support defined by the user (S _min ∈ (0,1)) and a maximum support error (e∈ (0, S _min )). , Divided into frequent items and non-frequent items.

If the frequency of occurrence is greater than the minimum support, it is considered as a frequent item. If the frequency of occurrence is smaller than the minimum support but more than the maximum support error, it is defined as a frequent item. If less than the maximum support error, it is regarded as an infrequent item and will not be treated.

In addition, in the existing studies, the frequent items and partial frequent items can be overlooked and passed over time-sensitive relative frequent items as the frequent items are searched by only comparing the aggregated values. As a small but relative occurrence frequency will need to be considered when the relative frequency is relatively large compared to the current frequent items, the present invention defines the relative frequent items as follows.

The relative frequent item set R is defined as a set of items in which the occurrence frequency is smaller than the occurrence frequency of the current frequent item, but the relative occurrence frequency is larger than the relative frequency of the current frequent item. Where f is the current frequent item and c is the current time point.

Relative frequency is the number of flexible transactions, m (m <N), divided by the sum of the differences in the frequency of occurrence, which is a criterion to search for relative frequent items.

In addition, the appearance frequency is a value obtained by counting whether an item has appeared or not appeared for a continuously input transaction.

The sum of the differences for the frequency of occurrence is to determine the time when the relative frequent items occur, and is defined as the sum of the time differences between the present time and the previous time. Here y _t means the time point when the transaction T _t containing the item x occurs.

As shown in the table of FIG. 2, _Smin is the minimum support, e is the maximum support error, f is the frequent item, R is the relative frequent item, and A (x) is the item x. The appearance determination function for, F (x) is the frequency of occurrence of item x, C (x) is the sum of the appearance intervals for item x, and E _m (x) is the relative frequency of occurrence of item x.

Based on these details, the relative frequent items are as follows.

3 is a conceptual diagram of relative frequent items. Here, each of I ₁ , I ₂ , and I ₃ represents a frequent item having a value greater than or equal to the minimum support and a partial frequent item smaller than the minimum support map but larger than the maximum support error, and the points of each arrow indicate the frequency of appearance.

In addition, T _N means a transaction to date. Looking at I ₁ on the first line, we can see that the frequency of occurrence so far is 28, the highest. That means frequent items. The second and third lines indicate partial frequent items.

In FIG. 3, the first item, that is, the frequent item, has a high frequency of appearance at the beginning and an interval of appearance gradually increases toward the middle. However, the second line shows a sharp increase in the frequency of appearance in the middle.

In addition, since the appearance of the first item is more frequent than the first item and the interval between appearances is narrower than that of the first item, the second item should be designated as the frequent item in the middle. That is, the second item is a relatively frequent item in the middle rather than the first item.

The important point here is to find the criteria for when to change to a relatively frequent item. Therefore, the present invention proposes a criterion for the following relatively frequent items.

1. The relative frequency of occurrence should be greater than the relative frequency of frequent items.

2. The time of appearance of relative frequent items is from the time when the result of subtracting the current relative frequency of occurrence from the value of the previous relative frequency of occurrence is less than zero to the time when it becomes larger than zero.

In other words, it means when the relative frequency of occurrence of the frequent items is increased and the interval of appearance is sharply narrowed compared to the frequent items.

Based on the above-described matters, the present invention provides a method using a FP-Tree algorithm of a prefix tree structure that efficiently maintains and manages all frequent items and relative items in a memory, which is the storage module 30. Save the frequent items.

In a data stream, data is considered to be an infinite set. As such, storing all items is virtually impossible. Therefore, in the FP-Tree, it is desirable to store only three items of items, frequency of appearance, and TID in order to efficiently maintain and manage frequent items and relative items.

Here, items means frequent items or partial frequent items, and the frequency of appearance is the total number of times the items have appeared. In addition, TID means the current transaction ID and is used as a measure to know when relative frequent items occur.

Figure 4 shows the FP-Tree algorithm according to the present invention, Figure 5 shows a flow diagram for this, the FP-Tree algorithm is composed of four steps, each time the transaction is added the following four steps repeatedly Will be performed.

The first step S110 is to search for transactions in the data stream and update the appearance frequency for each item. When a new transaction is added, the size of the entire data set | S _N | is increased by one.

If items appearing in the newly added transaction exist in the node of FP-Tree, the appearance frequency and transaction ID value are updated.

The second step (S120) is a step in which partial frequent items are added. If items appearing in the newly added transaction are not present in the node present in the FP-Tree, the second step (S120) may include an item having a value smaller than the minimum support but larger than the maximum support error. Separate into frequent items and insert them as nodes of FP-Tree.

Here, items smaller than the maximum support error are removed because they are unlikely to be frequent items. This reduces the memory footprint and reduces the execution time because there is no additional work to insert into the nodes of the FP-Tree.

The third step (S130) is to find the current frequent items, and outputs items, frequency of appearance, and TID information of items having the highest total frequency and the least support degree (S _min ) by the user's request. give.

The fourth step (S140) is to find a relatively frequent item. When the relative frequency of occurrence of the frequent item is increased, that is, when the interval of appearance becomes narrow, the fourth item (S140) searches for an item that appears more frequently than the current frequent item as a text file. Print information.

As such, the present invention ensures reliability and accuracy by searching for the current frequent items as well as the relative frequent items that may be overlooked and overlooked, and efficiently manages limited memory because it manages only the items, the frequency of occurrence, and the TID for the frequent items and the partial frequent items. It can be used as.

Next, an experimental example of the present invention will be described. In the experimental example, the relative frequent items and the performance of the FP-Tree are verified through various experiments.

The data set is T10 according to the data generation method proposed in "Rakesh Agrawal ,, & Ramakrishnan Srikant., (1994). Fast Algorithms for Mining Association Rules. Proc. 20th Int. Conf. Very Large Data Bases, 487-499." Create and use .I4.D1000K, T20.I4.D1000K and T10.I4.D100K and T15.I6.D1000K data sets.

In each data set, T is the average length of the transaction, and I is the average length of the potential maximum frequent items. D is also the total number of transactions for the data set. In the present invention, all experiments were conducted in a computer environment of AMD XP 2600+ having 512MB of RAM, and implemented in C language.

The experiment is largely divided into three parts.

The first is to verify the accuracy of frequent and relative frequent item search.

The second is to verify the execution time of searching for frequent items and relative items.

Finally, the third verifies the memory usage for the FP-Tree that manages frequent and partial frequent items.

6 shows the relative frequent items and the accuracy of the frequent items using the T10.I4.D1000K data set. Frequently, the frequent items show a somewhat higher accuracy than the relative frequent items.

The reason for this is that it is more difficult to search than the frequent items because the relative frequent items change more frequently under the influence of time than the frequent items. Nevertheless, it shows high accuracy.

FIG. 7 compares the average execution time of each section when the minimum map is variously changed. Average execution time refers to the average time spent searching for frequent and relative frequent items when a new transaction is added. The lower the minimum support, the greater the average execution time.

The reason is that the lower the minimum support, the wider the allowable range for the frequent items and the partial frequent items, which increases the number of comparisons to search for the frequent items. In other words, there is an inverse relationship between the average execution time and the minimum map.

8 shows memory usage in the FP-Tree according to each data set. The difference in memory usage between each data set is not significant.

The reason for this is that the FP-Tree algorithm manages only the frequent items and the partial frequent items by using the minimum support and the maximum support error, thereby reducing the memory usage. T15.I6.D1000K uses the most memory of each data set. T15.I6.D1000K uses relatively more memory because the average length of potential maximum frequent items is larger than other data sets.

As described above, although described with reference to a preferred embodiment of the present invention, those skilled in the art various modifications of the present invention without departing from the spirit and scope of the invention described in the claims below Or it may be modified.

As described above, the present invention calculates the relative frequent frequency according to the total frequency and the frequent interval, and compares the relative frequent frequency according to the frequent items and the partial frequent items, and searches for the relative frequent items that can be overlooked and overlooked. In addition, in order to efficiently manage frequent items and partial frequent items in FP-Tree, time-sensitive frequent items are stored by storing only three pieces of information such as unit items such as frequent items or partial frequent items, frequency of occurrence, and transaction ID. It can search, improve accuracy of frequent items search, and efficiently use limited memory.

Claims (10)

In searching for frequent items in a data mining system that mines an input data stream, An input module for processing an input data stream; In the data stream input through the input module, the frequent items are searched through the aggregation of items that appear in a predetermined transaction, and the relative frequency of occurrence, which is a value obtained by dividing the number of flexible transactions by the sum of the differences for the frequency of appearance, is frequent. The relative frequency of occurrence is greater than the relative frequency of, and the time of occurrence is searched for relative frequent items from the time when the result of subtracting the current relative frequency of occurrence from the value of the previous relative frequency of occurrence is less than zero to the time of greater than zero. Navigation module; And A storage module for storing the frequent items searched by the search module; Relative frequent items search system using a time difference characterized in that it comprises a. delete delete The system of claim 1, wherein the storage module stores search results according to an FP-Tree algorithm. The method of claim 4, wherein the storage module And a unit item including the frequent item or the partial frequent item, a frequency of appearance, which is a total number of times the unit item has appeared, and a transaction ID. In the frequent item search method in a data mining system for data mining an input data stream, Searching for a transaction newly added to the data stream and updating occurrence frequency and transaction ID information for each item; Adding an item having a value smaller than a predetermined minimum support but greater than a maximum support error among the items appearing in the added transaction as a partial frequent item; Outputting information of an item having the highest total frequency and the support of at least the minimum support to date at the request of the user; And The relative occurrence frequency, which is the number of the transaction divided by the sum of the differences for the appearance frequency interval, is greater than the relative occurrence frequency for the frequent items, and the time of appearance is the current relative occurrence frequency at the value of the previous relative occurrence frequency. Searching for a relatively frequent item from a time point at which the result of subtracting the value is smaller than 0 to a time point greater than 0; Relative frequent items search method using a time difference characterized in that it comprises a. delete The method of claim 6, wherein the output information is And a unit item including a frequent item, information about a frequency of appearance, a transaction ID, and the like, which are the total number of times the unit item has appeared so far. In the frequent item search method using the FP-Tree algorithm in the data mining system for data mining the input data stream, Searching for a transaction newly added to the data stream and updating the frequency of appearance of each item and the ID information of the transaction if items appearing in the transaction exist in the node of the FP-Tree; If items appearing in the added transaction do not exist in the node present in the FP-Tree, dividing an item having a value smaller than the minimum support but greater than the maximum support error into partial frequent items and inserting the item into the node of the FP-Tree; Outputting information of an item having the highest total frequency and the support of at least the minimum support to date at the request of the user; And The relative occurrence frequency, which is the number of the transaction divided by the sum of the differences for the appearance frequency interval, is greater than the relative occurrence frequency for the frequent items, and the time of appearance is the current relative occurrence frequency at the value of the previous relative occurrence frequency. Searching for a relatively frequent item from a time point at which the result of subtracting the value is smaller than 0 to a time point greater than 0; Relative frequent items search method using a time difference characterized in that it comprises a. The method of claim 9, wherein the output information is And a unit item including a frequent item, information about a frequency of appearance, a transaction ID, and the like, which are the total number of times the unit item has appeared so far.

Images