KR20100062854A - Method for stream pattern mining using bitwise map - Google Patents

Abstract

PURPOSE: A method for mining a stream pattern using a bitmap is provided to use a bitmap based on a sliding window, thereby efficiently compressing stream data which are continuously inputted. CONSTITUTION: Frequent pattern candidate groups about an item group are combined(S706). The item group includes more items than a previous time group. A bitmap of the combined frequent pattern candidate groups is calculated(S708). A bitmap support rate about the calculated bitmap is calculated.

Description

Method for Stream Pattern Mining using Bitwise Map}

The present invention relates to a stream frequent pattern mining method using a bitmap. In particular, the present invention relates to a method of expressing stream data continuously input using a bitmap, and identifying frequent patterns using stream frequent pattern mining based on bitmaps.

The present invention is derived from the research conducted as part of the postal technology development project of the Ministry of Knowledge Economy and the Ministry of Information and Communication Research and Development. [Task management number: 2006-X-001-03, Task name: Real-time postal logistics element technology development].

Computer technology is developing rapidly. With the development of computer input technology, it is possible to continuously input a large amount of data in a short time, and with the development of computer processing technology, a large amount of stream data is input.

The frequent pattern mining method is used as one of the techniques for processing such a large amount of stream data. In processing the input stream data using the stream frequent pattern mining technique, the following conditions must be satisfied.

First, a large amount of continuous stream data that is continuously input should be scanned only once, the input data must be processed quickly before new data is input, and the user must be notified of the processed result promptly. In addition, the input data must be processed within a limited memory capacity.

In a method of performing stream frequent pattern mining, various techniques exist. Conventionally, in performing the stream data representation compression technique, a large amount of memory is consumed. Accordingly, the stream frequent pattern mining technique also has a problem of consuming a large amount of memory.

SUMMARY OF THE INVENTION The present invention provides a bitmap compression technique represented by a small amount of memory without data loss for stream data, and a stream frequent pattern mining technique with improved fast computation and memory efficiency based on the bitmap compression technique. will be.

To this end, the frequent pattern checking method according to an embodiment of the present invention is a method of checking a frequent pattern for a transaction of continuously input stream data, and (a) confirming a bitmap support of a set of items included in the input transaction. Doing; (b) determining a frequent pattern if the bitmap support is greater than or equal to a preset support, and combining a frequent pattern candidate set for an item set further including a specific number of items than the previous item set; (c) calculating a bitmap of the combined set of one or more frequent pattern candidates and calculating bitmap support for the calculated bitmap; And (d) repeating the process after step (b) until there is no frequent pattern candidate set.

In addition, a bitmap compression method according to an embodiment of the present invention is a bitmap compression method for a transaction of stream data that is continuously input, the method comprising the steps of: (a) inserting the input transaction in the time interval of the sliding window; (b) Checking whether the item included in the transaction exists in the sliding window, and if the existence is confirmed, move the bitmap of the item by a predetermined bit, add a preset bit value, and then include the item included in the transaction. If not present in the sliding window, assigning a new bitmap and adding the new bitmap to the sliding window; And (c) deleting the item having the bitmap support of 0 in the sliding window.

According to the present invention, it is possible to efficiently compress stream data continuously input by using a bitmap based on a sliding window, and by using stream data compressed in a stream frequent pattern mining technique, an existing stream frequent pattern It can be expected to reduce the memory usage more efficiently than the mining technique and to reduce the processing time for stream frequent pattern detection.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise. In addition, the terms “… unit”, “… unit”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

The present invention relates to a stream frequent pattern mining technique using bitmaps that efficiently compresses continuously input stream data without loss, reduces memory usage, and processes the data rapidly.

1 is a diagram illustrating a time-sensitive sliding window (TimeSW) according to an embodiment of the present invention.

When an item of stream data input in the present invention is called 'i', the set of items ψ is' ψ = {i ₁ , i ₂ ,... , i _m } '. In the following, a set of items ψ will be described as an item set, and the item set is not an empty set. In addition, an item set consisting of k items will be described as a k-item set.

Transaction is a transaction data stream is continuously _{input, 'TSD = T 1, T} 2, ... , T _t '. In such transaction stream data, the last transaction entered is 'T _t '.

In addition, the transaction (T) is composed of an ID (TUid), a transaction ID (Tid) and an item set of a time interval, and is expressed as 'T = (TUid, Tid, itemset)'.

The sliding window TimeSW, which is sensitive to the time of transaction stream data, is a window for selecting a time interval TU. And, each time interval (TU _i ) is | TU _i | It consists of two transactions. Here, | TU _i | is the size of the time interval TU _i .

In addition, the sliding window TimeSW _{N−ω + 1} having a size of ω time intervals is defined as “[TU _{N−ω + 1} , TU _{N−ω + 2} ,... , TU _N ] '. Here, 'N−ω + 1' is an ID of the current view sliding window time interval and 'N' is a TUid of the last view sliding window time interval (TU _N ).

Each sliding window consists of ω time intervals whose magnitude value is' | ω | = | TU _{N-ω + 1} | + | TU _{N-ω + 2} | +… + | TU _N | ' The size of the entire sliding window is represented by '| TimeSW |'.

When the new stream data is input, the sliding window TimeSW moves. As the sliding window moves, the old time period transactions are deleted and new time period transactions are added.

For efficient deletion of previous time interval transactions, it is desirable to use a transaction size arrangement (TSA) within the time interval. Transaction size arrangement according to an embodiment of the present invention is shown in FIG.

When an item in the sliding window TimeSW is called 'X', the support of the item X is expressed as 'sup (X) ^TimeSW '. If the support for item X is greater than the predetermined minimum support, it is defined as a frequent pattern. The determination condition of the frequent pattern is expressed as 'sup (X) ^TimeSW ≥ s * | TimeSW |', where s is a predetermined minimum support degree.

Then, for each item X in the sliding window TimeSW _{N−ω + 1} having ω time interval sizes, | TimeSW _{N−ω + 1} | A bit sequence of size is represented by Bit (X) ^{TimeSWN−ω + 1} .

When transactions within a new time interval TU _N are input, a bitmap compression process that compresses the bitmap must be performed.

3 is a flowchart illustrating a bitmap compression method according to an embodiment of the present invention.

When transactions in the new time interval TU _N are input, it is checked whether the sliding window TimeSW is fully filled (S310).

If the sliding windows are all filled, the transaction in which the transaction is inserted for the first time, that is, the transaction stored in the TU _{N-ω + 1} is deleted from the sliding window (S320), and in the transaction size array (TSA), the | TU _N-ω Delete ₊₁ | (S330). Then, after moving the other transaction arrays forward one by one (S324), the transaction | TU _N | of the newly entered time interval is inserted into the last transaction size array TSA [ω] (S326).

However, if an empty section exists in the sliding window TimeSW in step S310, a transaction | TU _N | of a newly input time section is inserted in the last section of the transaction size array TSA (S330).

When a transaction of a newly input time interval is inserted into the sliding window through step S326 or S330, for each item x of all transactions in the newly input time interval TU _N , it is checked whether x exists in the sliding window. (S340). If x is present in the sliding window, the bitmap Bit (x) of the corresponding item x is moved one bit to the left (S350), and then a bit value '1' is added to the bitmap of the item x (S352).

However, as a result of checking in step S340, if x does not exist in the sliding window, a bit value '1' is assigned to the new bitmap Bit (x) (S360), and added to the sliding window (S362).

When the bitmap is added to the sliding window by the step S352 or the step S362, for the item x that is not within the newly input time interval TU _N in the sliding window, | TU _N | Moving forward by a bit (S370).

In operation S380, items having a support of '0' are deleted from all items in the sliding window.

Such a bitmap compression technique may be implemented with the algorithm described in FIG. 4.

5 and 6 are diagrams for describing a processing procedure of a bitmap compression technique using the algorithm described in FIG. 4 according to an embodiment of the present invention.

As shown in FIG. 5, stream data having a tid of T ₁ in TU ₁ is input for the first time. Accordingly, an item consisting of 'be' is inputted, and 1 is assigned to the bitmap bit (b) of b and the bitmap bit (e) of e, respectively. Accordingly, Bit (b) becomes '1' and Bit (e) becomes '1'.

After the first item is entered, an item with a tid of T ₂ in TU ₁ is entered. Accordingly, an item consisting of 'bce' is input, and bit value '1' is additionally added to bitmap bit (b) of b, bitmap bit (c) of c and bitmap Bit (e) of e, respectively. Is assigned. Accordingly, Bit (b) is '11', Bit (c) is '1', and Bit (e) is '11'.

After the second item is entered, an item consisting of 'ce' with tid of T ₃ in TU ₁ is entered. Accordingly, bit value '1' is additionally assigned to bitmap bit (c) of bit c and bitmap bit (e) of e, respectively, so that Bit (c) is '11' and Bit (e) is '111'. do. However, since the item for b has not been input, the bit value '0' is assigned to the bitmap Bit (b) of b, whereby Bit (b) becomes '110'.

In this way, when stream data is input, a bitmap is allocated as in 6A.

If TU ₃ is input to an item having a tid of T _7, the bitmap of b and the bitmap of e each have a size of 7 bits. Accordingly, a bit array of each item is formed in the first sliding window TimeSW _{1 as} shown in 6B. As described above, when the bit array of the first sliding window TimeSW ₁ has a size of 7 bits, a bitmap compression technique is performed to form a second sliding window TimeSW ₂ having a form such as 6C.

When an item 'bcde' having a tid of T ₈ is input in TU ₄ , '1' or '0' is added to a bitmap of each item of the second sliding window TimeSW ₂ .

As shown in FIG. 5, when the input from the TU ₄ to the item having the tid of T ₉ ends, each bitmap constituting the second sliding window TimeSW ₂ includes a bit array such as 6E.

Based on the stream data represented by such a bitmap compression technique, a frequent pattern in the sliding window TimeSW _{N−ω + 1} can be explored.

7 is a flowchart illustrating a frequent pattern mining technique according to an embodiment of the present invention.

First, for frequent pattern mining, the support of the bitmap Bit (x) of all items x in the sliding window is checked. At this time, the support for Bit (x) is calculated as sup (x) ^TimeSW . Here, sup (x) ^TimeSW is the number of 1 bits in Bit (x).

Then, the support of the identified bitmap is compared with a preset minimum support. Here, the minimum support is calculated as | TimeSW | * s, and s is a predetermined minimum support, which is a value between [0, 1] (S702).

As a result of comparing the support of the bitmap with the preset minimum support, if the bitmap support is greater than or equal to the minimum support, it is determined as a frequent pattern, and a 1-item set frequent pattern FI ₁ is added (S704).

When the 1-item set frequent pattern FI ₁ is determined, the candidate set CI ₂ of the 2-item set frequent pattern is combined based on the 1-item set frequent pattern FI ₁ . In this case, as an algorithm for generating a candidate set of the 2-item set frequent pattern, it is preferable to use a Generation of Candidate Itemsets using Apriori property (GCIA) based on an Apriori algorithm (S706).

When the combined 2-item set is configured, a bitmap is calculated through an AND operation on the frequent pattern candidate set CI ₂ of all the combined 2-item sets. For example, if the combined two-item is 'ab', each item set is composed of a and b, and the bitmap bit (ab) of the item set (ab) is a bit (a) and a bit (b). It can calculate by AND operation (S708).

When the bitmap of each item set for all generated 2-item set candidate sets CI ₂ is calculated, the support for the bitmap of each 2-item set c ₂ is checked. At this time, the support for the bitmap of each 2-item set (c ₂ ) is calculated as sup (c ₂ ) ^TimeSW .

When the support for the bitmap of the ₂ -item set (c ₂ ) is calculated, the calculated support of the 2-item set is compared with a preset minimum support (| TimeSW | * s) (S710).

If the support for the bitmap of the ₂ -item set (c ₂ ) (sup (c ₂ ) ^TimeSW ) is greater than or equal to the minimum support (| TimeSW | * s), it is determined as a frequent pattern, and the 2-item set frequent pattern (FI ₂ ) is added (S712).

In the same manner, a candidate set (CI ₃ ) of the 3-item set frequent pattern is combined based on the 2-item set frequent pattern (S716), and a bitmap using the candidate set (CI ₃ ) of the ₃ -item set frequent pattern. By calculating (S720), the support for the bitmap of the ₃ -item set (c ₃ ) and the predetermined minimum support (| TimeSW | * s) are compared (S722).

If the support for the bitmap of the ₃ -item set (c ₃ ) (sup (c ₃ ) ^TimeSW ) is greater than or equal to the minimum support (| TimeSW | * s), it is determined as a frequent pattern, and the 3-item set frequent pattern ( FI ₃ ) is added (S724).

The process after S716 for checking the frequent pattern of the k-item set is repeated while increasing the number of k one by one, for example, calculating a candidate of the four-item set frequent pattern based on the 3-item set frequent pattern. (S726).

This process is repeated until there is no more k k -item set candidate set CI _k (S718).

Such a frequent pattern mining technique may be implemented by the algorithm described in FIG. 8.

FIG. 9 is an exemplary diagram illustrating a result of a stepwise processing for a frequent pattern mining process using a bitmap using the bitmap data described in FIG. 6.

The embodiments of the present invention described above are not only implemented by the apparatus and method but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded, The embodiments can be easily implemented by those skilled in the art from the description of the embodiments described above.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

1 is a view showing a sliding window according to an embodiment of the present invention,

2 illustrates a transaction size arrangement in accordance with an embodiment of the present invention;

3 is a flowchart illustrating a bitmap compression method according to an embodiment of the present invention;

4 is a diagram illustrating a bitmap compression scheme algorithm according to an embodiment of the present invention;

5 is an exemplary diagram of stream data according to an embodiment of the present invention;

6 is a view for explaining a process of processing stream data using a bitmap compression scheme algorithm according to an embodiment of the present invention;

7 is a flowchart illustrating a frequent pattern mining technique according to an embodiment of the present invention;

8 illustrates a frequent pattern mining algorithm algorithm according to an embodiment of the present invention;

9 is an exemplary view showing a result of a step-by-step processing for the frequent pattern mining process using a bitmap according to an embodiment of the present invention.

Claims (10)

In the method of confirming a frequent pattern for a transaction of stream data continuously input, (a) checking the bitmap support of the item set included in the input transaction; (b) determining the frequent pattern if the bitmap support is greater than or equal to a preset support, and combining a frequent pattern candidate set for an item set further including a specific number of items than a previous item set; (c) calculating a bitmap of the combined set of one or more frequent pattern candidates and calculating bitmap support for the calculated bitmap; And (d) repeating the steps after step (b) until the frequent pattern candidate set does not exist Frequent pattern check method comprising a. The method of claim 1, Step (a) is, Identifying bitmaps for all items in the sliding window into which the transaction is inserted; And Confirming the bitmap support by checking the number of '1' bits included in the bitmap Frequent pattern check method comprising a. The method of claim 1, Step (b) is, Comparing the bitmap support with the size of the sliding window into which the transaction is inserted and a predetermined minimum support, wherein the minimum support is a value between 0 and 1, to determine whether there is a frequent pattern. Frequent pattern check method. The method according to claim 1 or 2, Step (c) is, (c1) calculating a bitmap through an AND operation on the frequent pattern candidate set; And (c2) checking the bitmap support by checking the number of '1' bits included in the bitmap Frequent pattern check method comprising a. The method of claim 1, Step (b) is, And if the bitmap support is greater than or equal to a preset minimum support, combining a frequent pattern candidate set for an item set including one more item than the previous item set. In the bitmap compression method for a transaction of stream data continuously input, (a) inserting an input transaction into a time interval of a sliding window; (b) checking whether an item included in the transaction exists in the sliding window, and if the existence is confirmed, moving the bitmap of the item by a predetermined bit, adding a preset bit value, and adding the transaction. Allocating a new bitmap to the sliding window if the item included in the sliding window does not exist in the sliding window; And (c) deleting an item having a bitmap support of 0 from the sliding window Bitmap compression method comprising a. The method of claim 6, Step (a) is (a1) confirming whether or not the time interval of the sliding window that accommodates the transaction is filled when the input of the transaction is confirmed; (a2) inserting the input transaction by deleting the first input transaction when the time intervals of the sliding window are all filled; And (a3) inserting the input transaction into a time interval located immediately after the time interval in which the transaction is filled in the sliding window when the sliding windows are not all filled; Bitmap compression method comprising a. The method of claim 7, wherein step (a2) (a21) deleting a transaction inserted in a time interval located at the front of the sliding window; (a22) moving forward the transactions filled in the time interval of the sliding window one by one; And (a23) inserting the input transaction in a time interval located at the end of the sliding window; Bitmap compression method comprising a. The method of claim 6, Step (b) is, (b1) checking whether an item included in the transaction exists in the sliding window; (b2) if an item included in the transaction exists in the sliding window, moving a bitmap of the item forward by one bit, and adding a bit value of '1' to the bitmap; And (b3) if an item included in the transaction does not exist in the sliding window, allocating a bitmap for the item and adding a bit value of '1' to the allocated bitmap. Bitmap compression method comprising a. The method of claim 6, Between step (b) and step (c), Among the bitmaps included in the sliding window, moving forward of the bitmap of the item not included in the transaction by the size of the time interval. Bitmap compression method further comprising.

Images