KR20190103918A - System and method for large-scale frequent pattern mining on distributed systems - Google Patents

Abstract

According to one embodiment of the present invention, provided is a frequent pattern mining method of a distributed system, comprising the steps of: determining a candidate (k + 1) -item set using a frequent k-item set; calculating partial supports of the candidate (k + 1) -item set using bitmap chunks distributed in the distributed system; summing the partial supports of the candidate (k + 1) -item set; and determining a frequent (k + 1) -item set based on the summation of the partial supports.

Description

Distributed system based large frequency pattern mining system and method {SYSTEM AND METHOD FOR LARGE-SCALE FREQUENT PATTERN MINING ON DISTRIBUTED SYSTEMS}

The present invention relates to a method of mining a frequent pattern in data. More specifically, the present invention relates to a method of mining frequent patterns in a distributed transaction database by using a MapRedcue technique. The present invention relates to a system and a method capable of quickly mining an equation.

The amount of data accumulated with the development of IT technology and the emergence of the Internet is increasing exponentially. It is becoming difficult to extract useful information from such huge data. In order to solve this problem, data mining techniques are being researched and developed.

Among the data mining techniques, frequent pattern mining is a technique for finding frequent patterns in data. Frequent pattern mining searches the itemset enumeration tree in depth-first or breadth-first search, while materializing the intermediate result patterns in memory, Since it checks whether the support is greater than or equal to the minimum support, it can basically have exponential time complexity. Moreover, as the amount of data increases, methods for finding frequent patterns quickly become important.

The frequent pattern mining system using a distributed system distributes and stores data on multiple machines and simultaneously distributes them using multiple machines, so that frequent pattern mining is performed for data larger than the data that a single machine-based method can handle. It may be possible to do this.

The frequent pattern mining system based on the conventional distributed system has the advantage of processing the input data larger than the memory or disk of a single machine, but the methods using the conventional distributed system have a high workload skewness and a large medium size. There is a problem that is slow due to large-scale intermediate data and network communication overhead. In addition, when the number of partitions increases due to the increase in the number of machines, there is a problem in that data redundancy between the machines increases, so that the frequent pattern mining performance is degraded, that is, the scale is poor due to the increase in the number of machines.

The present invention has been proposed to solve the above-described problems, and provides a system and method capable of effectively extracting frequent patterns from large-scale data, and the scale of the processed data and the system included in the system. It is an object of the present invention to provide a system and method having scalable performance in accordance with the number.

A frequent pattern mining method of a distributed system according to one side includes determining a candidate (k + 1) -item set using a frequent k-item set; Calculating partial supports of the candidate (k + 1) -item set using bitmap chunks distributed in the distributed system; Summing partial supports of the candidate (k + 1) -item set; And determining a frequent (k + 1) -item set based on the result of the sum of the partial supports.

The frequent pattern mining method may further include broadcasting the frequent (k + 1) -item set to the distributed system.

The distributed system may include a plurality of mappers corresponding to the bitmap chunks. Computing the partial supports of the candidate (k + 1) -item set includes loading each of the bitmap chunks into a corresponding mapper; In each of the plurality of mappers, bit vectors of candidate (k + 1) -items included in the candidate (k + 1) -item set are calculated through an AND operation of bit vectors included in a corresponding bitmap chunk. step; And calculating, at each of the plurality of mappers, partial support of the candidate (k + 1) -item set based on corresponding bit vectors.

The determining of the frequent (k + 1) -item set may include: a predetermined minimum support of a plurality of candidate (k + 1) -items included in the candidate (k + 1) -item set; And including the candidate (k + 1) -item with support or higher in the frequent (k + 1) -item set.

The frequent pattern mining method may further include terminating the frequent pattern mining when the candidate (k + 1) -item set does not exist.

The adding of the partial support may include partial support values corresponding to the same candidate (k + 1) -item among a plurality of candidate (k + 1) -items included in the candidate (k + 1) -item set. Shuffle the partial supports to be delivered to a reducer.

Determining the candidate (k + 1) -item set comprises receiving the frequent k-item set broadcast to the distributed system; And determining candidate (k + 1) -items by combining a plurality of frequent k-items included in the frequent k-item set.

The frequent pattern mining method may include: dividing a plurality of items included in data distributed and stored in the distributed system in a predetermined manner; Determining patterns based on the divided items; And generating a bitmap chunk for the corresponding mapper by determining bit vectors representing the patterns included in the corresponding data, using the mapper corresponding to each of the distributed and stored data. The items may correspond to a frequent 1-item set.

The bitmap chunk has a key-value structure, the key is a set of items corresponding to the patterns, and the value may include the bit vectors. The bitmap chunks may have the same key.

The generating of the bitmap chunk may include determining bit vectors corresponding to the combination of the divided frequent 1-item sets by using the distributed stored data as an input.

The generating of the bitmap chunk may include receiving the corresponding data by dividing the corresponding data by line; And a bit number corresponding to the number of lines, and determining the bit vector to have a first logical value when the data exists in the line and a second logical value when there is no data in the line.

The distributed system for frequent pattern mining according to one side determines a candidate (k + 1) -item set using a frequent k-item set, and uses the bitmap chunks distributed in the distributed system to store the candidate (k +). 1) -mapper module for calculating partial supports of the set of items; And a reducer module that adds the partial support of the candidate (k + 1) -item set to determine a frequent (k + 1) -item set.

The reducer module may broadcast the frequent (k + 1) -item set to the distributed system.

Each of the mappers included in the mapper module loads a corresponding bitmap chunk and is included in the candidate (k + 1) -item set through an AND operation of bit vectors included in the corresponding bitmap chunk. Compute bit vectors of candidate (k + 1) -items, and calculate partial support of the candidate (k + 1) -item set based on the corresponding bit vectors.

The reducer module selects a candidate (k + 1) -item whose support, which is the sum of partial support of the plurality of candidate (k + 1) -items included in the candidate (k + 1) -item set, is greater than or equal to a predetermined minimum support. It can be included in the frequent (k + 1) -item set.

Partial support values corresponding to the same candidate (k + 1) -item among a plurality of candidate (k + 1) -items included in the candidate (k + 1) -item set may be delivered to the same reducer in the reducer module. Can be.

The large-scale data frequent pattern mining system and method using the distributed system according to the present invention, as well as data smaller than the main memory, through a pre-computation phase not proposed in the prior art, the memory of a single machine or It is possible to process transaction data and candidate item set data larger than a disk with scalable performance according to the size of the data and the number of machines included in the system.

That is, even when processing data larger than main memory by suggesting a method of loading bitmap chunks and candidate item set data, which are transaction data, into each machine without a memory problem and generating an intermediate result during a mining operation. It is possible to handle frequent pattern mining with good performance.

Therefore, the present invention is applied to a wide variety of data processing such as network services, web services, business intelligence, life sciences, neurosciences, and the like by using pattern outputs obtained from large data, so that data streaming processing using distributed systems can be realized. .

The effects of the present invention are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating a data frequent pattern mining system using a distributed system according to an exemplary embodiment.
2 is a diagram illustrating an overall operation of a data frequent pattern mining system using a distribution system according to an exemplary embodiment.
3 is a flowchart illustrating a pretreatment method according to an embodiment.
4 illustrates a preprocessing step according to one embodiment.
5 is a flowchart illustrating a frequent pattern mining method of a Hadoop-based distributed system according to an embodiment.
FIG. 6 illustrates a method of obtaining a frequent two-item set using a bitmap chunk according to an embodiment. FIG.
FIG. 7 illustrates a method of obtaining a frequent 3-item set using a bitmap chunk according to an embodiment. FIG.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments so that the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, and substitutes for the embodiments are included in the scope of rights.

The terminology used herein is for the purpose of description and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same reference numerals and duplicate description thereof will be omitted. In the following description of the embodiment, when it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

Data mining is the systematic and automatic discovery of statistical rules or patterns in large amounts of stored data. Patterns from the data mining process can be used to analyze the characteristics of data or to obtain the information necessary for decision making, and to analyze various real data such as web data, customer data, product data, and traffic and communication data. have.

Pattern mining is one of the major areas of data mining research, and it is a technique for finding important association rules, or patterns, based on support rates. Frequent patterns are patterns that occur frequently in a data set. Frequent pattern mining can find all patterns in the database that have support above the support threshold.

Frequent pattern mining can find recurring relationships in a given data set. Market basket analysis, an example of frequent pattern mining, is shown in Table 1.

Customer number Beer (i1) Peanut (i2) Pepper (i3) Coffee (i4) Egg (i5) Milk (i6) t1 One One One 0 0 0 t2 One 0 One One 0 0 t3 One 0 One 0 One 0 t4 0 One 0 0 One One t5 0 0 One One 0 One

Table 1 shows the purchase or non-purchase of each product as a binary vector (data). According to Table 1, a list of five customers who bought a product can be seen. A transaction is a collection of operations in a database that are required to perform a single task. A transaction can be a logical unit of work in a database. For example, if the number and type of goods purchased are different for each customer, when a customer processes all the purchased goods, it can be said to process one transaction data. It can be called an item set. For example, {beer, peanut} can be a two-item set.

An item set having a minimum support or more may be referred to as a frequent itemset. Referring to Table 1, when the minimum support is 3, the {beer, pepper} item set that appears three or more times in the transaction data may be a frequent itemset.

The frequent k-item set may be an item set having k items among the item sets of data repeated over a predetermined minimum support degree in the data. For example, the {beer, pepper} item set may be a frequent two-item set because it appears at least three times in transaction data simultaneously and is a frequent item set having two items. The {beer} and {pepper} item sets may be frequent 1-item sets because they appear more than three times in the transaction data and are frequent item sets with one item.

Referring to Table 1, when a database consists of a total of n transaction data and contains a total of m items, the item set of all products is i = {i1, i2, ..., im} and all customers A set of transactions can be expressed as T = {t1, t2, ..., tn}.

1 is a block diagram illustrating a frequent pattern mining system using a distribution system according to an exemplary embodiment.

Referring to FIG. 1, a data frequent pattern mining system 100 using a distributed system according to an exemplary embodiment may include a data replication controller for replicating frequent (k-1) -item sets calculated in a previous step to all machines in a distributed system. 120 and a plurality of machines 140, 150, and 160. The machine 140 may include a mining module 141 and a disk 143 that obtain partial support of a set of candidate items using the stored bit vector data, the disk 143 being a dictionary 145. ), It may include a bitmap chunk 1 147. The data frequent pattern mining system 100 using a distributed system may include a mining module controller 110 for managing mining modules in a distributed system and a synchronization manager 130 for synchronizing partial support of the calculated candidate item set. .

On each machine, the mapper can create a set of candidate items and use bitmap chunks to get partial support for the set of candidate items. The reducer can obtain the frequent item set by adding the partial support and obtaining the support of the candidate item set for the entire data.

The system according to an embodiment may include a Hadoop based distributed system.

Each configuration of the data frequent pattern mining system 100 using the distributed system of FIG. 1 will be described in detail below.

2 is a diagram illustrating an overall operation of a data frequent pattern mining system using a distributed system according to an exemplary embodiment.

Most of the conventional pattern mining methods use a method of reducing computational amount by generating and reusing intermediate data. Conventional pattern mining methods tend to fail the mining operation itself due to insufficient memory when the size of the transaction database increases or grows medium.

Referring to FIG. 2, a data frequent pattern mining system using a distributed system according to an embodiment may include a pattern obtained in advance from a large-capacity database that is distributed and stored using a MapReduce technique to solve the above-mentioned problem. the through a bit vector (bit vector), the pre-treatment step (pre-computation phase) for distributed storage to produce 210 and finding the frequency pattern using the stored bit vector (finding F _k phase) (250) corresponding to Can handle data larger than a single machine's memory or disk.

According to an embodiment, in order to store bit vectors corresponding to previously obtained patterns in a distributed storage system, a data format called bitmap chunk is proposed. The input transaction database D is {D ₁ , D ₂ ,... , D _n } can be assumed to be distributed and stored. Each machine can load the partitioned data D _i into memory and generate a bitmap chunk TC _i associated with D _i . Bitmap chunks may follow a key-value structure. The key is a precalculated set of items and the value may be a bit vector associated with the set of items.

In the preprocessing step 210, the bitmap chunk 240 related to the data 220 allocated to each mapper may be generated using the mapper 230 and stored on each disk.

In the step 250 of finding frequent patterns, the mapper 230 generates candidate item sets, reads the bitmap chunk 240 associated with itself to obtain partial support, and then sends it to the reducer 260 to sum up the partial support. The frequent pattern 270 can be obtained.

In operation 250, the bitmap chunk 240 stored in each disk may not be moved through the network. Instead, network overhead can be negligibly small because only frequent patterns 270 of a size smaller than the bitmap chunk are moved through the network.

3 is a flowchart illustrating a pretreatment method according to an exemplary embodiment.

A preprocessing method according to an embodiment may include receiving data that is distributed and stored, outputting a bitmap chunk composed of bit vectors corresponding to predetermined patterns by receiving the data that is distributed and stored, and a bitmap. Distributedly storing the chunks on each disk of the distributed system.

Instead of directly mining the input transaction database in the step of finding frequent patterns, the preprocessing step corresponds to predetermined patterns suitable for data mining, because the data is mined using bitmap chunks created in the preprocessing step and stored on each disk. You can aim to generate bitmap chunks.

In order to generate bitmap chunks corresponding to predetermined patterns suitable for data mining, the predetermined patterns may be divided into a predetermined number of pieces, and thus, the bitmap chunks may also be divided by a predetermined number.

According to one embodiment, the predetermined pattern may be generated using a frequent 1-item set.

Referring to FIG. 3, the preprocessing method according to an exemplary embodiment includes loading 310 stored data, scanning 315 distributed data, and obtaining a frequent 1-item set 330. Generating 335, combining all sets of items P for each piece (340), calculating a bit vector of P (345), and storing a bit vector of P for a partition (350). It may include.

Obtaining a frequent 1-item set 330 may include obtaining 320 a partial support of the 1-item set, and adding up 325 of the partial support of the 1-item set.

4 is a diagram illustrating a pretreatment step according to an exemplary embodiment.

Referring to FIG. 4, it may be assumed that the input transaction database D is divided and stored in {D ₁ 410 and D ₂ 420}. D ₁ 410 and D ₂ 420 distributed and stored on each disk are composed of three transaction data, and each includes five items {A, B, C, D, E}.

The frequent 1-item set 430 may be obtained by calculating partial support of the 1-item set and summing partial support of the 1-item set.

To obtain the partial support of the 1-item set to obtain the frequent 1-item set 430, {A} of D ₁ 410 is 2, {B} is 3, {C} is 0, and {D} is 3, {E} has a partial support of ₂ , D ₂ 420 has {A} of 2, {B} of 2, {C} of 0, {D} of 1, and {E} of 1 Have support. When the partial support of the 1-item set is summed, {A} has 4, {B} is 5, {C} is 0, {D} is 4, and {E} is 3 support.

Assuming that the minimum degree of support is three, the frequent one-item set 430 may be an item set {A, B, D, E} that is greater than or equal to three.

A bitmap chunk may be generated using the frequent 1-item set 430 according to an embodiment. Bitmap chunk 1 (TC ₁ ) 440 may be generated using D ₁ 410, and bitmap chunk 2 (TC ₂ ) 460 may be generated using D ₂ 420.

The frequent 1-item set 430 may be divided into a predetermined size to generate a plurality of pieces, and a bitmap chunk may be generated using the pieces.

For example, when the frequent one-item set F ₁ 430 is {A, B, D, E}, it may be assumed that the size Q of the piece is two. Since the size of the pieces is two, the frequent one-item set 430 can be divided into two pieces {A, B} 431, {D, E} 432. As another example, if the frequent 1-item set F ₁ is {A, B, C, D, E, F}, and the size of the piece (Q) is 3, then {A, B, C}, {D, E, F} It can be divided into two pieces.

The mapper 480 and 490 of each machine according to an embodiment inputs distributed data, combines all frequent 1-item sets included in each piece for each piece, and previews a bit vector of the combined item set in advance. Can compute and output bitmap chunks.

For example, inputting D1 410 into the mapper 480 corresponds to the combination 441, 442 of all frequent 1-item sets included in the pieces 431, 432 for each piece 431, 432. The bitmap chunk 1 (TC ₁ ) 440 including the bit vectors 451 and 452 may be output. The mapper 480 uses {A} and {B} {A} and {B}, which is a frequent one-item set included in the {A, B} 431 fragment, based on D ₁ 410. A, B} item sets 441 and bit vectors 451 corresponding to the item sets 441 may be generated. {D, E} 432 fragments are similarly based on D ₁ 410, using {D} and {E}, the frequent 1-item sets included in {A, B} 431 fragments. , A {E} {D, E} item set 442 and a bit vector 452 corresponding to each may be generated.

Similarly, with D2 420 as input to mapper 490, bit vectors corresponding to the combination 461, 462 of all frequent 1-item sets included in pieces 431, 432 for pieces 431, 432, respectively. A bitmap chunk 2 (TC ₂ ) 460 including 471 and 472 may be output.

The bit vector of the bitmap chunk consists of the number of bits corresponding to the number of transactions of D _i (| D _i |), and has a first logical value if the data exists in the transaction and a second logical value if it does not exist. Can be determined. The first logical value may be 1 and the second logical value may be 0. For example, since {A} of bitmap chunk 1 440 exists only in the first and third transactions of D ₁ , the bit vector of {A} may be 101, and in the case of {D}, D ₁ The bit vector of {D} may be 111 because it exists in every transaction.

The support of the item set {D, E} is smaller than the minimum support, but the present invention does not remove the corresponding bit vector from the bitmap chunk. This is because the cost of removing the bit vector of the item set is high.

Each bitmap chunk TC _i according to an embodiment may be divided horizontally into several pieces. The key part (set of items of TC _i ) may be referred to as a dictionary. All bitmap chunks of a distributed system may have the same dictionary. The size of the dictionary depends on the number of pieces. The size of the dictionary may mean the height of the bitmap chunk. The number of pieces may be the size of the frequent 1-item set divided by the size of the pieces. The size of a dictionary can be obtained by multiplying the number of fragments by the number of subsets excluding the empty set of frequent 1-item sets included in each fragment. This may be represented as in Equation 1.

For example, if F1 = {A, B, C, D, E, F}, Q = 2, the size of the dictionary may be 6/2 × (2 ² −1) = 9.

The width of the bitmap chunk transaction number of D _i may be determined according to (| | D _i). Therefore, the present invention can adjust the size of the bitmap chunk through Q and | D _i |.

The bitmap chunks 440, 460 generated by the preprocessing step may be distributed and stored on a disk of a machine including the mappers 480, 490.

According to an embodiment of the present invention, the finding of the frequent pattern F _k may be performed by finding two sets of frequent items at each level of the item set lattice by repeating the two steps of the candidate generation and test method until all the frequent item sets are found. Can be. In each iteration, in the Map phase, each Mapper may generate a candidate k-item set C _k and calculate partial support. The reduce step may actually add partial support to find a frequent set of k-items. Different reducers can find different F _k . In order for the next mapper to share the complete F _k , we can copy the frequent k-item set F _k that results from the reducer to all machines. In the step of finding the frequent pattern F _k , the bitmap chunks stored in the distributed storage system may not be moved through the network. Instead, network overhead can be negligibly small because only F _{k, which} is smaller than the bitmap chunk, is moved across the network.

5 is a flowchart illustrating a frequent pattern mining method of a Hadoop-based distributed system according to an embodiment.

Referring to FIG. 5, the frequent pattern mining method of the Hadoop-based distributed system may start from the preprocessing step 510. The preprocessing step 510 may generate bitmap chunks for the transaction database and store them on disk of each machine.

If we store the bitmap chunks, we can find the frequent patterns F _k in order. First, it can be found in order from the frequent 2-item set (F ₂ ) with K = 2 (515).

In the previous step, the frequent k-item set F _k that results from the reducer may be copied to all machines. The copied frequent k-item set F _k may be loaded and used to determine a candidate (k + 1) -item set C _{k + 1} (520).

If the candidate (k + 1) -item set C _{k + 1-} no longer exists, the frequent pattern mining may be terminated (530).

If the candidate (k + 1) -item set C _{k + 1-} exists, the support of the candidate (k + 1) -item set is determined using the AND operation of the bitmap chunks distributed on each disk of the Hadoop-based distributed system. Can be calculated (535).

If the support of the candidate (k + 1) -item set is greater than or equal to the minimum support, the candidate (k + 1) -item set may be determined as the frequent (k + 1) -item set (545).

If the support of the candidate (k + 1) -item set is less than the minimum support, and if the candidate (k + 1) -item set is determined to be the frequent (k + 1) -item set, then update k to k + 1 The step may be repeated (550).

According to an embodiment, the present invention may efficiently calculate partial support using a bitmap chunk. Because bitmap chunks are stored in key-value form, they can be easily loaded into memory in the form of a hash map, making it easy to access the bit vector of any set of items x. have. Given a set of candidate items x, we can easily divide x into subitem sets P (x) that completely belong to each fragment without multiple overlaps. For example, given a candidate item set x = {A, B, E}, P (x) becomes {{A, B}, {E}}. Each mapper takes a bitwise AND operation on bit vectors TC _i [{A, B}] and TC _i [{E}] within a given bitmap chunk TC _i . Then, by counting the number of 1s in the result bit vector, the mapper can easily find the partial support of {A, B, E}.

According to an embodiment, the same set of candidate items may be generated in each machine, and partial support may be independently obtained using different bitmap chunks.

6 is a diagram illustrating a method of obtaining a frequent two-item set using a bitmap chunk according to an embodiment.

Referring to FIG. 6, when the minimum support is 3, a method of obtaining a frequent 2-item set F ₂ 655 is shown. It can be assumed that the number of reducers is one. The frequent 1-item set F ₁ = {A, B, D, E} 610 can be copied to all mappers. Each mapper 613, 614 may generate six candidate two-item sets 620 using the frequent one-item set F ₁ = {A, B, D, E} 610.

The bitmap chunks created and stored in the preprocessing step may be used to obtain partial support of the candidate item set. The mapper M ₁ 613 may obtain partial support using the bitmap chunk 611 to obtain partial support of the candidate two-item set. For example, a bit AND operation between TC ₁ [{B}] and TC ₁ [{D}] may be performed to obtain partial support of {B, D}. Since the result bit vector is 111, mapper M ₁ 613 may output ({B, D}, 3). Similarly, the mapper M ₂ 614 may output a bitwise ({B, D}, 1) by performing a bitwise AND operation between TC ₂ [{B}] and TC ₂ [{D}].

The candidate two-item sets 620 and 630 and the partial support 640 and 645 shuffle the candidate (k + 1) -item sets for which the partial support is calculated, with the same elements, and reducer 650. Can be combined by. A candidate two-item set with support greater than 3, which is the minimum support, may be determined as the frequent two-item set 655. For example, since the support of the candidate item set {B, D} is 4, which is larger than the minimum support of 3, it may be determined as the frequent 2-item set.

The support of the candidate item set {A, D} and {D, E} may not be output because it is smaller than the minimum support.

The determined frequent (k + 1) -item set can be copied to all mining modules included in the Hadoop-based distributed system. For example, the frequent two-item set 655 may be stored in a distributed storage system and copied to all machines.

You can repeat the above steps to get a frequent 3-item set.

FIG. 7 illustrates a method of obtaining a frequent 3-item set using a bitmap chunk according to an embodiment.

Referring to FIG. 7, when the minimum support is 3, a method of obtaining a frequent 3-item set F ₂ 755 is shown. It can be assumed that the number of reducers is one. The frequent two-item set F ₂ = {{A, B}, {A, E}, {B, D}, {B, E}} 710 obtained in FIG. 6 may be copied to all mappers. Each mapper 713, 714 uses candidate two-item set F ₂ = {{A, B}, {A, E}, {B, D}, {B, E}} 710 An item set 720 may be generated.

A frequent k-item set can be used to generate a candidate- (k + 1) item set. The method for generating the candidate- (k + 1) item set from the frequent k-item sets can be implemented in various ways.

For example, the candidate- (k + 1) item set may be generated only when the same item exists in the same line of the frequent k-item set. Referring to the embodiment of FIG. 7, the frequent two-item set {A, B} and {A, E} have {A, B} and {A, E} because the same item A exists in the first line. Can generate candidate 3-item sets {A, B, E}. However, since the frequent two-item sets {A, B} and {B, D} do not have the same items on the first line, they use {A, B} and {B, D} to form the candidate three-item sets. It cannot be created. Using the frequent two-item set, candidate-item sets {A, B, E}, {B, D, E} can be generated.

The bitmap chunks created and stored in the preprocessing step may be used to obtain partial support of the candidate item set. The mapper M ₁ 713 may obtain partial support using the bitmap chunk 711 to obtain partial support of the candidate 3-item set. For example, a bit AND operation between TC ₁ [{B}] and TC ₁ [{D, E}] may be performed to obtain partial support of {B, D, E}. Since the result bit vector is 101, the mapper M ₁ 713 may output ({B, D, E}, 2). Similarly, the mapper M ₂ 714 may perform a bitwise AND operation between TC ₂ [{B}] and TC ₂ [{D, E}] to output ({B, D, E}, 0).

The candidate three-item set 720, 730 and partial support 740, 745 may be shuffled and combined by the reducer 750. A candidate three-item set with support greater than 3, which is the minimum support, may be determined as the frequent three-item set 755. For example, since the support of the candidate item set {A, B, E} is 3, which is equal to the minimum support of 3, it may be determined as the frequent 3-item set.

The support of the candidate item set {A, D, E} may not be output because it is smaller than the minimum support.

The determined frequent (k + 1) -item set can be copied to all mining modules included in the Hadoop-based distributed system. For example, the frequent three-item set 755 may be stored in a distributed storage system and copied to all machines.

You can repeat the above steps to get a frequent 4-item set.

The candidate 4-item set may be generated using the frequent 3-item set, but since the frequent 3-item set is {A, B, E} one, the candidate 4-item set cannot be generated. If there is no candidate 4-item set, the frequent pattern mining may be terminated.

8 is a flowchart illustrating a method of obtaining partial support of a frequent pattern mining method using a dispersion system according to an exemplary embodiment.

Referring to FIG. 8, a method of obtaining partial support of a frequent pattern mining method using a distributed system includes dividing each candidate item set into partial item sets completely belonging to an arbitrary fragment without overlapping (step 810), C _k . Extracting a bit vector associated with the partial item sets from the bitmap chunk (820), obtaining a partial support of C _k using the bit vector (830), adding the partial support of C _k obtained from each machine, Obtaining support of C _k for the data may include 840.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described with reference to the accompanying drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the following claims.

Claims (19)

In the frequent pattern mining method of a distributed system,
Determining a candidate (k + 1) -item set using the frequent k-item set;
Calculating partial supports of the candidate (k + 1) -item set using bitmap chunks distributed in the distributed system;
Summing partial supports of the candidate (k + 1) -item set; And
Determining a frequent (k + 1) -item set based on the sum of the partial supports
Frequency pattern mining method of distributed systems.
The method of claim 1,
Broadcasting the frequent (k + 1) -item set to the distributed system
Frequency pattern mining method of the distributed system further comprising a.
The method of claim 1,
The distributed system includes a plurality of mappers corresponding to the bitmap chunks,
Computing partial supports of the candidate (k + 1) -item set
Loading each of the bitmap chunks into a corresponding mapper;
In each of the plurality of mappers, bit vectors of candidate (k + 1) -items included in the candidate (k + 1) -item set are calculated through an AND operation of bit vectors included in a corresponding bitmap chunk. step; And
Calculating, in each of the plurality of mappers, partial support of the candidate (k + 1) -item set based on corresponding bit vectors;
Frequency pattern mining method of a distributed system comprising a.
The method of claim 1,
Determining the frequent (k + 1) -item set
Among the candidate (k + 1) -item sets, a candidate (k + 1) -item having a support sum of partial support of a plurality of candidate (k + 1) -items or more is greater than or equal to a predetermined minimum support rate (k + 1) -item (k Steps to include in your +1) -item set
Frequency pattern mining method of a distributed system comprising a.
The method of claim 1,
If the candidate (k + 1) -item set does not exist, ending the frequent pattern mining
Frequency pattern mining method of the distributed system further comprising a.
The method of claim 1,
Summing the partial supports
The partial support values such that partial support values corresponding to the same candidate (k + 1) -item among a plurality of candidate (k + 1) -items included in the candidate (k + 1) -item set are transmitted to the same reducer Step to shuffle the ears
Frequency pattern mining method of the distributed system further comprising a.
The method of claim 1,
Determining the candidate (k + 1) -item set
Receiving the frequent k-item set broadcast to the distributed system; And
Determining candidate (k + 1) -items by combining a plurality of frequent k-items included in the frequent k-item set
Including, frequent pattern mining method of the distributed system.
The method of claim 1,
Dividing a plurality of items included in data distributed and stored in the distributed system in a predetermined manner;
Determining patterns based on the divided items; And
Generating a bitmap chunk for a corresponding mapper by determining bit vectors representing the patterns included in the corresponding data, using a mapper corresponding to each of the distributed stored data.
Further comprising, the frequent pattern mining method of the distributed system.
The method of claim 8,
The bitmap chunk has a key-value structure, the key is a set of items corresponding to the patterns, and the value is a frequent pattern of a distributed system including the bit vectors. Mining method.
The method of claim 8,
The frequent pattern mining method of a distributed system in which the bitmap chunks have the same key.
The method of claim 8,
The above items
A method of mining frequent patterns in a distributed system, corresponding to a frequent 1-item set.
The method of claim 11,
Generating the bitmap chunk
Determining bit vectors corresponding to the combination of the divided frequent 1-item sets by using the distributed stored data as an input;
Frequency pattern mining method of a distributed system comprising a.
The method of claim 8,
Generating the bitmap chunk
Receiving the data by dividing the corresponding line by line; And
Determining the bit vector having a number of bits corresponding to the number of lines and having a first logical value if the data is present in the line and a second logical value if the data is not present in the line;
Frequency pattern mining method of a distributed system comprising a.
A computer program stored in a medium in combination with hardware to carry out the method of any one of claims 1 to 13.
In a distributed system for frequent pattern mining,
A mapper that determines a candidate (k + 1) -item set using a frequent k-item set, and computes partial support of the candidate (k + 1) -item set using bitmap chunks distributed in the distributed system. module; And
Reducer module that adds partial support of the candidate (k + 1) -item set to determine a frequent (k + 1) -item set
Including, distributed system.
The method of claim 15,
The reducer module
Broadcast the frequent (k + 1) -item set to the distributed system.
The method of claim 15,
Each of the mappers included in the mapper module
Load the corresponding bitmap chunk,
Calculating bit vectors of candidate (k + 1) -items included in the candidate (k + 1) -item set by performing an AND operation on the bit vectors included in the corresponding bitmap chunk,
Calculating partial support of the candidate (k + 1) -item set based on corresponding bit vectors,
Distributed system.
The method of claim 15,
The reducer module
Among the candidate (k + 1) -item sets, a candidate (k + 1) -item having a support sum of partial support of a plurality of candidate (k + 1) -items or more is greater than or equal to a predetermined minimum support rate (k + 1) -item (k +1)-to include in the set of items,
Distributed system.
The method of claim 15,
Partial support values corresponding to the same candidate (k + 1) -item among a plurality of candidate (k + 1) -items included in the candidate (k + 1) -item set are transmitted to the same reducer in the reducer module. ,
Distributed system.

Images