CN110222090A - A kind of mass data Mining Frequent Itemsets - Google Patents

Abstract

The present invention provides a method for mining frequent itemsets of massive data, comprising: mining an original transaction data set TO by using a frequent _itemset mining algorithm to obtain all local frequent _itemsets corresponding to the original transaction data set TO; scanning the original transaction Data set T _O , corresponding to calculating the support count of each local frequent itemset obtained above on the original transaction data set T _O , filter the obtained local frequent itemsets, and obtain each local part whose support degree is not less than ω Frequent itemsets, and write the obtained local frequent itemsets and the corresponding support counts calculated into the file F _qf ; read the newly added transaction data set T _Δ , and judge the newly added transaction data set T _Δ Whether it is empty, then perform frequent itemset mining based on whether the newly added transaction data set T _Δ is empty. The present invention reuses the file F _qf , the set ST _CAD and the array cnt _Δ in the whole mining process, which reduces the computational cost to a certain extent, thereby improving the mining rate of frequent itemsets.

Description

A method for mining frequent itemsets in massive data

technical field

The invention relates to the technical field of data mining, in particular to a method for mining frequent itemsets of massive data.

Background technique

Frequent itemset mining has always been one of the most active areas in data mining. It has a very wide range of applications in real life, for example, it is widely used in data mining, software error detection, spatiotemporal data analysis, biological analysis and many other research fields. Due to its practical significance, frequent itemset mining has attracted extensive attention.

In the field of data storage, data is usually stored in read-only/add-only mode, and the entire transaction data set can be divided into two parts: the original transaction data set and the new transaction data set. At a certain time or condition, the data in the new transaction data set is merged into the original transaction data set. At this time, the data in the original transaction data set is increasing, and the data in the new transaction data set is emptied because it is merged into the original transaction data set. , when new data is written, the new data is written into the new transaction data set, and then when a certain time or condition is met again, the newly written data in the new transaction data set is merged into the original transaction again Data set, and the new transaction data set continues to be used to wait for the storage of new data, and so on. It can be seen that in the read-only/add-only mode for storage, the original transaction data set is always composed of the original transaction data set and the newly added transaction data set.

Over the years, researchers at home and abroad have proposed many related algorithms. Existing algorithms can be divided into two categories: those based on candidate generation and those based on pattern growth. Algorithms based on candidate generation first generate candidate itemsets, then scan the database to verify the candidate itemsets and identify frequent itemsets in them. Furthermore, candidate generation-based algorithms also exploit anti-monotonicity to clip the search space. However, such algorithms require multiple passes to scan the database, which will bring high I/O overhead when dealing with massive amounts of data. The algorithm based on pattern growth does not generate candidate itemsets directly, but it builds a special tree-based data structure to save the necessary information of frequent itemsets in the database. By using this data structure, frequent itemsets can be efficiently calculated. However, this kind of algorithm is very complicated to construct data structure, and when processing massive data, the memory requirement usually exceeds the available memory, so the data structure cannot be stored in memory. build correctly in .

To this end, the present invention provides a method for mining frequent itemsets of massive data, which is used to realize the mining of frequent itemsets of massive data in the read-only/add-only mode storage mode.

SUMMARY OF THE INVENTION

In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for mining frequent itemsets of massive data, which is used to realize the mining of frequent itemsets of massive data in the read-only/add-only mode storage mode, so as to improve the mining of frequent itemsets of massive data. mining rate.

The present invention provides a method for mining frequent itemsets of massive data. The method for mining frequent itemsets of massive data is used to mine frequent itemsets that satisfy the global minimum support minsup in the total transaction data set T. The global minimum support minsup is the minimum support degree on the preset total transaction data set T;

The total transaction data set T includes the original transaction data set T _O and the newly added transaction data set T _Δ ;

The massive data frequent itemset mining method includes the following steps:

The frequent _itemset mining algorithm is used to mine the original transaction data set TO, and obtain all the local frequent _itemsets corresponding to the original transaction data set TO;

Scan the original transaction data set T _O , correspondingly calculate the support count of each local frequent itemset obtained above on the original transaction data set T _O , and filter the obtained local frequent itemsets according to the local minimum support ω , obtain each local frequent item set whose support degree is not less than ω, and write the obtained local frequent itemsets whose support degree is not less than ω and the corresponding calculated support degree count into the file F _qf correspondingly;

Read the new transaction data set T _Δ and determine whether the new transaction data set T _Δ is empty:

Yes, then according to the number n of transactions in the total transaction data set T and the global minimum support minsup, the local frequent itemsets in the file F _qf are filtered, and the filtered support count is not less than the global The local frequent itemsets of the minimum support count n×minsup are output, and the outputted local frequent itemsets are all the frequent itemsets that satisfy the global minimum support minsup on the total transaction data set T;

If not, the incremental update method is used to mine frequent itemsets on the total transaction data set T;

Wherein, the local minimum support degree ω is a preset minimum support degree on the original transaction data set T _O , and the local minimum support degree ω<global minimum support degree minsup.

Further, the incremental update method includes the steps:

Scan the newly added transaction data set T _Δ , calculate the support count of each item set in the newly added transaction data set T _Δ on the newly added transaction data set T _Δ , and add the item sets in the newly added transaction data set T _Δ , and The calculated support count of each item set in the newly added transaction data set T _Δ on the newly added transaction data set T _Δ is stored in the array cnt _Δ correspondingly, and the maximum support count in the array cnt _Δ is recorded as mas _Δ ;

Scan the current file F _qf , for each local frequent itemset scanned from the current file F _qf , determine whether it is a frequent item set that satisfies the preset global minimum support minsup on the total transaction data set T, respectively, If the judgment result is yes, output the currently scanned local frequent itemset, and record the output local frequent itemset as the first frequent itemset; if the judgment result is no, then based on the mas _Δ , determine the current Whether the scanned local frequent itemsets are not the frequent itemsets that satisfy the preset global minimum support minsup on the total transaction data set T: if not, then count the currently scanned local frequent itemsets and the corresponding support, Correspondingly written into the collection ST _CAD ;

Based on the above-mentioned array cnt _Δ , corresponding to the support counts of each local frequent item set existing in the newly added transaction data set T _Δ in the calculation and update set ST _CAD on the total transaction data set T, the updated set ST _CAD is obtained;

Traverse the updated set ST _CAD , and respectively judge whether the local frequent itemsets in the currently traversed updated sets ST _CAD are frequent itemsets that satisfy the preset global minimum support minsup on the total transaction data set T , if the judgment result is yes, then correspondingly output the corresponding local frequent itemsets in the updated set ST _CAD currently traversed;

Then judge whether the expression (n _O ×ω-1)+mas _Δ <(n×minsup) holds:

If yes, the frequent itemset mining ends;

If not, continue to mine new frequent itemsets in the newly added transaction data set T _Δ , and output the new frequent itemsets mined;

Wherein, the new frequent itemset, whose support count on the original transaction data set T _O is greater than zero, satisfies the global minimum support minsup on the total transaction data set T, and is different from the above output all frequent itemsets;

n _O is the number of transactions in the newly added transaction data set T _Δ .

Further, mining new frequent itemsets in the newly added transaction data set T _Δ , and outputting the mined new frequent itemsets, including:

by formula Calculate the minimum support minsup _Δ of the newly added transaction data set T _Δ , where n _Δ is the number of transactions in the newly added transaction data set T _Δ ;

Divide the transactions in the newly added transaction data set T _Δ into multiple target transaction data sets;

Using the Eclat algorithm, perform local frequent itemsets mining on each target transaction data set, and obtain all local frequent itemsets corresponding to each target transaction data set that satisfy the minimum support minsup _Δ calculated above;

Denote the set of all local frequent itemsets corresponding to each target transaction data set obtained above as LF _Δ , traverse and delete the local frequent itemsets in the set LF _Δ that have appeared in the file F _qf to obtain the candidate set GF _Δ ; And based on the above-mentioned array cnt _Δ , the support counts of each local frequent item set in the candidate set GF _Δ in the newly added transaction data set T _Δ are correspondingly stored in the candidate set GF _Δ ;

Scan the current original transaction data set _TO , increase and update the support count of the local frequent itemsets in the candidate set GF _Δ , and obtain a new candidate set GF _Δ ;

Scan the above-mentioned new candidate set GF _Δ , correspondingly determine whether the local frequent itemsets in the new candidate sets GF _Δ currently scanned are frequent itemsets that satisfy the preset global minimum support minsup, if the judgment result is yes , then correspondingly output the local frequent itemsets in each currently scanned candidate set GF _Δ .

Further, the Eclat algorithm is used to perform local frequent itemset mining on each target transaction data set, and all local frequent itemsets corresponding to each target transaction data set that satisfy the minimum support minsup _Δ obtained by the above calculation are obtained, including :

P0, traverse each target transaction data set;

P1. For the current traversed target transaction data set:

P11. Use the Eclat algorithm to calculate and obtain the candidate frequent k-itemsets corresponding to the currently traversed target transaction data set, and when the generated candidate frequent k-items satisfies the minimum support minsup _Δ calculated above, record The generated candidate frequent k-itemsets are frequent k-itemsets and are stored in the set LF _k,Δ , k≥1;

P12. Generate a candidate frequent (k+1)-item set by merging the two frequent k-itemsets in the LF _k,Δ , and the generated candidate frequent (k+1)-item set satisfies the above calculation result When the minimum support minsup _Δ , the generated candidate frequent (k+1)-itemsets are recorded as frequent (k+1)-itemsets and stored in the set LF _k+1,Δ , wherein the two frequent The first k-1 items of the k-item set are the same and the last item is different;

P13. Repeat the above steps P11-P12, each time k increases by 1, until the new candidate frequent itemsets corresponding to the currently traversed target transaction data set can no longer be generated; then step P14 is performed;

P14. Continue to traverse the next target transaction data set, and repeat the above steps P11-P13 until each target transaction data set is traversed, so as to obtain all target transaction data sets that satisfy the minimum support minsup _Δ calculated above. The local frequent itemsets of .

Further, between step P11 and step P12, it also includes step S: a fine-pruning step for the set LF _k,Δ ;

Wherein, the fine pruning step for the set LF _k,Δ includes:

According to whether the first (k-1) items of the itemset are the same, obtain and divide the frequent k-item sets in the set LF _k,Δ into groups, and obtain the corresponding number of itemset groups, among which, the frequent k-items in the same item set grouping The first (k-1) items of the itemset are the same;

Count the number of frequent k-itemsets in each itemset grouping separately, and judge whether the counted quantities are equal to 1: Yes, delete the corresponding itemset grouping, and delete the corresponding items in the set LF _k,Δ The itemsets with the same frequent k-itemsets in the set grouping; in the corresponding itemset grouping, the number of frequent k-itemsets in it is equal to 1;

Correspondingly determine whether the current item set grouping is the union of any two frequent k-itemsets in the itemset grouping, which are included in the file F _qf : if yes, delete the corresponding itemset grouping, and Delete the itemsets in the set LF _k,Δ that are the same as the frequent k-itemsets in the corresponding itemset grouping.

Further, based on the above-mentioned array cnt _Δ , corresponding to the support count of each local frequent itemset existing in the newly added transaction data set T _Δ in the calculation and update set ST _CAD on the total transaction data set T, the updated Before the collection of ST _CAD , it also includes a fine-cutting step for the collection of ST _CAD ;

The fine-cutting steps of the set ST _CAD , including the streamlining steps of the first stage;

The simplified steps of the first stage include:

Traverse the file F _qf and the array cnt _Δ , and calculate the support counts of the 1-itemsets in each traversed file F _qf and the corresponding 1-itemsets in the array cnt _Δ corresponding to the same 1-item sets in the array cnt _Δ The sum of support counts, if the calculated sum of support is less than the global minimum support count n×minsup, the corresponding 1-item set in the traversed file F _qf is removed from the set ST _CAD .

Further, the fine-cutting step of the set ST _CAD also includes a second-stage streamlining step;

The simplified steps of the second stage include:

build the pip array;

Traverse the set ST _CAD reduced by the first-stage reduction step, and select two support counts in the corresponding item sets for each local frequent item set in the set ST _CAD reduced by the first-stage reduction step. The smallest item, the item pairs that form the corresponding itemset, are stored in the PIP array constructed above;

Calculate the support count of each item in the PIP array on the newly added transaction dataset T _Δ ;

Determine the sum of the support count of each item in the PIP array on the newly added transaction data set T _Δ and the support count of the corresponding item set on the newly added transaction data set T _Δ , and the global minimum support count n× The size relationship of minsup, and the local frequent itemsets corresponding to the pairs of items whose sum of the support counts is less than the global minimum support count n×minsup will be deleted from the set ST _CAD .

Further, when updating the newly added transaction data set T _Δ , and updating the original transaction data set T _O to be the difference between the original original transaction data set T _O and the original newly added transaction data set T _Δ and, and when the updated new transaction data set T _Δ is not empty, the method for mining frequent itemsets of massive data further includes the step of updating and mining;

The described steps of updating mining include:

Update the number n of transactions in the total transaction data set T, which is the sum of the number of transactions in the original original transaction data set T _O and the original new transaction data set T _Δ ;

Obtain all the local frequent item sets corresponding to the original original transaction data set T _O obtained above, and calculate the local frequent itemsets corresponding to the original original transaction data set T _O obtained on the original original transaction data set T _O The support count of , corresponding to the write file F _qf,O ;

Based on the array cnt _Δ corresponding to the original newly added transaction data set T _Δ , add and update the support counts of each local frequent itemset in the file F _qf,O on the original newly added transaction data set T _Δ ; For the local minimum support degree ω, filter the local frequent itemsets in the file F _qf,O , and obtain each local frequent item set whose support degree corresponding to the update total transaction data set T is not less than ω;

Afterwards, each local frequent item set whose support degree is not less than ω and its corresponding support degree in the file F _qf,O is written into a new file F correspondingly. _qf ;

Acquire the set LF _Δ corresponding to the original newly added transaction data set T _Δ , and delete the itemsets in the obtained set LF _Δ that exist in the file F _qf,O , correspondingly to obtain a new set LF _Δ ;

Based on the array cnt _Δ corresponding to the original newly added transaction data set T _Δ , in the new set LF _Δ , correspondingly write the item sets in the new set LF _Δ into the original newly added transaction data set T _Δ support count on ;

Then, according to the local minimum support ω, filter the itemsets in the new set LF _Δ , obtain the itemsets with the filtered support not less than ω, and use the obtained support not less than ω. the item set of ω and the support count written in the new set LF _Δ , corresponding to the new file F _qf written;

Then replace the original file F _qf with the above-mentioned new file F _qf , and replace the original original transaction data set T _O with the sum of the original original transaction data set T _O and the original newly added transaction data set T _Δ , And replace the original new transaction data set T _{Δ with the updated new transaction data set T Δ ,} based on the number n of transactions in the updated total transaction data set T, use the incremental update method to mine the above Frequent itemsets on the updated total transaction dataset T.

Further, according to the number n of transactions in the total transaction data set T and the global minimum support minsup, the local frequent itemsets in the file F _qf are filtered to obtain the filtered support count. Local frequent itemsets not less than the global minimum support count n×minsup, specifically:

Sequentially scan files F _qf ;

Determine whether the support count of the local frequent itemsets in the scanned file F _qf is greater than or equal to the global minimum support count n×minsup:

If yes, the local frequent itemsets in the scanned file F _qf are the local frequent itemsets whose filtered support count is not less than the global minimum support count n×minsup.

Further, in the described scanning of the current file F _qf , for each local frequent item set scanned from the current file F _qf , it is respectively judged whether the total transaction data set T satisfies the preset global minimum support degree. For the frequent itemset of minsup, if the judgment result is yes, output the currently scanned local frequent itemset, and record the output local frequent itemset as the first frequent itemset; if the judgment result is no, then based on the mas _Δ , to determine whether the currently scanned local frequent itemset is not necessarily a frequent itemset that satisfies the preset global minimum support minsup on the total transaction data set T: if not, the currently scanned local frequent itemsets and The corresponding support counts are written into the set ST _CAD , including:

Sequentially scan the files F _qf ;

For each local frequent item set in the scanned file F _qf , determine whether the support count of the local frequent item set in the scanned file F _qf is greater than or equal to the global minimum support count n×minsup:

If yes, output the scanned local frequent itemset, and the output local frequent itemset is the frequent itemset that satisfies the preset global minimum support minsup on the total transaction data set T;

No, then judge whether the sum of the support count of the local frequent itemsets in the scanned file F _qf and the mas _Δ is less than the global minimum support count n×minsup, and if the judgment result is yes, then the current scan to The local frequent itemsets must not be frequent itemsets that satisfy the preset global minimum support minsup on the total transaction data set T. Otherwise, the currently scanned local frequent itemsets and the corresponding scanned support counts will be written accordingly. The collection ST _CAD .

The beneficial effects of the present invention are:

(1) The method for mining frequent itemsets of massive data provided by the present invention adopts the file F _qf , the set ST _CAD , and the array cnt _Δ , and multiplexes the file F _qf , the set ST _CAD and the array cnt _Δ in the whole mining process , which avoids the traversal of the original transaction data set T _O and the new transaction data set T _Δ to a certain extent, reduces the computational overhead to a certain extent, and thus can improve the mining rate of frequent itemsets to a certain extent.

(2) The method for mining frequent itemsets of massive data provided by the present invention includes a fine-pruning step for the set ST _CAD , so that the set ST _CAD is further reduced before being used for subsequent calculations, thereby reducing I/ O overhead and computational overhead.

(3) The method for mining frequent itemsets of massive data provided by the present invention proposes a specific incremental update strategy, using existing calculation information, such as array cnt _Δ , set LF _Δ , etc., to speed up the update operation, thereby helping to improve Performance and practicability of frequent itemsets mining in massive data.

In addition, the present invention has reliable design principle and simple structure, and has a very wide application prospect.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. In other words, other drawings can also be obtained based on these drawings without creative labor.

FIG. 1 is a schematic flowchart of a method according to an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1:

FIG. 1 is a schematic flowchart of a method according to an embodiment of the present invention. Wherein, the execution body of FIG. 1 may be a computing node, a server, or a common PC. The massive data frequent itemset mining method is used to mine frequent itemsets that satisfy the global minimum support minsup in the total transaction data set T, where the global minimum support minsup is the preset minimum support on the total transaction data set T The total transaction data set T includes the original transaction data set T _O and the newly added transaction data set T _Δ .

Referring to Figure 1, the massive data frequent itemset mining method includes:

Step 110 , using the frequent itemset mining algorithm to mine the original transaction data set T _O to obtain all local frequent itemsets corresponding to the original transaction data set T _O.

Specifically, the transactions in the original transaction data set _TO can be read sequentially, and the retrieved transactions are placed in the memory buffer, and then the local frequent itemset mining algorithm is used to calculate the local frequent items on the buffer data set. Itemset, the calculated local frequent itemset is saved in a file F; after that, the buffer is emptied, and the transactions in the original transaction data set _TO continue to be read sequentially for the next iteration, and the calculated local frequent itemsets continue to be Saved in file F above. This process is repeated until all transactions in the original transaction data set _TO are read. So far, all local frequent _itemsets corresponding to the original transaction data set TO have been generated and obtained, and are stored in the in file F.

For the convenience of description, the steps corresponding to step 110 are denoted as the pre-calculation stage.

Step 120 , scan the original transaction data set T _O , correspondingly calculate the support count of each local frequent itemset obtained in the above step 110 on the original transaction data set T _O , according to the local minimum support ω, for the obtained Filter the local frequent itemsets, obtain each local frequent itemsets whose support is not less than ω, and write the obtained local frequent itemsets whose support is not less than ω and the corresponding calculated support counts into file F. in _qf .

In the specific implementation, first read all the local frequent itemsets in the file F into the memory, and then scan the original transaction data set T _O sequentially, and calculate the support count of each local frequent itemsets read into the memory above; Finally, according to the local minimum support ω, filter each local frequent item set read into the memory above, and store each local frequent item set whose support degree is not less than ω in the file F _qf .

Wherein, the storage mode of the above-mentioned file F _qf is expressed as F _qf (IS, SUP), IS represents the item set, and SUP represents the support count corresponding to the item set IS; the local frequent itemsets in the file F _qf are sorted in descending order according to the support count .

For the convenience of description, the step corresponding to step 120 is recorded as the purification stage.

Step 130, read the newly added transaction data set T _Δ , and determine whether the newly added transaction data set T _Δ is empty:

Yes, then according to the number n of transactions in the total transaction data set T and the global minimum support minsup, the local frequent itemsets in the file F _qf are filtered, and the filtered support count is not less than the global The local frequent itemsets of the minimum support count n×minsup are output, and the outputted local frequent itemsets are all the frequent itemsets that satisfy the global minimum support minsup on the total transaction data set T;

If not, the incremental update method is used to mine frequent itemsets on the total transaction data set T;

Wherein, the local minimum support degree ω is a preset minimum support degree on the original transaction data set T _O , and the local minimum support degree ω<global minimum support degree minsup.

Wherein, according to the number n of transactions in the total transaction data set T and the global minimum support minsup, the local frequent itemsets in the file F _qf are filtered, and the filtered support count is not Local frequent itemsets smaller than the global minimum support count n×minsup, specifically:

Sequentially scan files F _qf ;

Determine whether the support count of the local frequent itemsets in the scanned file F _qf is greater than or equal to the global minimum support count n×minsup:

If yes, then the local frequent itemsets in the scanned file F _qf are the local frequent itemsets whose support count is not less than the global minimum support count n×minsup (the frequent itemsets on the total transaction data set T).

Among them, when reading the newly added transaction data set T _Δ , first read a transaction in the T _Δ , and then read the items in the transaction, and then transfer to reading after the items in the transaction are read. Take the next transaction in the _TΔ . Among them, for t _Δ represents a transaction in the currently read new transaction data set T _Δ , i represents an item in the t _Δ , each time an item in the newly added transaction data set T _Δ is read, the count of i is (Initial value is 0) Increment by 1.

Preferably, the incremental update method includes steps (that is, the steps included in the incremental update method used to mine frequent itemsets on the total transaction data set T):

Scan the newly added transaction data set T _Δ , calculate the support count of each item set in the newly added transaction data set T _Δ on the newly added transaction data set T _Δ , and add the item sets in the newly added transaction data set T _Δ , and The calculated support count of each item set in the newly added transaction data set T _Δ on the newly added transaction data set T _Δ is stored in the array cnt _Δ correspondingly, and the maximum support count in the array cnt _Δ is recorded as mas _Δ ;

Sequentially scan the current file F _qf , and for each local frequent item set scanned from the current file F _qf , determine whether it is a frequent item set that satisfies the preset global minimum support minsup on the total transaction data set T respectively. , if the judgment result is yes, output the currently scanned local frequent itemset, and record the output local frequent itemset as the first frequent itemset; if the judgment result is no, then based on the mas _Δ , determine the Whether the currently scanned local frequent itemsets are not frequent itemsets that satisfy the preset global minimum support minsup on the total transaction data set T: No, then count the currently scanned local frequent itemsets and the corresponding support , correspondingly written into the set ST _CAD ;

Based on the above-mentioned array cnt _Δ , corresponding to the support counts of each local frequent item set existing in the newly added transaction data set T _Δ in the calculation and update set ST _CAD on the total transaction data set T, the updated set ST _CAD is obtained;

Traverse the updated set ST _CAD , and respectively judge whether the local frequent itemsets in the currently traversed updated sets ST _CAD are frequent itemsets that satisfy the preset global minimum support minsup on the total transaction data set T , if the judgment result is yes, then correspondingly output the corresponding local frequent itemsets in the updated set ST _CAD currently traversed;

Then judge whether the expression (n _O ×ω-1)+mas _Δ <(n×minsup) holds:

If yes, the frequent itemset mining ends;

If not, continue to mine new frequent itemsets in the newly added transaction data set T _Δ , and output the new frequent itemsets mined;

Wherein, the new frequent itemset, whose support count on the original transaction data set T _O is greater than zero, satisfies the global minimum support minsup on the total transaction data set T, and is different from the above output all frequent itemsets;

n _O is the number of transactions in the newly added transaction data set T _Δ .

Wherein, in this embodiment, the current file F _qf is scanned sequentially, and for each local frequent item set scanned from the current file F _qf , it is respectively judged whether the total transaction data set T satisfies the preset requirements. The frequent itemset of the given global minimum support minsup, if the judgment result is yes, output the currently scanned local frequent itemset, and record the output local frequent itemset as the first frequent itemset; if the judgment result is no , then based on the mas _Δ , it is determined whether the currently scanned local frequent itemset must not be a frequent itemset that satisfies the preset global minimum support minsup on the total transaction data set T: if not, then scan the current scan to The local frequent itemsets and the corresponding support counts are written into the set ST _CAD , including:

Sequentially scan the files F _qf ;

For each local frequent item set in the scanned file F _qf , determine whether the support count of the local frequent item set in the scanned file F _qf is greater than or equal to the global minimum support count n×minsup:

If yes, output the scanned local frequent itemset, and the output local frequent itemset is the frequent itemset that satisfies the preset global minimum support minsup on the total transaction data set T;

No, then judge whether the sum of the support count of the local frequent itemsets in the scanned file F _qf and the mas _Δ is less than the global minimum support count n×minsup, and if the judgment result is yes, then the current scan to The local frequent itemsets must not be frequent itemsets that satisfy the preset global minimum support minsup on the total transaction data set T. Otherwise, the currently scanned local frequent itemsets and the corresponding scanned support counts will be written accordingly. The collection ST _CAD .

It should be noted that the local frequent itemsets in the file F _qf in the present invention are divided into three parts: (1) the part of frequent itemsets that absolutely belong to the total transaction data set T, (2) the part that absolutely does not belong to the total transaction data set T The part of frequent itemsets of (3) may belong to the part of frequent itemsets of the total transaction dataset T. visible for t represents the currently read local frequent itemset, if t has satisfied the global minimum support minsup, then t is the frequent itemset of the total transaction data set T; it is assumed that all transactions in the T _Δ include the t, but t still cannot satisfy the global minimum support minsup, then t must not be the frequent itemset of the total transaction data set T; in other cases, t may be the frequent itemset of the total transaction data set T, which needs further verification. Save t, which may be frequent itemsets, in the set ST _CAD . It can be seen that, based on the classification of frequent itemsets in the entire total transaction data set T, the present invention mines the frequent itemsets in the total transaction data set T by category, which helps to improve the mining efficiency to a certain extent.

Preferably, in this embodiment, mining a new frequent itemset in the newly added transaction data set T _Δ , and outputting the mined new frequent itemsets, includes:

by formula Calculate the minimum support minsup _Δ of the newly added transaction data set T _Δ , where n _Δ is the number of transactions in the newly added transaction data set T _Δ , n is the number of transactions in the total transaction data set T, and n _O is the original transaction data the number of transactions in the set _TO ;

Divide the transactions in the newly added transaction data set T _Δ into multiple target transaction data sets;

Using the Eclat algorithm, perform local frequent itemsets mining on each target transaction data set, and obtain all local frequent itemsets corresponding to each target transaction data set that satisfy the minimum support minsup _Δ calculated above;

Denote the set of all local frequent itemsets corresponding to each target transaction data set obtained above as LF _Δ , traverse and delete the local frequent itemsets in the set LF _Δ that have appeared in the file F _qf to obtain the candidate set GF _Δ ; And based on the above-mentioned array cnt _Δ , the support counts of each local frequent item set in the candidate set GF _Δ in the newly added transaction data set T _Δ are correspondingly stored in the candidate set GF _Δ ;

Scan the current original transaction data set _TO , increase and update the support count of the local frequent itemsets in the candidate set GF _Δ , and obtain a new candidate set GF _Δ ;

Scan the above-mentioned new candidate set GF _Δ , correspondingly determine whether the local frequent itemsets in the new candidate sets GF _Δ currently scanned are frequent itemsets that satisfy the preset global minimum support minsup, if the judgment result is yes , then correspondingly output the local frequent itemsets in each currently scanned candidate set GF _Δ .

Among them, in this embodiment, the Eclat algorithm is used to perform local frequent itemset mining on each target transaction data set, and all local parts corresponding to each target transaction data set that satisfy the minimum support minsup _Δ obtained by the above calculation are obtained. Frequent itemsets, including:

P0, traverse each target transaction data set;

P1. For the current traversed target transaction data set:

P11. Use the Eclat algorithm to calculate and obtain the candidate frequent k-itemsets corresponding to the currently traversed target transaction data set, and when the generated candidate frequent k-items satisfies the minimum support minsup _Δ calculated above, record The generated candidate frequent k-itemsets are frequent k-itemsets and are stored in the set LF _k,Δ , k≥1;

P12. Generate a candidate frequent (k+1)-item set by merging the two frequent k-itemsets in the LF _k,Δ , and the generated candidate frequent (k+1)-item set satisfies the above calculation result When the minimum support minsup _Δ , the generated candidate frequent (k+1)-itemsets are recorded as frequent (k+1)-itemsets and stored in the set LF _k+1,Δ , wherein the two frequent The first k-1 items of the k-item set are the same and the last item is different;

P13. Repeat the above steps P11-P12, each time k increases by 1, until the new candidate frequent itemsets corresponding to the currently traversed target transaction data set can no longer be generated; then step P14 is performed;

P14. Continue to traverse the next target transaction data set, and repeat the above steps P11-P13 until each target transaction data set is traversed, so as to obtain all target transaction data sets that satisfy the minimum support minsup _Δ calculated above. The local frequent itemsets of .

It should be noted that, see Figure 1:

Each t in the figure is the local frequent item set in the file F _qf currently scanned by the corresponding step; each t.SUP is the corresponding local frequent item set t currently scanned by the corresponding step in the file F _qf . support count;

The "s" shown in the figure is the item set in the updated set ST _CAD currently scanned in the corresponding step, and "s.SUP" is the corresponding support count of the item set s in the updated set ST _CAD . ;

"r" shown in the figure is the item set in the new candidate set GF _Δ currently scanned in the corresponding step, and "r.SUP" is the corresponding support degree of the item set r in the new candidate set GF _Δ count.

In addition, it should be noted that, with reference to Figure 1, it can be seen that:

When it is determined that the newly added transaction data set T _Δ is empty, each item set t corresponding to the output in FIG. 1 is all the frequent itemsets on the total transaction data set T mined by the method shown in the present invention;

When it is determined that the newly added transaction data set T _Δ is non-empty, each item set t, each item set s and each item set r correspondingly output in FIG. 1 are the total transaction data set T mined by the method shown in the present invention All frequent itemsets on .

In addition, it should be noted that, when the present invention is specifically implemented, the values of minsup and ω can be selected by those skilled in the art based on empirical values. For example, minsup can be 0.2, ω can be 0.1, and so on.

To sum up, the method for mining frequent itemsets of massive data provided by the present invention adopts the file F _qf , the set ST _CAD , and the array cnt _Δ , and multiplexes the file F _qf , the set ST _CAD and the array cnt _Δ in the whole mining process , which avoids the traversal of the original transaction data set T _O and the new transaction data set T _Δ to a certain extent, reduces the computational overhead to a certain extent, and thus can improve the mining rate of frequent itemsets to a certain extent.

Example 2:

Compared with Embodiment 1, the difference is that, in the method for mining frequent itemsets of massive data described in Embodiment 2, in order to further improve the mining rate of the present invention, between the above steps P11 and P12, the method further includes the following steps: S: fine-pruning step for the set LF _k,Δ ;

Wherein, the fine pruning step for the set LF _k,Δ includes:

According to whether the first (k-1) items of the itemset are the same, obtain and divide the frequent k-item sets in the set LF _k,Δ into groups, and obtain the corresponding number of itemset groups, among which, the frequent k-items in the same item set grouping The first (k-1) items of the itemset are the same;

Count the number of frequent k-itemsets in each itemset grouping separately, and judge whether the counted quantities are equal to 1: Yes, delete the corresponding itemset grouping, and delete the corresponding items in the set LF _k,Δ The itemsets with the same frequent k-itemsets in the set grouping; in the corresponding itemset grouping, the number of frequent k-itemsets in it is equal to 1;

Correspondingly determine whether the current item set grouping is the union of any two frequent k-itemsets in the itemset grouping, which are included in the file F _qf : if yes, delete the corresponding itemset grouping, and Delete the itemsets in the set LF _k,Δ that are the same as the frequent k-itemsets in the corresponding itemset grouping.

In addition, in order to further improve the mining rate of the present invention, in this embodiment, based on the above-mentioned array cnt _Δ , each local frequent itemset existing in the newly added transaction data set T _Δ in the calculation and update set ST _CAD is corresponding to the total transaction The support count on the data set T, before obtaining the updated set ST _CAD , also includes a fine-cutting step for the set ST _CAD ;

The fine-cutting steps of the set ST _CAD , including the streamlining steps of the first stage;

The simplified steps of the first stage include:

Traverse the file F _qf and the array cnt _Δ , and calculate the support counts of the 1-itemsets in each traversed file F _qf and the corresponding 1-itemsets in the array cnt _Δ corresponding to the same 1-item sets in the array cnt _Δ The sum of support counts, if the calculated sum of support is less than the global minimum support count n×minsup, the corresponding 1-item set in the traversed file F _qf is removed from the set ST _CAD .

In addition, in order to further improve the digging rate of the present invention, the fine-cutting step of the set ST _CAD also includes a second-stage streamlining step;

The simplified steps of the second stage include:

build the pip array;

Traverse the set ST _CAD reduced by the first-stage reduction step, and select two support counts in the corresponding item sets for each local frequent item set in the set ST _CAD reduced by the first-stage reduction step. The smallest item, the item pairs that form the corresponding itemset, are stored in the PIP array constructed above;

Calculate the support count of each item in the PIP array on the newly added transaction dataset T _Δ ;

Determine the sum of the support count of each item in the PIP array on the newly added transaction data set T _Δ and the support count of the corresponding item set on the newly added transaction data set T _Δ , and the global minimum support count n× The size relationship of minsup, and the local frequent itemsets corresponding to the pairs of items whose sum of the support counts is less than the global minimum support count n×minsup will be deleted from the set ST _CAD .

To sum up, it can be seen that the method for mining frequent itemsets of massive data provided by the present invention further includes a step of fine-cutting the set ST _CAD , so that the set ST _CAD is further reduced before being used for subsequent calculations, thereby reducing the size of the set ST CAD. I/O overhead and computational overhead.

Example 3:

Compared with Embodiment 2, the difference is that the method for mining frequent itemsets of massive data described in Embodiment 3 is updating the newly added transaction data set T _Δ and updating the original transaction data. When the set T _O is the sum of the original original transaction data set T _O and the original new transaction data set T _Δ , and the updated new transaction data set T _Δ is not empty, the step of updating and mining is also included.

Specifically, the steps of updating mining described in this embodiment include:

Update the number n of transactions in the total transaction data set T, which is the sum of the number of transactions in the original original transaction data set T _O and the original new transaction data set T _Δ ;

Obtain all the local frequent item sets corresponding to the original original transaction data set T _O obtained above, and calculate the local frequent itemsets corresponding to the original original transaction data set T _O obtained on the original original transaction data set T _O The support count of , corresponding to the write file F _qf,O ;

Based on the array cnt _Δ corresponding to the original newly added transaction data set T _Δ , add and update the support counts of each local frequent itemset in the file F _qf,O on the original newly added transaction data set T _Δ ; For the local minimum support degree ω, filter the local frequent itemsets in the file F _qf,O , and obtain each local frequent item set whose support degree corresponding to the update total transaction data set T is not less than ω;

Afterwards, each local frequent item set whose support degree is not less than ω and its corresponding support degree in the file F _qf,O is written into a new file F correspondingly. _qf ;

Acquire the set LF _Δ corresponding to the original newly added transaction data set T _Δ , and delete the itemsets in the obtained set LF _Δ that exist in the file F _qf,O , correspondingly to obtain a new set LF _Δ ;

Based on the array cnt _Δ corresponding to the original newly added transaction data set T _Δ , in the new set LF _Δ , correspondingly write the item sets in the new set LF _Δ into the original newly added transaction data set T _Δ support count on ;

Then, according to the local minimum support ω, filter the itemsets in the new set LF _Δ , obtain the itemsets with the filtered support not less than ω, and use the obtained support not less than ω. the item set of ω and the support count written in the new set LF _Δ , corresponding to the new file F _qf written;

Then replace the original file F _qf with the above-mentioned new file F _qf , and replace the original original transaction data set T _O with the sum of the original original transaction data set T _O and the original newly added transaction data set T _Δ , And replace the original new transaction data set T _{Δ with the updated new transaction data set T Δ ,} based on the number n of transactions in the updated total transaction data set T, use the incremental update method to mine the above Frequent itemsets on the updated total transaction dataset T.

It can be seen that the method for mining frequent itemsets of massive data provided by the present invention proposes a specific incremental update strategy, and utilizes existing calculation information, such as array cnt _Δ , set LF _Δ , etc., to speed up update operations, thereby helping to improve massive data Performance and practicality of frequent itemset mining.

It should be noted that the same and similar parts among the various embodiments in this specification may be referred to each other.

Although the present invention has been described in detail in conjunction with the preferred embodiments with reference to the accompanying drawings, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should all fall within the scope of the present invention/any Those skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should all be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (10)

1. A mass data frequent item set mining method is used for mining a frequent item set which meets a global minimum support degree minsup in a total transaction data set T, wherein the global minimum support degree minsup is a preset minimum support degree on the total transaction data set T; it is characterized in that the preparation method is characterized in that,

said total transaction data set T comprising an original transaction data set T_OAnd adding a new transaction data set T_Δ；

The method for mining the mass data frequent item set comprises the following steps:

adopting a frequent item set mining algorithm to carry out on an original transaction data set T_OMining is carried out to obtain an original transaction data set T_OAll corresponding local frequent item sets;

scanning an original transaction data set T_OCorrespondingly calculating each local frequent item set obtained in the above step in the original transaction data set T_OThe above support degree counting is to filter the obtained local frequent item sets according to the local minimum support degree omega, obtain each local frequent item set with the support degree not less than omega, and correspondingly write each obtained local frequent item set with the support degree not less than omega and the corresponding support degree obtained by calculation into the file F_qfPerforming the following steps;

reading newly added transaction data set T_ΔAnd judging the newly added transaction data set T_ΔWhether it is empty:

if yes, the file F is processed according to the number n of the transactions in the total transaction data set T and the global minimum support degree minsup_qfFiltering the local frequent item sets to obtain and output the local frequent item sets with the filtered support counts not less than the global minimum support count n multiplied by min, wherein each output local frequent item set is all frequent item sets meeting the global minimum support count min on the total transaction data set T;

if not, mining a frequent item set on the total transaction data set T by adopting an incremental updating method;

wherein, the local minimum support degree omega is a preset original transaction data set T_OThe local minimum support omega is less than the global minimum support minsup.

2. The mass data frequent itemset mining method according to claim 1, wherein the incremental updating method comprises the following steps:

scanning for newly added transaction data set T_ΔCalculating a newly added transaction data set T_ΔIn the new transaction data set T_ΔCount the support degree of the transaction data set and add a new transaction data set T_ΔEach ofItem set and newly added transaction data set T obtained through calculation_ΔIn the new transaction data set T_ΔCount of support degree of (1), and correspondingly store into array cnt_ΔAnd count the group cnt_ΔThe maximum support count of the medium is mas_Δ；

Scanning a current document F_qfFor each slave current file F_qfRespectively judging whether the local frequent item sets scanned in the total transaction data set T meet the preset global minimum support degree minsup, if so, outputting the currently scanned local frequent item sets, and recording the output local frequent item sets as first frequent item sets; if the judgment result is no, based on the mas_ΔJudging whether the currently scanned local frequent item set is not a frequent item set which meets a preset global minimum support degree minsup on the total transaction data set T: if not, counting the currently scanned local frequent item set and the corresponding support degree, and correspondingly writing into the set ST_CADPerforming the following steps;

based on the above array cnt_ΔCorresponding calculation and update set ST_CADNewly added transaction data set T_ΔThe support degree of each local frequent item set in the total transaction data set T is counted to obtain an updated set ST_CAD；

Traversing updated set ST_CADAnd respectively judging each current traversed updated set ST_CADIf the local frequent item set in the set is the frequent item set meeting the preset global minimum support degree minsup on the total transaction data set T, correspondingly outputting each current traversed updated set ST_CADThe corresponding local frequent itemset in (c);

then judging the expression (n)_O×ω-1)+mas_ΔWhether or not (n × min) holds:

if yes, finishing the mining of the frequent item set;

otherwise, continuing to add the transaction data set T_ΔExcavating a new frequent item set, and outputting the excavated new frequent item set;

wherein,the new frequent item set is in the original transaction data set T_OThe support degree on the system is more than zero, the global minimum support degree minsup is met on the total transaction data set T, and the system is different from all the output frequent item sets;

n_Ofor newly adding transaction data set T_ΔThe number of transactions in (2).

3. The mass data frequent itemset mining method according to claim 2, wherein the new transaction data set T_ΔMining a new frequent item set, and outputting the mined new frequent item set, wherein the method comprises the following steps:

by the formulaCalculating a newly added transaction data set T_ΔMinimum support degree of (Min)_ΔIn the formula n_ΔFor newly adding transaction data set T_ΔThe number of transactions in;

splitting a newly added transaction dataset T_ΔThe transactions in (1) are a plurality of target transaction data sets;

adopting an Eclat algorithm to carry out local frequent item set mining on each target transaction data set to obtain the minimum support degree minsup which corresponds to each target transaction data set and meets the calculation_ΔAll local frequent itemsets of (1);

the set of all local frequent item sets corresponding to the target transaction data sets is recorded as LF_ΔGo through and delete the set LF_ΔHas appeared in the file F_qfTo obtain a candidate set GF_Δ(ii) a And based on the above array cnt_ΔCandidate set GF_ΔAdding local frequent item sets in the newly added transaction data set T_ΔThe corresponding support metric in (b) is stored in the candidate set GF_Δ；

Scanning a current original transaction data set T_OAdding and updating candidate set GF_ΔCounting the support degree of the middle and local frequent item sets to obtain new candidatesSet GF_Δ；

Scanning the new candidate set GF_ΔCorrespondingly judging each currently scanned new candidate set GF_ΔIf the local frequent item set in (1) is the frequent item set meeting the preset global minimum support degree minsup, correspondingly outputting each currently scanned candidate set GF_ΔLocal frequent itemses in (1).

4. The method as claimed in claim 3, wherein the Eclat algorithm is used to perform local frequent itemset mining on each target transaction data set to obtain the minimum support degree minsup corresponding to each target transaction data set and satisfying the above calculation_ΔAll local frequent itemsets of (1), including:

p0, traversing each target transaction data set;

p1, for the currently traversed target transaction data set:

p11, calculating and acquiring a candidate frequent k-item set corresponding to the currently traversed target transaction data set by adopting an Eclat algorithm, and meeting the minimum support degree minsup obtained by the calculation in the generated candidate frequent k-item set_ΔAnd then, the generated candidate frequent k-item set is recorded as a frequent k-item set and stored in a set LF_k，ΔIn the middle, k is more than or equal to 1;

p12 by merging the LF_k，ΔGenerating candidate frequent (k +1) -item sets by the two medium frequent k-item sets, and satisfying the minimum support degree minsup obtained by the calculation in the generated candidate frequent (k +1) -item sets_ΔThen, the generated candidate frequent (k +1) -item set is recorded as a frequent (k +1) -item set and stored in a set LF_k+1，ΔWherein the first k-1 terms of the two sets of frequent k-terms are the same and the last term is different;

p13, repeating the above steps P11-P12, increasing k by 1 each time until a new candidate frequent item set corresponding to the currently traversed target transaction data set can not be generated; then step P14 is executed;

p14, continue traversing the next target transaction data set, and repeatedly executing the above stepsP11-P13 until all target transaction data sets are traversed, so that the minimum support degree minsup which is obtained by correspondingly meeting the calculation and corresponds to each target transaction data set is obtained_ΔAll of the local frequent itemsets of (c).

5. The mass data frequent item set mining method according to claim 4, further comprising, between step P11 and step P12, step S: for the set LF_k，ΔFine shearing;

wherein, for the set LF_k，ΔThe fine shearing step comprises the following steps:

obtaining and dividing a set LF according to whether the first (k-1) items of the item set are the same or not_k，ΔGrouping medium-frequent k-item sets to obtain a corresponding number of item set groups, wherein the first (k-1) items of the frequent k-item sets in the same item set group are the same;

respectively counting the number of the frequent k-item sets in each item set group, and correspondingly judging whether the counted number is equal to 1: if yes, deleting the corresponding item set group and deleting the set LF_k，ΔThe same set of items as the frequent set of k-items in the corresponding set of items grouping; the corresponding item set group, wherein the number of the frequent k-item sets is equal to 1;

correspondingly judging whether the union of any two frequent k-item sets in the item set group is contained in the file F in the currently existing item set group_qfThe method comprises the following steps: if yes, deleting the corresponding item set group and deleting the set LF_k，ΔThe same set of items as the frequent set of k-items in the corresponding group of sets of items.

6. The mass data frequent itemset mining method according to claim 2, wherein the array cnt is based on_ΔCorresponding calculation and update set ST_CADNewly added transaction data set T_ΔThe support degree of each local frequent item set in the total transaction data set T is counted to obtain an updated set ST_CADBefore, also include to the said set ST_CADFine shearing;

for the set ST_CADThe fine shearing step comprises a first stage of simplification step;

the first stage of the reduction step comprises:

traverse said file F_qfAnd array cnt_ΔAnd respectively calculate each traversed file F_qfSupport count and array cnt for 1-item set in (1)_ΔThe same 1-item set in the array cnt_ΔIf the calculated sum of the support degrees is smaller than the global minimum support degree count n multiplied by min, the traversed file F_qfFrom said set ST_CADIs removed.

7. The mass data frequent itemset mining method of claim 6, wherein the set ST is_CADThe fine shearing step of (2) further comprises a second stage of simplification step;

the second stage of the reduction step comprises:

building a PIP array;

traversing the set ST reduced by the first stage of reduction_CADAnd is a set ST reduced by the first stage of reduction_CADSelecting two items with the minimum support counts in the corresponding item sets respectively for each local frequent item set to form an item pair of the corresponding item sets, and storing the item pair in the constructed PIP array;

calculating the new transaction data set T of each item pair in the PIP array_ΔA support count of (a);

determining that each item pair in the PIP array is in the newly added transaction data set T_ΔCount of support degree and corresponding item set in newly added transaction data set T_ΔThe sum of the support counts and the global minimum support count n × min, and a local frequent item set self-set ST corresponding to each item pair for which it is determined that the sum of the support counts is smaller than the global minimum support count n × min_CADIs deleted.

8. The mass data frequent item set mining method according to claim 1, 2, 3, 4, 5, 6 or 7, wherein said newly added transaction data set T is updated_ΔAnd updating said original transaction data set T_OFor the original transaction data set T_OAnd the original newly added transaction data set T_ΔSum and updated newly added transaction data set T_ΔWhen the mass data is not empty, the method for mining the frequent item set of the mass data further comprises the step of updating and mining;

the step of updating the mining comprises the following steps:

updating the number n of the transactions in the total transaction data set T to the original transaction data set T_OAnd the original newly added transaction data set T_ΔThe sum of the number of transactions of (c);

obtaining the original transaction data set T obtained above_OCorresponding all local frequent item sets, and calculating the acquired original transaction data set T_OThe corresponding local frequent item sets are in the original transaction data set T_OCount of degree of support above, corresponding to write file F_qf，OPerforming the following steps;

newly-added transaction data set T based on original_ΔCorresponding array cnt_ΔAdding and updating file F_qf，OThe local frequent item sets are in the original newly added transaction data set T_ΔA support count of (a); then, according to the local minimum support degree omega, the file F is processed_qf，OFiltering the local frequent item sets in the total transaction data set to obtain each local frequent item set with the support degree not less than omega corresponding to the updated total transaction data set T;

then, each local frequent item set with the support degree not less than omega corresponding to the obtained updated total transaction data set T and the file F thereof_qf，ORespectively corresponding to the support degree counts, correspondingly writing a new file F_qf；

Obtaining the original newly added transaction data set T_ΔCorresponding sets LF_ΔAnd delete the acquired set LF_ΔIs present in the file F_qf，OThe item set in (1) is corresponded to obtain a new set LF_Δ；

Newly-added transaction data set T based on original_ΔCorresponding array cnt_ΔIn the new set LF_ΔIn correspondence with writing the new set LF_ΔAll items in the original newly added transaction data set T_ΔA support count of (a);

then, according to the local minimum support degree omega, the new set LF is processed_ΔFiltering the item sets in the collection, acquiring the filtered item sets with the support degree not less than omega, and acquiring the acquired item sets with the support degree not less than omega and the new set LF_ΔThe support degree count of the write-in is written into the new file F correspondingly_qf；

Thereafter using the new file F_qfReplacing original file F_qfUsing original transaction data set T_OAnd the original newly added transaction data set T_ΔThe sum replaces the original transaction data set T_OAnd using the updated new transaction data set T_ΔReplace the original newly added transaction data set T_ΔAnd mining a frequent item set on the updated total transaction data set T by adopting the incremental updating method based on the number n of the transactions in the updated total transaction data set T.

9. The mass data frequent item set mining method according to claim 1, 2, 3, 4, 5, 6 or 7, characterized in that said file F is mined according to the number n of transactions in the total transaction data set T and said global minimum support degree min_qfFiltering the local frequent item set to obtain a local frequent item set of which the filtered support count is not less than the global minimum support count nxmin, which specifically comprises the following steps:

sequential scanning document F_qf；

Respectively judging the scanned documents F_qfWhether the support count of the local frequent item set in (1) is greater than or equal to a global minimum support count n × minisup:

if yes, the scanned file F_qfThe local frequent item set in (1) is the local frequent item set of which the filtered support count is not less than the global minimum support count n multiplied by min.

10. The mass data frequent item set mining method according to claim 2, 3, 4, 5, 6 or 7, characterized in that said scanning of the current file F_qfFor each slave current file F_qfRespectively judging whether the local frequent item sets scanned in the total transaction data set T meet the preset global minimum support degree minsup, if so, outputting the currently scanned local frequent item sets, and recording the output local frequent item sets as first frequent item sets; if the judgment result is no, based on the mas_ΔJudging whether the currently scanned local frequent item set is not a frequent item set which meets a preset global minimum support degree minsup on the total transaction data set T: if not, counting the currently scanned local frequent item set and the corresponding support degree, and correspondingly writing into the set ST_CADThe method specifically comprises the following steps:

sequentially scanning said document F_qf；

For each scanned document F_qfThe scanned file F is respectively judged according to the local frequent item set_qfWhether the support count of the local frequent item set in (1) is greater than or equal to a global minimum support count n × minisup:

if so, outputting the scanned local frequent item set, wherein the output local frequent item set is a frequent item set meeting the preset global minimum support minsup on the total transaction data set T;

if not, judging the scanned file F_qfThe support count of the local frequent item set in (1) and the mas_ΔIs less than the global minimum support degree minsup, and if the result of the determination is yes, the currently scanned local frequent item set must not be the frequent item set satisfying the preset global minimum support degree minsup on the total transaction data set T, and if not, the local frequent item set satisfies the global minimum support degree minsupCorrespondingly writing the currently scanned local frequent item set and the correspondingly scanned support degree count into the set ST_CAD。