CN110471960B - High-utility item set mining method containing negative utility - Google Patents

Abstract

The invention discloses a high-utility item set mining method containing negative utility, which is characterized by comprising the following steps: 1) data mining operational parametersNumber initialization; 2) scanning the database D; 3) covering the linked list; 4) establishing an EUCS structure and a PNU-List linked List; 5) calling a search program; 6) calculating P_x.putils、P_xNulls and P_xRputils; 7) determining a delivery branch item set pruning strategy containing negative utility; 8) recursively calling a search program; 9) and outputting the efficient item set. The method considers the condition that the unit utility of the items in the database is positive and negative, introduces the chain table coverage theory, compresses when establishing the chain table for the first time, constructs the utility chain table early filtering strategy and the transfer utility branch pruning strategy containing the negative utility, can reduce the construction of the low utility chain table, can reduce the search space, can reduce the high utility item set mining operation time and the memory consumption, and can improve the efficiency of the high utility item set mining method.

Description

High-utility item set mining method containing negative utility

Technical Field

The invention relates to a high-utility data mining technology, in particular to a high-utility item set mining method containing negative utility.

Background

It is common in the retail industry to provide decision-making data to decision-makers based on how many times goods are purchased. In real life, the cost, selling price, profit, weight and risk of each commodity are different, so that the real situation cannot be reflected only by considering the number of times that the commodity is purchased in the transaction database. The High-Utility item set Mining (HUIM) algorithm not only considers the times of occurrence of commodities in transactions, but also considers the unit profit (Utility) of the commodities, the High-Utility item set Mining algorithm aims at finding items and item sets which bring considerable profit for retailers in a transaction database, and the mined item sets are called High-efficiency item sets, namely HUIs. However, most efficient use item set algorithms only consider the utility of the item set as positive, and very few algorithms consider the case where the utility of the item set is negative. In commercial competition, many companies adopt marketing schemes, that is, a certain product is sold to consumers in a zero-profit or negative-profit mode, and then the consumers are guided to buy other high-profit commodities under certain conditions, so as to increase the operating income. If the efficient item set algorithm without considering the negative item set is adopted, the efficient item set is estimated to be the low-efficiency item set when the high-efficiency item set is mined, so that a large number of efficient item sets are not mined, and the decision of a company decision maker is influenced. The FHN (fast High-Utility parameter sets with Negative unit properties, abbreviated as FHN) algorithm is an efficient item set algorithm with Negative unit profit proposed by Lin et al, and adopts a linked List structure of PNU-List (Positive and Negative Utility-List, abbreviated as PNU-List) to store key information in a database, and the FHN algorithm greatly improves the problems that the original two-stage algorithm needs to scan the database for many times and generates a large number of candidate sets through the structure of a vertical linked List and a corresponding pruning strategy, thereby improving the efficiency of the algorithm. However, in the method, (1) the vertical linked list is not established for data compression, which wastes a large amount of space; (2) the low-utility linked list is difficult to filter, and the linked list can be calculated to be the low-efficiency linked list only after the establishment of the utility linked list is completed; (3) the effectiveness of the over-estimation (Overestimation) mode metric term set is too loose, many low-efficiency term sets are overestimated into high-efficiency term sets, the search space is too large, and the running time of the mining process is long.

Disclosure of Invention

The invention aims to provide a high-utility item set mining method with negative utility aiming at the defects of the prior art. The method considers the condition that the unit utility of the items in the database is positive and negative, introduces the chain table coverage theory, compresses when establishing the chain table for the first time, constructs the utility chain table early filtering strategy and the transfer utility branch pruning strategy containing the negative utility, can reduce the construction of the low utility chain table, can reduce the search space, can reduce the high utility item set mining operation time and the memory consumption, and can improve the efficiency of the high utility item set mining method.

The technical scheme for realizing the purpose of the invention is as follows:

compared with the prior art, the high-utility item set mining method with negative utility comprises the following steps:

1) initializing data mining operation parameters: setting a database D to be mined, and specifying a minimum utility threshold min _ util and a profit table ptable;

2) scanning the database D: scanning the entries in database D, calculating transaction weighted utility TWU and redefined transaction utility RTU for the entries, placing linked list I in a specified order for each entry I having a value of TWU no less than min _ util^*Performing the following steps;

3) covering a linked list: if TWU values of the item set { I } and the item set { ij } are equal, namely TWU (I) ═ TWU (ij), then the item j is called to be contained in the item I, when the database D is scanned for the first time and the utility linked list is constructed, whether the same elements in the linked list have the same value TWU is judged through retrieval, if not, the elements are added, if yes, the elements are abandoned, and the original linked list I is compressed in this way^*Generating a new chain table CovL;

4) establishing an EUCS structure and a PNU-List linked List: scanning a database D and a linked List CovL, and establishing an evaluation utility co-occurrence structure EUCS and a positive and negative utility linked List PNU-List;

5) calling a search program: taking the empty set, the CovL linked list, the minimum utility threshold value min _ util and the EUCS structure as parameters to call a search program;

6) calculating P_x.putils、P_xNulls and P_xRputils: first traverse the branch item set P of the item set P_xIf P is_xPositive effect sum of (i.e. P)_xAdds the sum of negative effects, i.e. P_xThe sum of nulls is not less than the minimum utility threshold min _ util, then P_xThe items are efficiently used and output; if P is_xPulils plus the remaining positive effect sum, i.e. P_xThe sum of rputils is not less than the minimum utility threshold min _ util, then P_xThe branch of (A) is a high-efficiency item set, and P needs to be continuously mined_xIs circularly traversed through each branch entry set P of the entry set P_ySo that y > x and TWU ({ x, y }) > min _ util, with P_xForm P_xyThe branch continues to perform mining and connects P_xAnd P_yAnd substituting a transfer branch utility calculation formula tenu (X, y) containing negative utility to calculate a transfer branch utility upper limit:

tenu(X，y)≥SUM(X_y.putil)+SUM(X_y.nutil)+SUM(X_y.rputil)，

if tenu (P)_x，P_y) Less than min _ util, then P_xyThe single item branch and the transmission branch are low-efficiency item sets, and the mining P is directly abandoned_xyDirectly jumping out of the loop by adopting a delivery branch pruning strategy containing negative effect, and searching the next P meeting the conditions_y；

7) Determining a delivery branch item set pruning strategy containing negative utility: set of items P, P_x，P_yAs a parameter call utility linked list construction process, P is firstly selected_xThe sum of the positive utility of (1) and the remaining positive utility rputil is set to total, for each at P_xTuple ex in the linked list is searched in P by adopting a binary search mode_ySearching a tuple ey which is the same as the transaction unique identifier tid in the tuple in the linked list, if not, reducing the value of total through rputil and putil, checking whether the value of total is less than min _ util, if so, stopping constructing the P_xyAnd does not mine P_xyThe branch of (1);

8) recursively invoking a search program: if P is_xyIf the list is not empty, P_xyAnd the branch will enter the searching program again, and the searching process is called recursively until the branch can not be generated;

9) and outputting the efficient item set.

The technical scheme has the following advantages:

(1) covering the same items when the linked list is constructed through a covering theory so as to compress the memory space occupied by the linked list;

(2) the pruning strategy of early filtering is provided, the utility value of the utility linked list is calculated before the utility linked list is constructed, the construction of the low utility linked list is filtered, and the space and the time required by the algorithm are saved;

(3) a new calculation formula of the upper limit of the utility of the transmission branch item set containing the negative utility is provided, the utility of the item set is more accurately estimated, the calculation formula is integrated in a pruning strategy of the transmission branch containing the negative utility, and the low-efficiency item set is estimated and pruned, so that the search space is reduced, and the algorithm efficiency is improved.

The technical scheme is mainly applied to data mining in transaction system background transaction databases in retail industry, electronic commerce and the like.

The method considers the condition that the unit utility of the items in the database is positive and negative, introduces the chain table coverage theory, constructs the utility chain table early filtering strategy and the transfer utility branch pruning strategy containing the negative utility, can reduce the construction of the low utility chain table, can reduce the search space, can reduce the mining operation time and the memory consumption of the high utility item set, and can improve the mining efficiency of the high utility item set.

Drawings

FIG. 1 is a diagram illustrating the comparison of the run time effect of the method and FHN method in the embodiment;

FIG. 2 is a diagram illustrating the comparison of memory consumption effects between the embodiments of the method and the FHN method;

FIG. 3 is a diagram illustrating the comparison between the method and the FHN method for generating the number of candidate sets in the embodiment;

FIG. 4 is a schematic flow chart of an example method;

FIG. 5 is a flowchart illustrating a linked list overlay policy in a method according to an embodiment;

FIG. 6 is a flow diagram of a search routine in an embodiment method;

FIG. 7 is a flow diagram illustrating a pre-filtering strategy in an exemplary method.

Detailed Description

The invention will be further elucidated with reference to the drawings and examples, without however being limited thereto.

Example (b):

referring to fig. 4, a method for mining a high utility item set containing negative utilities includes the following steps:

1) initializing data mining operation parameters: setting a database D to be mined, and specifying a minimum utility threshold min _ util and a profit table ptable;

2) scanning the database D: scanning the entries in database D, calculating transaction weighted utility TWU and redefined transaction utility RTU for the entries, placing linked list I in a specified order for each entry I having a value of TWU no less than min _ util^*Performing the following steps;

3) covering a linked list: if the set of items { i } and the set of itemsTWU values of { ij } are equal, namely TWU (I) ═ TWU (ij), then item j is called to be contained in item I, when the database D is scanned for the first time and the utility linked list is constructed, whether the same elements in the linked list have the same TWU values is judged through searching, if not, the elements are added, if so, the elements are abandoned, and the original linked list I is compressed in this way^*Generating a new chain table CovL, as shown in FIG. 5;

4) establishing an EUCS structure and a PNU-List linked List: scanning a database D and a linked List CovL, and establishing an evaluation utility co-occurrence structure EUCS and a positive and negative utility linked List PNU-List;

5) calling a search program: calling a search program by taking the empty set, the CovL linked list, the minimum utility threshold min _ util and the EUCS structure as parameters, as shown in FIG. 6;

6) calculating P_x.putils、P_xNulls and P_xRputils: first traverse the branch item set P of the item set P_xIf P is_xPositive effect sum of (i.e. P)_xAdds the sum of negative effects, i.e. P_xThe sum of nulls is not less than the minimum utility threshold min _ util, then P_xThe items are efficiently used and output; if P is_xPulils plus the remaining positive effect sum, i.e. P_xThe sum of rputils is not less than the minimum utility threshold min _ util, then P_xThe branch of (A) is a high-efficiency item set, and P needs to be continuously mined_xIs circularly traversed through each branch entry set P of the entry set P_ySo that y > x and TWU ({ x, y }) > min _ util, with P_xForm P_xyThe branch continues to perform mining and connects P_xAnd P_yAnd substituting a transfer branch utility calculation formula tenu (X, y) containing negative utility to calculate a transfer branch utility upper limit:

tenu(X，y)≥SUM(X_y.putil)+SUM(X_y.nutil)+SUM(X_y.rputil)，

if tenu (P)_x，P_y) Less than min _ util, then P_xyThe single item branch and the transmission branch are low-efficiency item sets, and the mining P is directly abandoned_xyDirectly jumping out of the loop by adopting a delivery branch pruning strategy containing negative effect, and searching the next P meeting the conditions_y；

7) Determination of negative effectsDelivery branch item set pruning strategy of (1): set of items P, P_x，P_yAs a parameter call utility linked list construction process, P is firstly selected_xThe sum of the positive utility of (1) and the remaining positive utility rputil is set to total, for each at P_xTuple ex in the linked list is searched in P by adopting a binary search mode_ySearching a tuple ey which is the same as the transaction unique identifier tid in the tuple in the linked list, if not, reducing the value of total through rputil and putil, checking whether the value of total is less than min _ util, if so, stopping constructing the P_xyAnd does not mine P_xyThe branch of (1);

8) recursively invoking a search program: if P is_xyIf the list is not empty, P_xyAnd the branch will enter the searching program again, and the searching process is called recursively until the branch can not be generated;

9) and outputting the efficient item set.

The data Mining method EHUIN (Efficient High-Utility item Mining with reactive Utility, EHUIN for short) can be used for more efficiently Mining data of transaction system data warehouses in retail industry, e-commerce and the like.

The method has the following advantages:

(1) the item set mining operation time is short: as can be seen from fig. 1, the method of this embodiment is more efficient than the FHN algorithm, almost all the data sets are run with time consumption lower than that of the FHN algorithm, the data set pumsb, when min _ util is 5000k and 5100k, the run time of the EHUIN algorithm is 381.6s and 316.7s, respectively, and the FHN algorithm is very inefficient with time consumption of 783.8s and 580.2s, respectively, and in addition, compared with the FHN algorithm, the run time of the method of this embodiment is much less sensitive to the change of min _ util threshold than that of the FHN algorithm, when the min _ util threshold is lower, the run time of the FHN algorithm increases sharply, and the method of this embodiment is more gentle, as can be seen from fig. 1, the threshold of the data set chess increases sharply from 110k to 135k, the run time increases from 6.9s to 109.4s, and the method of this embodiment increases from 2.7 s to 6.8 s, and is very smooth;

(2) the memory space required by the excavation process is small: in terms of memory usage, the method of the present embodiment uses less memory on all data sets than the FHN algorithm, and it can be seen in fig. 2 that the method of the present embodiment performs well on dense data sets, not only at run time, the number of pruned candidate sets, but also significantly reduces the memory space, for example, on data set chess, when min _ util is 120k, 125k, and 130k, respectively, the memory space of FHN algorithm is as high as 673.7MB, 673.9MB, and 671.7MB, while the memory space of the method of the present embodiment is 322.3MB, 269.1MB, and 271.4MB, respectively, and the memory space of the method of the present embodiment is about half of that of FHN algorithm;

(3) the candidate item set is few: as can be seen from FIG. 3, the method of the present embodiment generates fewer candidate sets on all data sets than the FHN algorithm, and particularly in dense data sets, for example, when the data set chess has a min _ util threshold of 110k, the candidate set generated by the FHN algorithm is 711923, while the method of the present embodiment is only 50857, and when the min _ util threshold is 1k, the method of the present embodiment excavates only 179011 efficient term sets, and the efficient term set excavated by the FHN algorithm is 239165.

Claims (1)

1. A high-utility item set mining method containing negative utility is characterized by comprising the following steps:

1) initializing data mining operation parameters: setting a database D to be mined, and specifying a minimum utility threshold min _ util and a profit table ptable;

2) scanning the database D: scanning the entries in database D, calculating transaction weighted utility TWU and redefined transaction utility RTU for the entries, placing linked list I in a specified order for each entry I having a value of TWU no less than min _ util^*Performing the following steps;

3) covering a linked list: if TWU values of the item set { I } and the item set { ij } are equal, namely TWU (I) ═ TWU (ij), then the item j is called to be contained in the item I, when the database D is scanned for the first time and the utility linked list is constructed, whether the same elements in the linked list have the same value TWU is judged through retrieval, if not, the elements are added, if yes, the elements are abandoned, and the original linked list I is compressed in this way^*Generating a new chain table CovL;

4) establishing an EUCS structure and a PNU-List linked List: scanning a database D and a linked List CovL, and establishing an evaluation utility co-occurrence structure EUCS and a positive and negative utility linked List PNU-List;

5) calling a search program: taking the empty set, the CovL linked list, the minimum utility threshold value min _ util and the EUCS structure as parameters to call a search program;

6) calculating P_x.putils、P_xNulls and P_xRputils: first traverse the branch item set P of the item set P_xIf P is_xPositive effect sum of (i.e. P)_xAdds the sum of negative effects, i.e. P_xThe sum of nulls is not less than the minimum utility threshold min _ util, then P_xThe items are efficiently used and output; if P is_xPulils plus the remaining positive effect sum, i.e. P_xThe sum of rputils is not less than the minimum utility threshold min _ util, then P_xThe branch of (A) is a high-efficiency item set, and P needs to be continuously mined_xIs circularly traversed through each branch entry set P of the entry set P_ySo that

And TWU ({ x, y }) is equal to or more than min _ util, and P_xForm P_xyThe branch continues to perform mining and connects P_xAnd P_yAnd substituting a transfer branch utility calculation formula tenu (X, y) containing negative utility to calculate a transfer branch utility upper limit:

tenu(X，y)≥SUM(X_y.putil)+SUM(X_y.nutil)+SUM(X_y.rputil)，

if tenu (P)_x，P_y) Less than min _ util, then P_xyThe single item branch and the transmission branch are low-efficiency item sets, and the mining P is directly abandoned_xyDirectly jumping out of the loop by adopting a delivery branch pruning strategy containing negative effect, and searching the next P meeting the conditions_y；

7) Determining a delivery branch item set pruning strategy containing negative utility: set of items P, P_x，P_yAs a parameter call utility linked list construction process, P is firstly selected_xThe sum of the positive utility of (1) and the remaining positive utility rputil is set to total, for each at P_xTuple ex in the linked list is searched in P by adopting a binary search mode_yLooking up the tuple ey which is the same as the transaction unique identifier tid in the tuple in the linked list, if notIf yes, reducing the value of total through rputil and putil, checking whether the value of total is less than min _ util, if yes, stopping constructing P_xyAnd does not mine P_xyThe branch of (1);

8) recursively invoking a search program: if P is_xyIf the list is not empty, P_xyAnd the branch will enter the searching program again, and the searching process is called recursively until the branch can not be generated;

9) and outputting the efficient item set.

Images