WO2017143908A1 - 关联分析方法和装置 - Google Patents

关联分析方法和装置 Download PDF

Info

Publication number
WO2017143908A1
WO2017143908A1 PCT/CN2017/073135 CN2017073135W WO2017143908A1 WO 2017143908 A1 WO2017143908 A1 WO 2017143908A1 CN 2017073135 W CN2017073135 W CN 2017073135W WO 2017143908 A1 WO2017143908 A1 WO 2017143908A1
Authority
WO
WIPO (PCT)
Prior art keywords
item set
database
association analysis
projection
frequent
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/073135
Other languages
English (en)
French (fr)
Inventor
代斌
杨旭
姜晓燕
蔡宁
王少萌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2017143908A1 publication Critical patent/WO2017143908A1/zh
Priority to US16/107,927 priority Critical patent/US10956395B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/15—Correlation function computation including computation of convolution operations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
    • G06F2216/03—Data mining

Definitions

  • the present invention relates to computer technology, and in particular, to an association analysis method and apparatus.
  • Association analysis is an analysis technique in data mining. It is mainly used to discover the correlation between projects based on a large amount of data.
  • a typical application example of association analysis is shopping basket analysis: correlation analysis based on shopping data, and the correlation between customers entering different products in the shopping basket. Furthermore, the correlation obtained by this correlation analysis reflects the customer's buying habits, and understanding these purchasing habits can help the retailer to develop marketing strategies.
  • the sequential pattern association analysis is different from the correlation analysis in other modes.
  • the sequence pattern also considers the time when the project occurs, so that there is a certain order between the projects.
  • the objects to be analyzed are often very large-scale databases. Therefore, the amount of computation is large and it is not suitable for single-machine data processing.
  • the data in the database is usually simply fragmented, and each piece of data is separately analyzed in each node to obtain a candidate frequent item set and its support degree, and then merged to obtain each candidate frequent item set.
  • the global support degree is obtained, and then the global frequent item set is obtained after pruning according to the preset screening condition.
  • the data amount of the candidate frequent itemsets is an exponential multiple of the data amount of the fragment data, and therefore, the candidate frequent itemsets generated in each node are generated.
  • the summation and its support are summarized to perform the merge operation, the amount of data transferred is too large, resulting in low execution efficiency.
  • the present invention provides an association analysis method and apparatus for solving the technical problem of low execution efficiency due to excessive data transmission when summarizing data in parallel correlation analysis in the prior art.
  • an association analysis method including:
  • the original database is divided into projection databases, and the projection databases do not contribute to the support of frequent itemsets;
  • Each node performs a correlation analysis of the sequence patterns on the corresponding projection database to obtain a local frequent item set. And corresponding support;
  • the local frequent itemsets obtained by the correlation analysis of each node and the corresponding support degrees are summarized to obtain a global frequent item set and corresponding support degrees.
  • an association analysis apparatus including:
  • a dividing module configured to divide the original database into each projection database, where the projection databases do not contribute to the support of frequent itemsets;
  • An analysis module configured to perform a correlation analysis of the sequence patterns on the projection database by each node, to obtain a local frequent item set and a corresponding support degree;
  • the summary module is configured to summarize the local frequent itemsets obtained by correlation analysis of each node and the corresponding support degrees, and obtain a global frequent item set and corresponding support degree.
  • the association analysis method and apparatus divide the original database into projection databases that do not contribute support degrees to each other, and each node performs correlation analysis on the sequence schema of the corresponding projection database, and then The local frequent itemsets obtained by the nodes and the corresponding support levels are summarized. Since the established projection databases do not contribute to the support of frequent itemsets, different nodes can perform association mining including pruning steps for different projection databases, and each node obtains a small amount of data. The local frequent item set avoids the situation in the prior art that a local candidate frequent item set with a large amount of data obtained by each node without the pruning step needs to be transmitted, thereby saving transmission overhead and improving efficiency.
  • FIG. 1 is a schematic flowchart of a correlation analysis method according to Embodiment 1 of the present invention.
  • FIG. 2 is a schematic flowchart of a correlation analysis method according to Embodiment 2 of the present invention.
  • FIG. 3 is a schematic diagram of an execution process of an association analysis method
  • FIG. 4 is a schematic structural diagram of an association analysis apparatus according to an embodiment of the present invention.
  • FIG. 5 is a schematic structural diagram of another correlation analysis apparatus provided in Embodiment 3.
  • FIG. 6 is a schematic structural diagram of an association analysis apparatus provided in Embodiment 4.
  • each transaction is usually recorded, and the transaction includes each item, and each item includes at least one element.
  • One element is used to indicate an operation object
  • the item is used to indicate each operation object operated by the same user at the same time
  • an operation object operated by a user at different times can be identified by a transaction. Since the association analysis in the sequence mode needs to consider the sequence, each item included in each transaction has a certain order. This sequence of items can be called a sequence. Therefore, it can be said that the transaction is Expressed by a sequence.
  • a transaction in the database is the sequence abc, abc, ac, d, cf, which can be used to indicate that a user bought the goods a, b, and c on the first day, and bought the goods a, b, and c the next day.
  • the total items bought for each day are called items, and each item is called an element.
  • the inventor analyzes the association analysis method in the prior art, and finds that in the prior art, when the candidate frequent itemsets generated by each node and their support are summarized to perform the merge operation, the amount of data transmitted is too large, mainly due to The step of performing pruning locally is not performed by each node.
  • the inventors conducted further analysis.
  • the original data is simply fragmented, and each fragment data has the possibility of contributing support for a certain frequent item level, so that the clipping cannot be performed locally.
  • the step of branching therefore, the original data needs to be divided into databases that do not have support between each other to be pruned locally, thereby avoiding transmission of frequent itemsets and their support.
  • the inventors have proposed the association analysis method provided by the present invention.
  • Sequence pattern association analysis refers to: given a database, where the database includes individual transactions, each transaction is represented by a sequence, and each sequence consists of items with a certain order of each other. Sequence pattern mining is to find all sub-sequences that satisfy the frequency of occurrence in the database not lower than the support threshold based on a support threshold. Find these sub-sequences as frequent itemsets and find out from the database. The operation of these frequent itemsets is sequential pattern association analysis.
  • An item set is a collection of items that are ordered by the database for items that appear in the same transaction.
  • the length of the item set refers to the length of the sequence used to represent the item set, which is numerically equal to the number of items included in the item set.
  • Supportability refers to the frequency at which an item set appears in the database. If a transaction contains a set of items, it is recorded as one time, so that the support is equal to the number of transactions in the database that contain a certain set.
  • a frequent item set is a set of items in the database that are not less than the support threshold.
  • the projection database refers to a database obtained by performing a projection operation on the original database.
  • the projection database of ⁇ is a suffix of all sequences prefixed by ⁇ with respect to ⁇ in S.
  • FIG. 1 is a schematic flowchart of a method for association analysis according to Embodiment 1 of the present invention. As shown in FIG. 1, the method includes:
  • Step 101 Divide the original database into each projection database.
  • each projection database does not contribute to the support of frequent itemsets.
  • the support degree refers to the number of transactions that contain an episode in the database.
  • Each projection database does not contribute to the support of frequent itemsets, that is, the frequent itemsets obtained by sequential pattern mining based on a projection database do not appear in another projection database, so that another projection database does not increase frequently.
  • the initial item set is a set of items obtained by performing association analysis on the original database, used to construct a projection database, and any two initial item sets in at least two initial item sets There is no mutual inclusion relationship between them.
  • the at least two sets of the foregoing may be obtained by using an association analysis algorithm for the original database, and the two sets may be the same length or different lengths, and may be a frequent item set obtained by performing the pruning step. It is a candidate frequent item set obtained by the pruning step, which is not limited in this embodiment.
  • each initial item set is used as a prefix to establish a projection database of each prefix.
  • the initial item set is used as a prefix
  • the suffix corresponding to the prefix is queried in each transaction of the original database, and then the suffix of each transaction is summarized to form a projection database of the prefix.
  • the suffix is a sequence after the location where the prefix first appears in each transaction, and if the last element of the prefix is the same as the timing of the first element of the found sequence, then "_" and the searched The sequence to which it is obtained is used as a suffix. Otherwise, the sequence found is directly suffixed.
  • the projection databases established according to this do not contribute to the support of frequent itemsets, and different nodes may separately perform pruning steps on different projection databases.
  • the local frequent itemsets obtained by each node are small in data volume, and avoid the need to transmit the local candidate frequent itemsets with large data amount obtained by each node without the pruning step in the prior art. , which saves transmission overhead and improves efficiency.
  • inclusion relationship defined here is the inclusion relationship of the prefix, that is, whether the initial item set is a prefix of another initial item set, and if so, there is an inclusion relationship, for example:
  • ⁇ a,b> is not a prefix of ⁇ a,c,b>, so neither ⁇ a,b> nor ⁇ a,c,b> Existence relationship
  • ⁇ a> is a prefix of ⁇ a, b>, so ⁇ a, b> and ⁇ a> belong to the inclusion relationship.
  • the K_ frequent itemsets of the original database may be calculated, that is, the frequent itemsets of the K items in the item set, where K is a positive integer, and 1 ⁇ K ⁇ N, N is in the original database. The number of elements included, with the K_ frequent itemset as the initial item set.
  • the K_ frequent item set of the original database is calculated, the K_ frequent item set is filtered according to the preset support degree threshold, and the K_ frequent with the support degree greater than the support threshold is retained.
  • the item set uses the filtered K_ frequent item set as the initial item set.
  • Step 102 Perform correlation analysis of sequence patterns on the corresponding projection databases by each node to obtain a local frequent item set and corresponding support degree.
  • a node is allocated for each projection database, for example, a node whose load capacity matches the data amount of the projection database can be allocated to each projection database.
  • each node uses the preset association analysis algorithm to perform the association analysis of the sequential patterns in parallel, such as the Generalized Sequential Pattern (GSP) algorithm, specifically, the steps of each node performing scanning, merging, and pruning on the projection database. , get frequent itemsets and corresponding support.
  • GSP Generalized Sequential Pattern
  • Each node can only obtain a part of the frequent itemsets in each frequent item set of the original database.
  • the frequent itemsets obtained by each node are called local frequent itemsets, which will be original. All frequent itemsets of a database are called global frequent itemsets.
  • the node mentioned here is running on a single machine, and the single machine may be a physical machine or a virtual machine. This embodiment does not limit this.
  • Step 103 Summarize the local frequent item set obtained by each node association analysis and the corresponding support degree, and obtain a global frequent item set and corresponding support degree.
  • the local frequent item set obtained by each node association analysis and the corresponding support degree are summarized into one file, so that the global frequent item set and the corresponding support degree are recorded in the file without performing merge and pruning. step.
  • each node by dividing the original database into projection databases that do not contribute support to each other, each node performs a correlation analysis of the sequence patterns on the corresponding projection databases, and then the local frequent itemsets obtained for each node. And the corresponding support level is summarized. Since the established projection databases do not contribute to the support of frequent itemsets, different nodes can perform association mining including pruning steps for different projection databases, and each node obtains a small amount of data.
  • the local frequent item set avoids the situation in the prior art that a local candidate frequent item set with a large amount of data obtained by each node without the pruning step needs to be transmitted, thereby saving transmission overhead and improving efficiency.
  • FIG. 2 is a schematic flowchart of a method for association analysis according to Embodiment 2 of the present invention.
  • the association analysis method provided in this embodiment may be executed by software, and the software may be run in an Open Data Processing Service (ODPS).
  • ODPS Open Data Processing Service
  • the method provided in this embodiment can be executed by multiple nodes under the framework of MapReduce software, and includes a master node and multiple slaves under the MapReduce software framework.
  • the master node can assign a map job to each slave node (used to map a set of key-value pairs into a new set of key-value pairs) or a reduction job (to ensure that each of the mapped key-value pairs is guaranteed
  • a node that shares the same key group, the slave node assigned to the Map job can also be called a Map node, and the slave node assigned to the Reduce job can also be called a Reduce node.
  • Each node can be run on a physical machine or a virtual machine, which is not limited in this embodiment. As shown in Figure 2, the method includes:
  • Step 201 The master node schedules the slave node to perform data preprocessing on the slave node.
  • the master node takes the data preprocessing as a MapReduce task, and schedules the slave node to perform the task, thereby performing data serialization operations on the original data.
  • the different operation objects that each user performs the operation described in the original data are identified by a numerical or alphabetic sequence according to the mapping table, and the operation objects that perform operations for the same user are sorted according to the operation time.
  • Step 202 The slave node calculates a K_ frequent item set based on the original database D under the master node scheduling, and each of the The K_ frequent itemsets are assigned to a Map node for processing.
  • N is the number of elements included in the original database.
  • Step 203 Each Map node traverses one piece of fragment data in the original database D according to each K_ frequent item set to obtain a part of the projection database D' of each K_ frequent item set.
  • each Map node divides data in one piece of data of the original database D into two parts, a prefix and a suffix according to each K_ frequent item set, and a corresponding K_ frequent item set obtained by all Map nodes.
  • the projection database D' formed by the suffix portion is the projection database D' of the K_ frequent item set.
  • the K_ frequent item set is used as a prefix to query the sequence after the first occurrence of the prefix, if the last element of the prefix is the same as the first element of the found sequence. , then "_" and the sequence found are suffixed, otherwise, the sequence that is found is directly suffixed.
  • a projection database D' of the K_frequent item set composed of suffixes in each transaction is constructed.
  • the number of Map nodes can be multiple.
  • Each Map node obtains a piece of fragment data in the original database D.
  • the fragments here are simple data fragments, and each Map node uses each K_ frequent item set as a key. Traversing each transaction in the fragment data, obtaining key values corresponding to each key or each K_ frequent item set, and outputting the obtained key value to the Reduce node corresponding to the K_ frequent item set, so that the K_ frequent item
  • the data received by the corresponding Reduce node from all Map nodes constitutes the projection database D' of the K_ frequent itemsets.
  • Map node 1 divides the first fragment data of the original database D into two parts, prefix and suffix, based on m K_ frequent itemsets.
  • Node 2 divides the second fragment data of the original database D into two parts, a prefix and a suffix, based on m K_ frequent itemsets, and
  • Map node 3 divides the third fragment of the original database D based on m K_ frequent itemsets.
  • the data is divided into two parts, a prefix and a suffix.
  • _ in the projection database D' is used to represent the prefix itself, and the prefix is an element of the first item in the suffix, and the first item has other elements in addition to the prefix.
  • the method provided in this embodiment is different from the current frequent item set obtained by the correlation analysis of different Reduce nodes, and thus the candidate frequent items set obtained by the correlation analysis of the Reduce node need not be summarized. Only then can merge and pruning, and finally get the global frequent itemsets. Therefore, this implementation
  • the method in the example can effectively avoid the aggregation of the candidate frequent itemsets and the local support, and the frequent itemsets of the reduce nodes are often the exponential multiple of the data volume of the slice data of the reduce node. Therefore, in this embodiment, The method greatly reduces the amount of data that is transferred.
  • Step 204 Each Reduce node receives a part of the projection database D′ of the corresponding K_ frequent item set from all Map nodes, obtains a projection database D′ of the corresponding K_ frequent item set, and performs parallel association on the projection database D′. Analytical processing.
  • the MapReduce software framework includes multiple Reduce nodes, and each Reduce node corresponds to a K_ frequent item set.
  • Each Reduce node receives a part of the projection database D′ of the corresponding K_ frequent item set from all Map nodes, thereby obtaining a projection database D′ of the corresponding K_ frequent item set, and further, the K_ frequent item set
  • the projection database D' performs association analysis processing to obtain a frequent item set and its support.
  • the Reduce node 1 performs association analysis processing based on the projection database D' 1
  • the Reduce node 2 performs association analysis processing based on the projection database D' 2
  • the Reduce node m performs association analysis processing based on the projection database D' m .
  • the reduction node may use the GSP algorithm to perform the association analysis process to obtain the foregoing frequent item set and its support degree, and may also use other association analysis algorithms without affecting the use effect of the method provided by the embodiment.
  • the medium GSP algorithm is merely illustrative of the method provided by this embodiment.
  • the GSP algorithm obtains a sequence of length i as an initial sequence by scanning the projection database D', and then generates a sequence of length i+1 by combining and pruning according to an initial sequence of length i, and generates a sequence.
  • the operations of scanning, merging, and pruning are iteratively performed until a new sequence is no longer generated, and the obtained frequent itemsets are represented by the obtained sequence.
  • the support degree of the candidate frequent item set is calculated based on the projection database D', and then filtered according to preset screening conditions to obtain a frequent item set.
  • the initial value of i should be equal to the sequence length of the frequent item set as a prefix.
  • GSP algorithm pseudo code as shown below can be used:
  • ⁇ (a,b)> indicates that a, b occur simultaneously
  • ⁇ a, b> indicates that a occurs first, and then b occurs.
  • k takes a positive integer greater than 2, for the two sequences s 1 and s 2 , if the first item of s 1 is removed, the resulting remaining string is the same as the remaining string obtained by removing the last item of s 2 , you can combine s 1 and s 2 .
  • the resulting candidate sequence is to add the last item of s 2 to the end of s 1 , where there are two ways to add for two different situations:
  • the last item of s 2 is a single element, the item will be added to the end of s 1 as a separate item, otherwise the last item of s 2 will be merged as part of the last item of s 1 s 1 .
  • Pruning Pruning the k_candidate frequent item set F k obtained by the combination, and obtaining the k_ frequent item set F k .
  • pruning means that if any subset of a k_candidate frequent item set F k is infrequent, then this k_candidate frequent item set F k will be removed.
  • the Reduce node obtains the result of the association analysis process, that is, the frequent item set and its support.
  • Step 205 Each Reduce node outputs an association analysis processing result.
  • each node may directly output the correlation analysis processing result, and may further deserialize the result according to the mapping table before outputting, thereby outputting the correlation analysis processing result after the deserialization processing.
  • Step 206 Summarize the results of the association analysis processing output by each Reduce node.
  • the master node may schedule the slave node to directly merge the output of eachuce node, and summarize the results into one file. Since the projection database D' of each K_ frequent item set does not contribute support to each other, the association mining including the pruning step can be performed on different projection databases by different nodes, and the correlation analysis output by each node is performed.
  • the processing result is a local frequent item set with a small amount of data, which avoids the need to transmit a local candidate frequent item set with a large amount of data obtained by each node without the pruning step in the prior art, thereby saving transmission overhead and improving The efficiency.
  • this embodiment provides a specific example to explain the method provided by this embodiment in detail.
  • FIG. 3 is a schematic diagram of the execution process of the association analysis method.
  • the original data in Table 1 is serialized.
  • each row of the original data corresponds to one user, and the column vector is, from left to right, the user identification and operation object, and the operation time of performing an operation on the operation object.
  • Table 1 Record table of raw data
  • the original data in Table 1 records that the users of A, B, and C performed three purchase operations on different dates, according to the following mapping table:
  • each row represents a transaction, and each transaction is separated by a comma in each transaction.
  • the support degree 2 is set as the minimum support degree threshold in advance, that is, the candidate frequent itemsets whose support degree is less than 2 will be filtered out, thereby obtaining the 1_frequent item set ⁇ a>, ⁇ b>, ⁇ c> .
  • a part of a projection database of each 1_frequent item set is constructed in each Map node, and the Reduce node receives a part of the projection database corresponding to the 1_frequent item set from each Map node, and obtains a projection database of the 1_frequent item set. Perform correlation analysis processing.
  • a prefix of the constructed database projection D' a is as follows:
  • the projection data D' b of the prefix b is constructed, and the projection database D'b corresponding to the constructed prefix b is as follows:
  • the projection data D' c of the prefix c is constructed, and the projection database D'c corresponding to the constructed prefix c is as follows:
  • the 1_frequent item set ⁇ a>, ⁇ b>, ⁇ c> has been obtained in the previous steps.
  • the 2_candidate frequent itemsets obtained from the 1_frequent item set are as follows:
  • ⁇ (a,b)> indicates that a, b occur simultaneously
  • ⁇ a, b> indicates that a occurs first, and then b occurs.
  • the database D has been divided into a projection database D' a of prefix a , a projection database D' b of prefix b , a projection database D' c of prefix c , and associated analysis processing by different Reduce nodes, respectively, as in Reduce node 1.
  • ⁇ (a,b)>, ⁇ (a,c)>, ⁇ a, a>, ⁇ a, b>, ⁇ a, c> only need to calculate the support based on the projection database D' a .
  • Reduce node 2 only needs to calculate support based on projection database D' b for ⁇ (b,c)>, ⁇ b, a>, ⁇ b, b>, ⁇ b, c>.
  • the Reduce node 3 only needs to calculate the support based on the projection database D' c for ⁇ c, a>, ⁇ c, b>, ⁇ c, c>.
  • the same support degree threshold can be set as the filtering condition for each Reduce node.
  • the filter passes and remains, otherwise it is filtered out without being reserved.
  • the support threshold is 1, each Reduce node retains the following frequent itemsets:
  • Each Reduce node pairs the filtered frequent itemsets, and performs deserialization processing according to the mapping table, and then each The Reduce node outputs the result of the correlation analysis processing after deserialization processing, and Table 9 shows the result of the correlation analysis processing after the deserialization processing output by each Reduce node.
  • the initial item set is used as a prefix to establish a projection database of each prefix, and each node separately performs sequence on the corresponding projection database.
  • the association analysis of the pattern is then summarized for the local frequent itemsets obtained by each node and the corresponding support degrees. Since there is no mutual inclusion relationship between at least two initial item sets, the projection databases thus established do not contribute to the support of frequent item sets, and different nodes may separately perform pruning steps on different projection databases.
  • the local frequent itemsets obtained by each node are small in data volume, and avoid the need to transmit the local candidate frequent itemsets of the data obtained by the nodes without the pruning step in the prior art. Situation, which saves transmission overhead and improves efficiency.
  • FIG. 4 is a schematic structural diagram of an association analysis apparatus according to an embodiment of the present invention. As shown in FIG. 4, the method includes: a division module 42, an analysis module 43, and a summary module 44.
  • the dividing module 42 is configured to divide the original database into the projection databases, and the projection databases do not contribute to the support of the frequent item sets.
  • the analysis module 43 is configured to perform a correlation analysis of the sequence patterns on the projection database by each node to obtain a local frequent item set and a corresponding support degree.
  • the analyzing module 43 is specifically configured to perform a step of scanning, merging, and pruning the projection database by using a preset association analysis algorithm to obtain a local frequent item set and a corresponding support degree.
  • the correlation analysis algorithm includes a GSP algorithm.
  • the summary module 44 is configured to summarize the local frequent item set obtained by each node association analysis and the corresponding support degree, and obtain a global frequent item set and a corresponding support degree.
  • FIG. 5 is a schematic structural diagram of another association analysis device provided in the third embodiment.
  • the division module 42 includes: an obtaining unit 421. , unit 422 is established.
  • the obtaining unit 421 is configured to perform association analysis of the sequence mode on the original database to obtain at least two initial item sets.
  • the establishing unit 422 is configured to use each initial item set as a prefix to establish a projection database of each prefix.
  • the obtaining unit 421 includes: a calculating subunit 4211, a determining subunit 4212, and a screening subunit 4213.
  • the calculation subunit 4211 is configured to calculate a k_ frequent item set of the original database.
  • k is a positive integer and 1 ⁇ k ⁇ N, where N is the number of elements contained in the original database.
  • the determining subunit 4212 is configured to use the k_ frequent item set as the initial item set.
  • the filtering sub-unit 4213 is configured to filter the k_ frequent item set according to the preset support degree threshold, and retain the k_ frequent item set whose support degree is greater than the support degree threshold.
  • the establishing unit 422 includes: a query subunit 4221 and a setup subunit 4222.
  • a query subunit 4221 configured to use the initial item set as a prefix, and query a suffix corresponding to the prefix in each transaction of the original database;
  • a subunit 4222 is created for summarizing the suffixes of the transactions to form a projection database of the prefix.
  • the initial item set is used as a prefix to establish a projection database of each prefix, and each node separately performs sequence on the corresponding projection database.
  • Association analysis of patterns Since there is no mutual inclusion relationship between at least two initial item sets, the projection databases thus established do not contribute to the support of frequent item sets, and may be different by different nodes.
  • the projection database performs association mining including the pruning step, and each node obtains a local frequent item set with a small amount of data, thereby avoiding the amount of data obtained in the prior art that needs to transmit the nodes without the pruning step. The case of large local candidate frequent itemsets saves transmission overhead and improves efficiency.
  • FIG. 6 is a schematic structural diagram of an association analysis apparatus according to the fourth embodiment.
  • the apparatus in the embodiment further includes: an allocation module 61.
  • the allocating module 61 is configured to allocate, to each projection database, a node whose load capacity matches the data amount of the projection database.
  • association analysis device further includes: a serialization module 62 and a deserialization module 63.
  • the serialization module 62 is configured to serialize the original data according to the mapping table to obtain the original database.
  • the deserialization module 63 is configured to deserialize the local frequent item set before summarizing the local frequent item set obtained by each node association analysis and the corresponding support degree; or obtain a global frequent item set and corresponding support After the degree, the global frequent item set is deserialized.
  • the initial item set is used as a prefix to establish a projection database of each prefix, and each node separately performs sequence on the corresponding projection database.
  • Association analysis of patterns Since there is no mutual inclusion relationship between at least two initial item sets, the projection databases thus established do not contribute to the support of frequent item sets, and different nodes may separately perform pruning steps on different projection databases.
  • the local frequent itemsets obtained by each node are small in data volume, and avoid the need to transmit the local candidate frequent itemsets of the data obtained by the nodes without the pruning step in the prior art. Situation, which saves transmission overhead and improves efficiency.
  • each projection database is assigned a node whose load capacity matches the data volume of the projection database, so that some nodes are relatively idle, and other nodes are overloaded. Further improve the efficiency of correlation analysis
  • the aforementioned program can be stored in a computer readable storage medium.
  • the program when executed, performs the steps including the foregoing method embodiments; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Strategic Management (AREA)
  • Finance (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Accounting & Taxation (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Software Systems (AREA)
  • Algebra (AREA)
  • Game Theory and Decision Science (AREA)
  • Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Operations Research (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明提供了关联分析方法和装置，通过将原始数据库划分为相互之间不贡献频繁项集支持度的各投影数据库，由各节点分别对所对应的投影数据库进行序列模式的关联分析，然后对各节点获得的局部频繁项集以及对应支持度进行汇总。由于所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。

Description

关联分析方法和装置
本申请要求2016年02月22日递交的申请号为201610096728.8、发明名称为“关联分析方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及计算机技术，尤其涉及一种关联分析方法和装置。
背景技术
关联分析是进行数据挖掘中的一种分析技术，主要用于根据大量数据发现项目之间的关联性。关联分析的一个典型应用实例便是购物篮分析：基于购物数据进行关联分析，发现顾客放入购物篮中的不同商品之间的关联性。进而由这种关联分析所获得的关联性体现出顾客的购买习惯，通过了解这些购买习惯可以有利于零售商制定营销策略。
序列模式关联分析不同于其他模式下的关联分析，序列模式还考虑了项目发生的时间，从而使得各项目之间有一定的顺序性。针对序列模式的关联分析，其所进行分析的对象往往是超大规模的数据库，因此，运算量很大不适宜采用单机进行数据处理。现有技术中通常采用对数据库中的数据进行简单分片，将每个分片数据在各节点进行单独的关联分析，获得候选频繁项集及其支持度，然后进行合并获得各候选频繁项集的全局支持度，进而依据预设的筛选条件进行剪枝后获得全局频繁项集。
但是，由于在进行关联分析生成候选频繁项集的过程中存在数据膨胀，导致候选频繁项集的数据量是分片数据的数据量的指数倍，因此，在对各节点生成的候选频繁项集及其支持度进行汇总以便执行合并操作时，传输数据量过大，导致执行效率较低。
发明内容
本发明提供一种关联分析方法和装置，用于解决现有技术中进行并行关联分析时由于汇总数据时数据传输量过大导致执行效率较低的技术问题。
为达到上述目的，本发明的实施例采用如下技术方案：
第一方面，提供一种关联分析方法，包括：
将原始数据库划分为各投影数据库，所述各投影数据库相互之间不贡献频繁项集的支持度；
由各节点分别对所对应的投影数据库进行序列模式的关联分析，获得局部频繁项集 以及对应支持度；
对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总，获得全局频繁项集以及对应支持度。
第二方面，提供一种关联分析装置，包括：
划分模块，用于将原始数据库划分为各投影数据库，所述各投影数据库相互之间不贡献频繁项集的支持度；
分析模块，用于由各节点分别对所述投影数据库进行序列模式的关联分析，获得局部频繁项集以及对应支持度；
汇总模块，用于对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总，获得全局频繁项集以及对应支持度。
本发明实施例提供的关联分析方法和装置，通过将原始数据库划分为相互之间不贡献支持度的各投影数据库，由各节点分别对所对应的投影数据库进行序列模式的关联分析，然后对各节点获得的局部频繁项集以及对应支持度进行汇总。由于所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。
上述说明仅是本发明技术方案的概述，为了能够更清楚了解本发明的技术手段，而可依照说明书的内容予以实施，并且为了让本发明的上述和其它目的、特征和优点能够更明显易懂，以下特举本发明的具体实施方式。
附图说明
通过阅读下文优选实施方式的详细描述，各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的，而并不认为是对本发明的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：
图1为本发明实施例一提供的一种关联分析方法的流程示意图；
图2为本发明实施例二提供的一种关联分析方法的流程示意图；
图3为关联分析方法执行过程的示意图；
图4为本发明实施例所提供的一种关联分析装置的结构示意图；
图5为实施例三提供的另一种关联分析装置的结构示意图；
图6为实施例四提供的一种关联分析装置的结构示意图。
具体实施方式
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例，然而应当理解，可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更透彻地理解本公开，并且能够将本公开的范围完整的传达给本领域的技术人员。
在待进行关联分析的数据库中，通常记载了各个事务，事务又包括了各个项目，每一个项目包括至少一个元素。其中，一个元素用于指示一个操作对象，项目用于指示由同一用户同时进行操作的各操作对象，而一个用户在不同时刻所进行操作的操作对象可以用一个事务标识。由于序列模式下的关联分析需要考虑时序性，因此，每一个事务所包括的各个项目是具有一定顺序性的，这种具有顺序的各个项目可以称之为一个序列，因此，也可以说事务是由一个序列进行表示的。
比如数据库中的一个事务为序列abc,abc,ac,d,cf，可以用于表示一个用户分别在第一天买了商品a、b和c，第二天又买了商品a、b和c，第三天买了商品a和c。针对每一天买的总商品叫项目，每件商品叫元素。
发明人针对现有技术中的关联分析方法进行分析，发现现有技术中在对各节点生成的候选频繁项集及其支持度进行汇总以便执行合并操作时，传输数据量过大，主要是由于各节点未在本地执行剪枝的步骤。发明人在此基础上，进行了进一步分析，现有技术中对原始数据仅进行了简单分片，各个分片数据对于某个频繁项级都存在贡献支持度的可能，从而无法在本地执行剪枝的步骤，因此，需要将原始数据划分为相互之间不存在支持度的数据库才能够在节点本地进行剪枝，进而避免传输候选频繁项集及其支持度。基于这一思路，发明人提出了本发明所提供的关联分析方法。
为了便于理解本发明所提供的关联分析方法，在描述具体实施例之前，对实施例中所涉及的技术术语进行解释：
序列模式关联分析是指：给定一个数据库，其中，数据库包括了各个事务，每一个事务由一个序列表示，每个序列由相互之间具有一定顺序性的项目组成。序列模式挖掘就是在给定一个支持度阈值的基础上，找出所有满足在数据库中的出现频次不低于该支持度阈值的子序列，将这些子序列作为频繁项集，从数据库中找出这些频繁项集的操作便是序列模式关联分析。
项集是指：数据库针对同一事务中所出现的项目有序排列所构成的集合。
项集的长度是指：用于表示该项集的序列的长度，数值上等于项集所包括的项目个数。
支持度是指：项集在数据库中出现的频次，若一个事务中包含该项集，则记为一次，从而支持度等于数据库中包含某一项集的事务个数。
频繁项集是指：数据库中所有出现频次不小于支持度阈值的项集。
投影数据库是指：针对原始数据库进行投影操作所获得的数据库，具体来说α的投影数据库为S中所有以α为前缀的序列相对于α的后缀。
此处简要解释了技术术语的含义，以上技术术语会在后续结合具体实施例进行进一步地解释。
下面结合附图对本发明实施例提供的关联分析方法和装置进行详细描述。
实施例一
图1为本发明实施例一提供的一种关联分析方法的流程示意图，如图1所示，包括：
步骤101、将原始数据库划分为各投影数据库。
其中，各投影数据库相互之间不贡献频繁项集的支持度。其中，支持度是指在数据库中包含某一项集的事务个数。各投影数据库相互之间不贡献频繁项集的支持度，也就是说基于一投影数据库进行序列模式挖掘所获得的频繁项集未出现在另一投影数据库中，从而另一投影数据库不会增加频繁项集的支持度，因此说另一投影数据库不贡献一投影数据库的频繁项集的支持度。
具体地，首先，获得至少两个初始项集，其中初始项集是对原始数据库进行关联分析所获得的项集，用于构建投影数据库，并且至少两个初始项集中的任意两初始项集之间不存在相互包含关系。具体可以通过针对原始数据库可以采用关联分析算法进行计算，获得前述的至少两项集，这两项集可以是相同长度也可以是不同长度，可以是执行过剪枝步骤获得的频繁项集也可以是未执行过剪枝步骤获得的候选频繁项集，本实施例中对此不做限定。
进而，将各初始项集作为前缀，建立各前缀的投影数据库。具体可以通过将初始项集作为前缀，在原始数据库的各事务中查询前缀所对应的后缀，进而将各事务的后缀进行汇总，形成前缀的投影数据库。其中，后缀是通过在每一条事务中查询该前缀首次出现的位置之后的序列，若前缀的最后一个元素与所查找到的序列的第一个元素的时序相同，则将“_”和所查找到的序列作为后缀，否则，直接将所查找到的序列作为后缀。
由于初始项集两两之间的不存在包含关系，所以据此所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。
需要说明的是，这里定义的包含关系是前缀的包含关系，也就说一个初始项集是否为另一初始项集的前缀，若是，则存在包含关系，例如：
针对<a，b>和<a，c，b>，<a，b>不是<a，c，b>的前缀，因此<a，b>和<a，c，b>两者之间不存在包含关系；
针对<a，b>和<a>，<a>是<a，b>的前缀，因此<a，b>和<a>属于包含关系。
作为一种可能的实现方式，可以计算原始数据库的K_频繁项集，也就是项集中包含K个项目的频繁项集，其中K为正整数，且1≤K＜N，N为原始数据库中所包含的元素数，将K_频繁项集作为初始项集。
作为另一种可能的实现方式，还可以计算原始数据库的K_频繁项集之后，根据预设的支持度阈值，对K_频繁项集进行筛选，保留支持度大于支持度阈值的K_频繁项集，将筛选后的K_频繁项集作为初始项集。经过筛选步骤之后，合理减小了后续进行处理的数据量，减轻了节点构建投影数据库以及基于投影数据库进行关联分析的运算压力，同时，也减少了后续对节点关联分析所获得的局部频繁项集进行汇总时的数据传输总量。
步骤102、由各节点分别对所对应的投影数据库进行序列模式的关联分析，获得局部频繁项集以及对应支持度。
具体的，为每一投影数据库分配节点，例如：可以为各投影数据库分配负载能力与所述投影数据库的数据量相匹配的节点。然后由各节点采用预设关联分析算法并行进行序列模式的关联分析，如广义序贯模式(Generalized Sequential Pattern，GSP)算法，具体来说，各节点对投影数据库执行扫描、合并和剪枝的步骤，获得频繁项集以及对应支持度。每个节点仅能够获得原始数据库的各频繁项集中的一部分频繁项集，为了与原始数据库的全部频繁项集进行区分，将每个节点所获得的频繁项集称为局部频繁项集，将原始数据库的全部频繁项集称为全局频繁项集。另外需要说明的是，这里所说的节点运行在一个单机上，单机可以是物理机也可以是虚拟机本实施例中对此不做限定。
通过为各投影数据库分配负载能力与所述投影数据库的数据量相匹配的节点，避免出现有些节点负载过重而另一些节点空闲的情况出现，从而更加合理和高效地利用现有 节点，加快关联分析速度，提高关联分析的效率。
步骤103、对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总，获得全局频繁项集以及对应支持度。
具体地，将各节点关联分析所获得的局部频繁项集以及对应支持度汇总到一个文件既可，从而该文件中记录的为全局频繁项集以及对应支持度，而无需执行合并和剪枝的步骤。
本实施例中，通过将原始数据库划分为相互之间不贡献支持度的各投影数据库，由各节点分别对所对应的投影数据库进行序列模式的关联分析，然后对各节点获得的局部频繁项集以及对应支持度进行汇总。由于所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。
实施例二
图2为本发明实施例二提供的一种关联分析方法的流程示意图，本实施例所提供的关联分析方法可以由软件执行，该软件可以运行在大数据计算服务(Open Data Processing Service,简称ODPS)平台上，本实施例所提供的方法可以运行在映射规约(MapReduce)软件框架下由多个节点执行，在MapReduce软件框架下包括一个主节点(master)，和多个从节点(workers)，主节点可以对各从节点分配映射(Map)作业(用来把一组键值对映射成一组新的键值对)或者规约(Reduce)作业(用来保证所有映射的键值对中的每一个共享相同的键组)，分配到Map作业的从节点又可以称为Map节点，分配到Reduce作业的从节点又可以称为Reduce节点。每一个节点可以运行在一个物理机或者虚拟机上，本实施例中对此不做限定。如图2所示，方法包括：
步骤201、主节点对从节点进行调度以使从节点进行数据预处理。
具体地，主节点将数据预处理作为一项MapReduce任务，调度从节点执行该任务，从而针对原始数据进行数据序列化操作。具体通过根据映射表将原始数据中所记载的各用户执行操作的不同的操作对象分别用数字或者字母序列进行标识，并针对同一用户执行操作的操作对象按照操作时间进行排序。
步骤202、从节点在主节点调度下基于原始数据库D计算K_频繁项集，将每一个 K_频繁项集分配至一个Map节点进行处理。
需要说明的是，1≤K＜N，N为原始数据库中所包含的元素数。
步骤203、各Map节点根据每一个K_频繁项集遍历原始数据库D中的一个分片数据，获得每一个K_频繁项集的投影数据库D’的一部分。
具体地，每一个Map节点均根据各K_频繁项集将原始数据库D的一个分片数据中的数据划分为前缀和后缀两部分，由全部Map节点所获得的对应某一个K_频繁项集的后缀部分所构成的投影数据库D’即为该K_频繁项集的投影数据库D’。具体来说，在各事务中，将该K_频繁项集作为前缀，查询该前缀首次出现的位置之后的序列，若前缀的最后一个元素与所查找到的序列的第一个元素的时序相同，则将“_”和所查找到的序列作为后缀，否则，直接将所查找到的序列作为后缀。构建由各事务中的后缀所构成的该K_频繁项集的投影数据库D’。
Map节点的个数可以为多个，各Map节点获取到原始数据库D中的一个分片数据，这里的分片是简单的数据分片，每一个Map节点将各K_频繁项集作为键，遍历分片数据中的各事务，获得各个键或者说各个K_频繁项集对应的键值，将所获得的键值输出至该K_频繁项集对应的Reduce节点，从而该K_频繁项集对应的Reduce节点从全部Map节点所接收到的数据构成了该K_频繁项集的投影数据库D’。例如：若存在m个K_频繁项集，3个分片数据时，Map节点1基于m个K_频繁项集将原始数据库D的第一个分片数据划分为前缀和后缀两部分，Map节点2基于m个K_频繁项集将原始数据库D的第二个分片数据划分为前缀和后缀两部分，Map节点3基于m个K_频繁项集将原始数据库D的第三个分片数据划分为前缀和后缀两部分。
需要说明的是，投影数据库D’中“_”用于表示前缀自身，且该前缀为后缀中第一个项目的元素，该第一个项目除了该前缀外还有其他元素。
对于相同前缀的各事务，可以看出其生成的频繁项集也具有相同的前缀，因此将各事务中同一前缀首次出现的位置之后的序列合并为投影数据库，不同投影数据库所关联分析获得的频繁项集不会出现重复的情况，因而也就互不贡献支持度。从而不同的Reduce节点可以针对不同的投影数据库独立的进行关联分析挖掘，包括对候选频繁项集剪枝获得频繁项集的过程也在本地Reduce节点，避免了汇总候选频繁项集及其本地支持度。
可见，本实施例中所提供的方法不同于目前不同的Reduce节点关联分析获得的候选频繁项集会出现重复的情况，因而不需要汇总Reduce节点所关联分析获得的候选频繁项集其本地支持度之后，才能够进行合并和剪枝，最终获得全局频繁项集。因此，本实施 例中的方法，能够有效避免汇总候选频繁项集及其本地支持度，而各Reduce节点候选频繁项集的往往是该Reduce节点的分片数据的数据量的指数倍，因此，本实施例中的方法极大减少了进行数据传输的数据量。
步骤204、各Reduce节点从全部Map节点接收所对应的K_频繁项集的投影数据库D’的一部分，获得所对应的K_频繁项集的投影数据库D’，对投影数据库D’进行并行关联分析处理。
具体的，MapReduce软件框架包括多个Reduce节点，每一个Reduce节点对应一个K_频繁项集。每一个Reduce节点从全部Map节点接收所对应的K_频繁项集的投影数据库D’的一部分，从而获得所对应的K_频繁项集的投影数据库D’，进而对该K_频繁项集的投影数据库D’进行关联分析处理，获得频繁项集及其支持度。
例如：Reduce节点1基于投影数据库D’1进行关联分析处理，Reduce节点2基于投影数据库D’2进行关联分析处理，……Reduce节点m基于投影数据库D’m进行关联分析处理。
其中，各Reduce节点可以采用GSP算法进行关联分析处理从而获得前述的频繁项集及其支持度，也可以采用其他关联分析算法而不会影响本实施例所提供的方法的使用效果，本实施例中GSP算法仅作为示例说明本实施例所提供的方法。GSP算法是通过扫描投影数据库D’得到长度为i的序列作为初始序列，然后根据长度为i的初始序列，经过合并和剪枝的操作，生成长度为i+1的序列，并将生成的序列作为新的初始序列，重复迭代执行扫描、合并和剪枝的操作，直至不再生成新的序列，用所获得的序列表示候选频繁项集。基于投影数据库D’计算候选频繁项集的支持度，然后根据预设的筛选条件进行筛选，获得频繁项集。
需要说明的是，i的初始取值应当等于作为前缀的频繁项集的序列长度。
具体可以采用如下所示的GSP算法伪代码：
Figure PCTCN2017073135-appb-000001
Figure PCTCN2017073135-appb-000002
其中，candidate-gen-SPM(Fk-1)的算法流程如下：
1、合并：对所生成的k-1_候选频繁项集Fk-1进行合并，生成k_候选频繁项集Fk。
具体地，当k＝2时，对于两个1_候选项集F1的序列s1和s2，需要将s2的项目要以s1的项目中的一部分和以一个单独的项目两种方式合并到s1。即合并<a>和<b>，生成的候选频繁项级有<(a，b)>、<a，a>、<a，b>、<b，a>和<b，b>。
需要说明的是，<(a，b)>表示a，b同时发生，<a，b>表示先发生a，后发生b。
当k取大于2的正整数时，对于两个序列s1和s2，如果将s1的第1个项目去掉后得到的余串和将s2的最后一个项目去掉后得到的余串相同，则可以将s1和s2合并。所得的候选序列是将s2的最后一个项目添加到s1末尾，这里针对两种不同情况有两种添加方式：
如果s2最后一个项目是一个单独的元素，则这个项目将以一个单独项目的形式加到s1的末尾，否则，s2最后一个项目将作为s1的最后一个项目中的一个部分合并入s1。
2、剪枝：对合并所获得的k_候选频繁项集Fk进行剪枝，获得k_频繁项集Fk。
具体地，剪枝是指如果一个k_候选频繁项集Fk的任何一个子集是非频繁的，则这个k_候选频繁项集Fk将被去除。
Reduce节点基于前述扫描、合并和剪枝的步骤以及筛选的步骤，获得关联分析处理结果，即频繁项集及其支持度。
步骤205、各Reduce节点对关联分析处理结果输出。
具体的，各节点可以直接输出关联分析处理结果，还可以在输出之前，根据映射表对结果进行反序列化处理，从而输出反序列化处理后的关联分析处理结果。
步骤206、对各Reduce节点输出的关联分析处理结果进行汇总。
具体的，可以主节点调度从节点直接合并各Reduce节点输出的结果，汇总为一个文件既可。由于各K_频繁项集的投影数据库D’相互之间不贡献支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所输出的关联分析 处理结果为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。
为了清楚说明本实施例，本实施例提供了一个具体的实例以对本实施例所提供的方法进行详细说明。
例如：
图3为关联分析方法执行过程的示意图，如图3所示，针对表1中的原始数据进行序列化。在表1中，原始数据的每一行对应一个用户，列向量从左至右依次为用户标识和操作对象以及对该操作对象执行操作的操作时间。
表1原始数据的记录表
甲用户 苹果梨桃 2月1日 苹果梨桃 2月2日 苹果桃 2月3日
乙用户 梨桃 2月3日 苹果 2月4日 苹果梨 2月5日
丙用户 苹果 2月6日 梨 2月7日 香蕉 2月8日
表1中的原始数据记录到了甲、乙、丙用户分别于不同日期执行了三次购买操作，根据如下映射表：
苹果→a
梨→b
桃→c
香蕉→d
执行数据序列化操作，获得原始数据库D。在原始数据库D中所记载的内容如下所示：
abc,abc,ac；
bc,a,ab；
a,b,d。
需要说明的是，每一行代表一个事务，在每一个事务中，用逗号分隔各个项目。
基于前述原始数据库D，进行序列模式的关联分析，计算1_候选频繁项集及其支持度，如下表所示。
表2 1_候选频繁项集及其支持度
候选频繁项集 支持度
<a> 3
<b> 3
<c> 2
<d> 1
若预先设置了以支持度2作为最小支持度阈值，也就是说支持度小于2的候选频繁项集将会被过滤掉，从而获得1_频繁项集<a>，<b>，<c>。
在每一个Map节点中构建各1_频繁项集的投影数据库的一部分，Reduce节点从每个Map节点接收对应1_频繁项集的投影数据库一部分，汇总获得该1_频繁项集的投影数据库，进行关联分析处理。
将1_频繁项集<a>作为前缀a，构建前缀a的投影数据D’a，所构建的前缀a的投影数据库D’a如下所示：
_bc,abc,ac
ab
b,d。
将1_频繁项集<b>作为前缀b，构建前缀b的投影数据D’b，所构建的前缀b对应的投影数据库D’b如下所示：
_c,abc,ac
_c,a,ab
d。
将1_频繁项集<c>作为前缀c，构建前缀c的投影数据D’c，所构建的前缀c对应的投影数据库D’c如下所示：
abc,ac
a,ab
在前面的步骤中已获得1_频繁项集<a>，<b>，<c>。根据1_频繁项集所获得的2_候选频繁项集如下所示：
<(a，b)>，<(a，c)>，<(b，c)>，<a，a>，<a，b>，<a，c>，<b，a>，<b，b>，<b，c>，<c，a>，<c，b>，<c，c>。
需要说明的是，<(a，b)>表示a，b同时发生，<a，b>表示先发生a，后发生b。
数据库D已被划分为前缀a的投影数据库D’a，前缀b的投影数据库D’b，前缀c的投影数据库D’c，并分别由不同Reduce节点进行关联分析处理，如在Reduce节点1中，对于<(a，b)>，<(a，c)>，<a，a>，<a，b>，<a，c>只需要基于投影数据库D’a计算支持度。
表3 Reduce节点1的支持度计算结果
频繁项集 支持度
<(a，b)> 2
<(a，c)> 1
<a，a> 2
<a，b> 3
<a，c> 1
同理，Reduce节点2对于<(b，c)>，<b，a>，<b，b>，<b，c>只需要基于投影数据库D’b计算支持度。
表4 Reduce节点2的支持度计算结果
频繁项集 支持度
<(b，c)> 2
<b，a> 2
<b，b> 2
<b，c> 1
Reduce节点3对于<c，a>，<c，b>，<c，c>只需要基于投影数据库D’c计算支持度。
表5 Reduce节点3的支持度计算结果
频繁项集 支持度
<c，a> 2
<c，b> 2
<c，c> 1
可以针对各个Reduce节点设置相同的支持度阈值作为筛选条件，当频繁项集的支持度大于该支持度阈值时，筛选通过并保留，否则筛选掉而不进行保留。当支持度阈值为1时，各Reduce节点分别保留如下频繁项集：
表6 Reduce节点1的筛选结果
频繁项集 支持度
<(a，b)> 2
<a，a> 2
<a，b> 3
表7 Reduce节点2的筛选结果
频繁项集 支持度
<(b，c)> 2
<b，a> 2
<b，b> 2
表8 Reduce节点3的筛选结果
频繁项集 支持度
<c，a> 2
<c，b> 2
各个Reduce节点对筛选后的频繁项集，根据映射表进行反序列化处理，进而各个 Reduce节点输出反序列化处理后的关联分析处理结果，表9为各Reduce节点输出的反序列化处理后的关联分析处理结果示意。
表9各Reduce节点输出的结果
Figure PCTCN2017073135-appb-000003
对各Reduce节点输出的结果进行汇总，汇总获得的文件内容如下所示：
表10汇总结果
频繁项集 支持度
<(苹果，梨)> 2
<苹果，苹果> 3
<苹果，梨> 3
<(梨，桃)> 2
<梨，苹果> 2
<梨，梨> 2
<桃，苹果> 2
<桃，梨> 2
本实施例中，通过对原始数据库进行关联分析计算，获得至少两个初始项集之后，将各初始项集作为前缀，建立各前缀的投影数据库，由各节点分别对所对应的投影数据库进行序列模式的关联分析，然后对各节点获得的局部频繁项集以及对应支持度进行汇总。由于至少两个初始项集之间不存在相互包含关系，所以据此所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。
实施例三
图4为本发明实施例所提供的一种关联分析装置的结构示意图，如图4所示，包括：划分模块42、分析模块43和汇总模块44。
划分模块42，用于将原始数据库划分为各投影数据库，所述各投影数据库相互之间不贡献频繁项集的支持度。
分析模块43，用于由各节点分别对所述投影数据库进行序列模式的关联分析，获得局部频繁项集以及对应支持度。
具体地，分析模块43具体用于采用预设关联分析算法，由节点对所述投影数据库执行扫描、合并和剪枝的步骤，获得局部频繁项集以及对应支持度。其中，关联分析算法包括GSP算法。
汇总模块44，用于对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总，获得全局频繁项集以及对应支持度。
进一步，本实施例还提供了一种关联分析装置的结构示意图，图5为实施例三提供的另一种关联分析装置的结构示意图，如图5所示，划分模块42，包括：获得单元421、建立单元422。
获得单元421，用于对原始数据库进行序列模式的关联分析，获得至少两个初始项集。
其中，所述至少两个初始项集之间不存在相互包含关系。
建立单元422，用于将各初始项集作为前缀，建立各前缀的投影数据库。
进一步，获得单元421，包括：计算子单元4211、确定子单元4212筛选子单元4213。
计算子单元4211，用于计算原始数据库的k_频繁项集。
其中k为正整数，且1≤k＜N，N为原始数据库中所包含的元素数。
确定子单元4212，用于将所述k_频繁项集作为所述初始项集。
筛选子单元4213，用于根据预设的支持度阈值，对k_频繁项集进行筛选，保留支持度大于所述支持度阈值的k_频繁项集。
进一步，建立单元422，包括：查询子单元4221和建立子单元4222。
查询子单元4221，用于将所述初始项集作为前缀，在所述原始数据库的各事务中查询所述前缀所对应的后缀；
建立子单元4222，用于将各事务的后缀进行汇总形成所述前缀的投影数据库。
本实施例中，通过对原始数据库进行关联分析计算，获得至少两个初始项集之后，将各初始项集作为前缀，建立各前缀的投影数据库，由各节点分别对所对应的投影数据库进行序列模式的关联分析。由于至少两个初始项集之间不存在相互包含关系，所以据此所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同 的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。
实施例四
图6为实施例四提供的一种关联分析装置的结构示意图，在上一实施例的基础上，本实施例中的装置，还包括：分配模块61。
分配模块61，用于为各投影数据库分配负载能力与所述投影数据库的数据量相匹配的节点。
进一步，关联分析装置还包括：序列化模块62和反序列化模块63。
序列化模块62，用于根据映射表，对原始数据进行序列化获得原始数据库。
反序列化模块63，用于对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总之前，对所述局部频繁项集进行反序列化；或者，获得全局频繁项集以及对应支持度之后，对所述全局频繁项集进行反序列化。
本实施例中，通过对原始数据库进行关联分析计算，获得至少两个初始项集之后，将各初始项集作为前缀，建立各前缀的投影数据库，由各节点分别对所对应的投影数据库进行序列模式的关联分析。由于至少两个初始项集之间不存在相互包含关系，所以据此所建立的投影数据库相互之间不贡献频繁项集的支持度，可以由不同节点分别对不同的投影数据库进行包括剪枝步骤在内的关联挖掘，各节点所获得的为数据量较小的局部频繁项集，避免现有技术中需要传输各节点未经过剪枝步骤所获得的数据量较大的局部候选频繁项集的情况，从而节省了传输开销，提高了效率。另外，针对各投影数据库可能具有不同的规模的特点，为各投影数据库分配负载能力与所述投影数据库的数据量相匹配的节点，避免出现有些节点较为空闲，而另外一些节点过载的情况发生从而进一步提高关联分析的效率
本领域普通技术人员可以理解：实现上述各方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成。前述的程序可以存储于一计算机可读取存储介质中。该程序在执行时，执行包括上述各方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其 依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (20)

  1. 一种关联分析方法，其特征在于，包括：
    将原始数据库划分为各投影数据库，所述各投影数据库相互之间不贡献频繁项集的支持度；
    由各节点分别对所对应的投影数据库进行序列模式的关联分析，获得局部频繁项集以及对应支持度；
    对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总，获得全局频繁项集以及对应支持度。
  2. 根据权利要求1所述的关联分析方法，其特征在于，所述将原始数据库划分为各投影数据库，包括：
    对所述原始数据库进行序列模式的关联分析，获得至少两个初始项集；其中，各初始项集之间不存在相互包含关系；
    将各初始项集作为前缀，建立各前缀的投影数据库。
  3. 根据权利要求2所述的关联分析方法，其特征在于，所述对原始数据库进行关联分析计算，获得至少两个初始项集，包括：
    计算原始数据库的K_频繁项集；其中K为正整数，且1≤K＜N，N为原始数据库中所包含的元素数；
    将所述K_频繁项集作为所述初始项集。
  4. 根据权利要求3所述的关联分析方法，其特征在于，所述将K_频繁项集作为所述初始项集之前，还包括：
    根据预设的支持度阈值，对K_频繁项集进行筛选，保留支持度大于所述支持度阈值的K_频繁项集。
  5. 根据权利要求2所述的关联分析方法，其特征在于，所述将各初始项集作为前缀，建立各前缀的投影数据库，包括：
    将所述初始项集作为前缀，在所述原始数据库的各事务中查询所述前缀所对应的后缀；
    将各事务的后缀进行汇总形成所述前缀的投影数据库。
  6. 根据权利要求1所述的关联分析方法，其特征在于，所述由各节点分别对所对应的投影数据库进行序列模式的关联分析之前，包括：
    为各投影数据库分配负载能力与所述投影数据库的数据量相匹配的节点。
  7. 根据权利要求1所述的关联分析方法，其特征在于，所述由各节点分别对所对应的投影数据库进行序列模式的关联分析，获得局部频繁项集以及对应支持度，包括：
    采用预设关联分析算法，由所述节点对所述投影数据库执行扫描、合并和剪枝的步骤，获得局部频繁项集以及对应支持度。
  8. 根据权利要求7所述的关联分析方法，其特征在于，所述关联分析算法包括GSP算法。
  9. 根据权利要求1-8任一项所述的关联分析方法，其特征在于，所述对原始数据库进行关联分析计算，获得至少两个初始项集之前，还包括：
    根据映射表，对原始数据进行序列化获得原始数据库。
  10. 根据权利要求9所述的关联分析方法，其特征在于，
    所述对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总之前，对所述局部频繁项集进行反序列化；
    或者，获得全局频繁项集以及对应支持度之后，对所述全局频繁项集进行反序列化。
  11. 一种关联分析装置，其特征在于，包括：
    划分模块，用于将原始数据库划分为各投影数据库，所述各投影数据库相互之间不贡献频繁项集的支持度；
    分析模块，用于由各节点分别对所述投影数据库进行序列模式的关联分析，获得局部频繁项集以及对应支持度；
    汇总模块，用于对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总，获得全局频繁项集以及对应支持度。
  12. 根据权利要求11所述的关联分析装置，其特征在于，所述划分模块，包括：
    获得单元，用于对原始数据库进行序列模式的关联分析，获得至少两个初始项集；其中，各初始项集之间不存在相互包含关系；
    建立单元，用于将各初始项集作为前缀，建立各前缀的投影数据库。
  13. 根据权利要求12所述的关联分析装置，其特征在于，所述获得单元，包括：
    计算子单元，用于计算原始数据库的K_频繁项集；其中K为正整数，且1≤K＜N，N为原始数据库中所包含的元素数；
    确定子单元，用于将所述K_频繁项集作为所述初始项集。
  14. 根据权利要求13所述的关联分析装置，其特征在于，所述获得单元，还包括：
    筛选子单元，用于根据预设的支持度阈值，对K_频繁项集进行筛选，保留支持度 大于所述支持度阈值的K_频繁项集。
  15. 根据权利要求12所述的关联分析装置，其特征在于，所述建立单元，包括：
    查询子单元，用于将所述初始项集作为前缀，在所述原始数据库的各事务中查询所述前缀所对应的后缀；
    建立子单元，用于将各事务的后缀进行汇总形成所述前缀的投影数据库。
  16. 根据权利要求11所述的关联分析装置，其特征在于，所述装置，还包括：
    分配模块，用于为各投影数据库分配负载能力与所述投影数据库的数据量相匹配的节点。
  17. 根据权利要求11所述的关联分析装置，其特征在于，
    所述分析模块，具体用于采用预设关联分析算法，由节点对所述投影数据库执行扫描.合并和剪枝的步骤，获得局部频繁项集以及对应支持度。
  18. 根据权利要求17所述的关联分析装置，其特征在于，所述关联分析算法包括GSP算法。
  19. 根据权利要求11-18任一项所述的关联分析装置，其特征在于，所述装置，还包括：
    序列化模块，用于根据映射表，对原始数据进行序列化获得原始数据库。
  20. 根据权利要求19所述的关联分析装置，其特征在于，所述装置，还包括：
    反序列化模块，用于对各节点关联分析所获得的局部频繁项集以及对应支持度进行汇总之前，对所述局部频繁项集进行反序列化；或者，获得全局频繁项集以及对应支持度之后，对所述全局频繁项集进行反序列化。
PCT/CN2017/073135 2016-02-22 2017-02-09 关联分析方法和装置 Ceased WO2017143908A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/107,927 US10956395B2 (en) 2016-02-22 2018-08-21 Association analysis method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610096728.8 2016-02-22
CN201610096728.8A CN107102999B (zh) 2016-02-22 2016-02-22 关联分析方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/107,927 Continuation US10956395B2 (en) 2016-02-22 2018-08-21 Association analysis method and apparatus

Publications (1)

Publication Number Publication Date
WO2017143908A1 true WO2017143908A1 (zh) 2017-08-31

Family

ID=59658628

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/073135 Ceased WO2017143908A1 (zh) 2016-02-22 2017-02-09 关联分析方法和装置

Country Status (4)

Country Link
US (1) US10956395B2 (zh)
CN (1) CN107102999B (zh)
TW (1) TWI730043B (zh)
WO (1) WO2017143908A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10956395B2 (en) 2016-02-22 2021-03-23 Alibaba Group Holding Limited Association analysis method and apparatus
CN117931930A (zh) * 2024-01-26 2024-04-26 西安电子科技大学 一种基于活动图的协作模式挖掘方法

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105590223A (zh) * 2014-12-29 2016-05-18 中国银联股份有限公司 商户的商圈信息的标定
CN107766442B (zh) * 2017-09-21 2019-02-01 深圳金融电子结算中心有限公司 一种海量数据关联规则挖掘方法及系统
CN108304465A (zh) * 2017-12-27 2018-07-20 重庆邮电大学 一种基于传感节点标识符平台的信息管理和分析方法
CN109766337B (zh) * 2018-11-28 2023-05-09 杭州云为科技有限公司 树形结构数据的存储方法、电子设备、存储介质及系统
US11270321B2 (en) 2019-08-27 2022-03-08 International Business Machines Corporation Association analysis on noisy transaction data
CN111783318B (zh) * 2019-10-15 2023-03-24 上海大学 一种基于三维模型的装配质量数据分析和可视化方法
CN111221650A (zh) * 2019-12-31 2020-06-02 青岛海尔科技有限公司 基于进程类型关联的系统资源回收方法及装置
CN111489165B (zh) * 2020-04-15 2022-08-12 支付宝(杭州)信息技术有限公司 目标对象的数据处理方法、装置和服务器
CN114676563B (zh) * 2022-03-14 2025-07-18 中国人民解放军93114部队 装备体系贡献度的评估方法及其装置
CN115953073A (zh) * 2023-01-06 2023-04-11 国能信控互联技术有限公司 基于火电生产指标管理的数据关联分析方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110289590A1 (en) * 2008-04-16 2011-11-24 Nokia Corporation Privacy management of data
CN102541934A (zh) * 2010-12-31 2012-07-04 北京安码科技有限公司 一种在电子商务平台上客户访问页面常见序列的提取方法和装置
CN104834751A (zh) * 2015-05-28 2015-08-12 成都艺辰德迅科技有限公司 基于物联网的数据分析方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6567936B1 (en) * 2000-02-08 2003-05-20 Microsoft Corporation Data clustering using error-tolerant frequent item sets
US6952693B2 (en) * 2001-02-23 2005-10-04 Ran Wolff Distributed mining of association rules
US7305378B2 (en) * 2004-07-16 2007-12-04 International Business Machines Corporation System and method for distributed privacy preserving data mining
WO2007043199A1 (ja) * 2005-10-07 2007-04-19 Hitachi, Ltd. 相関ルールを抽出する方法及びシステム
US8775230B2 (en) * 2008-11-03 2014-07-08 Oracle International Corporation Hybrid prediction model for a sales prospector
US9110969B2 (en) * 2012-07-25 2015-08-18 Sap Se Association acceleration for transaction databases
CN103914528B (zh) * 2014-03-28 2017-02-15 南京邮电大学 一种关联分析算法的并行化方法
US10467236B2 (en) * 2014-09-29 2019-11-05 International Business Machines Corporation Mining association rules in the map-reduce framework
CN107102999B (zh) 2016-02-22 2021-09-10 阿里巴巴集团控股有限公司 关联分析方法和装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110289590A1 (en) * 2008-04-16 2011-11-24 Nokia Corporation Privacy management of data
CN102541934A (zh) * 2010-12-31 2012-07-04 北京安码科技有限公司 一种在电子商务平台上客户访问页面常见序列的提取方法和装置
CN104834751A (zh) * 2015-05-28 2015-08-12 成都艺辰德迅科技有限公司 基于物联网的数据分析方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10956395B2 (en) 2016-02-22 2021-03-23 Alibaba Group Holding Limited Association analysis method and apparatus
CN117931930A (zh) * 2024-01-26 2024-04-26 西安电子科技大学 一种基于活动图的协作模式挖掘方法

Also Published As

Publication number Publication date
TW201741903A (zh) 2017-12-01
CN107102999B (zh) 2021-09-10
TWI730043B (zh) 2021-06-11
CN107102999A (zh) 2017-08-29
US10956395B2 (en) 2021-03-23
US20190102383A1 (en) 2019-04-04

Similar Documents

Publication Publication Date Title
TWI730043B (zh) 關聯分析方法和裝置
CN109766345B (zh) 元数据处理方法及装置、设备、可读存储介质
EP3602297B1 (en) Systems and methods for performing data processing operations using variable level parallelism
US11030196B2 (en) Method and apparatus for processing join query
US10866970B1 (en) Range query capacity allocation
WO2017096892A1 (zh) 索引构建方法、查询方法及对应装置、设备、计算机存储介质
US9330158B1 (en) Range query capacity allocation
CN105740264A (zh) 一种分布式xml数据库的排序方法及装置
Karim et al. An efficient distributed programming model for mining useful patterns in big datasets
US10599614B1 (en) Intersection-based dynamic blocking
CN107291734A (zh) 一种频繁项集的挖掘方法、装置及系统
CN114443659A (zh) 联接表的方法和装置
CN113868434A (zh) 图数据库的数据处理方法、设备和存储介质
CN106599122B (zh) 一种基于垂直分解的并行频繁闭序列挖掘方法
CN110362387B (zh) 分布式任务的处理方法、装置、系统和存储介质
CN110825786A (zh) 一种基于Spark的大数据关联规则挖掘方法
JP6393193B2 (ja) データ仮想化装置及び大規模データ処理プログラム
CN106547907B (zh) 一种频繁项集获取方法及装置
CN104021169B (zh) 一种基于SDD‑1算法的Hive连接查询方法
JP7311020B2 (ja) 制御方法、情報処理装置及び制御プログラム
JP6523823B2 (ja) 仮想データベースシステム管理装置、管理方法及び管理プログラム
CN105630789B (zh) 一种查询计划转化方法及装置
Yu et al. An efficient frequent patterns mining algorithm based on MapReduce framework
CN106776616B (zh) 合并对称实体组的方法及装置
CN114945902A (zh) 减少i/o开销的混洗归约任务

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17755737

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17755737

Country of ref document: EP

Kind code of ref document: A1