CN105159922A - Label propagation algorithm-based posting data-oriented parallelized community discovery method - Google Patents

Abstract

The present invention relates to a parallelized community discovery method based on tag propagation algorithm for mailing data, including: step S1: preprocessing mailing data, and structuring it into text data according to a set format; step S2: mailing between nodes in the integrated text data Exchange information, standardize the weights of directed edges between nodes, and finally construct a delivery directed and weighted relationship network model in the form of an adjacency table; Step S3: use the improved label propagation algorithm, and use the MapReduce framework to parallelize mining in the delivery network Community structure; step S4: analyze the community structure obtained in step S3, and discover the communities in the delivery network. Compared with the prior art, the present invention improves the expansibility and operating efficiency of the traditional label propagation algorithm, and finally realizes accurate and efficient mining of communities in the delivery network.

Description

A Parallelized Community Discovery Method for Mailing Data Based on Label Propagation Algorithm

technical field

The invention relates to a method for constructing a delivery network based on delivery data, in particular to a method for discovering a parallel community based on label propagation algorithm for delivery data.

Background technique

The study of social network analysis originated in the early 1920s, focusing on the study of relationships between social entities, such as: communication within group members, trade between countries, or economic transactions between companies. With the rapid development of information, the complexity of social networks is increasing. Both network managers and network researchers hope to have a clear understanding of the structure of social networks. Community mining is of great significance to understanding the structure of social networks. The discovery of network community structure has very important theoretical significance and practical value for network topology analysis, network functional analysis and network behavior prediction. It has a wide range of applications in the fields of social networks and biological networks. It has been widely used in many fields such as social network and terrorist organization identification.

First of all, clustering-based community discovery algorithms often only consider the attribute information of nodes, resulting in ignoring other useful information (such as edge weights), and it requires a pre-specified input parameter (the number of communities in the network), resulting in The accuracy of community division is not high. Secondly, considering that the label-based transfer algorithm does not require any input parameters, and has linear time complexity, fast convergence speed, and high mining accuracy, it is suitable for community mining in large-scale networks. Finally, due to the rapid development of computer technology and Internet technology, people's ability to obtain data has been continuously enhanced, and the scale of the network to be analyzed has also increased from tens to hundreds of nodes to a scale of millions to tens of millions. Distributed algorithms are no longer suitable for community discovery in larger networks. The MapReduce computing framework in the Hadoop platform is very suitable for processing large-scale data. Therefore, it is a feasible solution to introduce the MapReduce computing framework into the community mining algorithm and use distributed computing to solve the large-scale delivery network community discovery.

Contents of the invention

The purpose of the present invention is to provide a parallel community discovery method based on the label propagation algorithm for delivery data in order to overcome the defects in the above-mentioned prior art. On the basis of building a delivery relationship network model, using the MapReduce distributed computing framework, Improve the scalability and operating efficiency of the traditional label propagation algorithm, and finally realize the accurate and efficient mining of communities in the delivery network.

The purpose of the present invention can be achieved through the following technical solutions:

A parallel community discovery method for delivery data based on label propagation algorithm, including:

Step S1: Preprocess the delivery data, and structure it into text data according to the set format;

Step S2: Synthesize the posting and posting information between nodes in the text data, standardize the weights of directed edges between nodes, and finally construct a posting directed and entitled relationship network model in the form of an adjacency list;

Step S3: Use the improved label propagation algorithm to mine the community structure in the delivery network in parallel using the MapReduce framework;

Step S4: Analyze the community structure obtained in step S3, and discover the communities in the delivery network.

The text data is uploaded to the HDFS (Hadoop Distributed File System) of the Hadoop platform for storage and processing.

The step S1 is specifically: for each piece of delivery data, extract the sender's name, sender's phone number, recipient's name, recipient's phone number, the sender's name, sender's phone number , recipient's name, and recipient's phone number correspond to four columns of information for each row of text data.

The step S2 is specifically:

201: For each sender, obtain the adjacency list of the frequency of logistics exchanges between the sender and other recipients, and standardize the adjacency list;

202: For any sender and receiver who have logistics exchanges, count the number A of the same recipients corresponding to them when they are senders respectively, and this number A is recorded as the number of shared sending neighbors;

203: For any sender and receiver who have logistics exchanges, count the number B corresponding to the same sender when they are the recipients respectively, and record the number B as the number of shared receiving neighbors;

204: For any sender and recipient that have a logistics relationship, obtain the sum of the number of shared sending neighbors and the number of shared receiving neighbors between them, and the sum is used as the shared value between the sender and the recipient The number of neighbors, and normalize the number of shared neighbors;

205: Add the weight of the adjacency list obtained in step 201 and the number of shared neighbors obtained in step 204 according to the ratio of α: 1-α, and then obtain the effective value considering both the frequency of sending mail and the number of common sending neighbors and common receiving neighbors. To the edge weight, and update the adjacency list, where, 0<α<1.

The improved label propagation algorithm adopts multiple iterations, and an iteration process is specifically:

301: At the end of the adjacency list obtained in step S2, add the unique ID of the corresponding sender node as the sender node label Label, and complete the initialization label;

302: Output multiple key-value pairs in the form of <key, value> according to the adjacency list with node labels, which are divided into sender key-value pairs and receiver key-value pairs;

303: Obtain the key-value pair with the same key value, traverse each value, first obtain the value of the sender's key-value pair to represent the value of the adjacency list of the key value, and store it in the variable adjacent; secondly, for the recipient For the value of the key-value pair, count the sum of the weight values under different Labels, and update the node label NewLabel of the key value according to the proportion of different Labels;

304: Add NewLabel to the end of adjacent, output a new key-value pair in the form of <key,value>, and update the label of the adjacency list. The community structure in the delivery network corresponds to the adjacency list containing the label.

The iteration termination condition of the improved label propagation algorithm includes: the labels of the nodes whose labels are greater than the set percentage in the previous two iterations do not change or reach the set number of iterations.

The set percentage is 90%.

The set iteration times are 20-30 times.

The step S4 specifically includes: according to the adjacency list obtained in the step S3, the nodes with the same label are regarded as the same community, so as to discover the community in the delivery network.

Compared with the prior art, the present invention has the following advantages:

1) The existing technology is mainly based on a stand-alone algorithm to mine communities, which is not suitable for community mining in large-scale networks. The present invention builds a delivery network based on delivery data, and uses a parallel label propagation algorithm in the delivery network to accurately and efficiently Mining communities in delivery networks is especially suitable for mining large-scale networks. Compared with traditional stand-alone algorithms for mining communities, the method provided by the present invention has obvious advantages.

2) In the calculation of the weight of the delivery network side, three indicators are considered: 1. The frequency of logistics exchanges between the two parties; 2. The number of corresponding recipients that exist when both parties are senders; 3. Count the number of the same sender when both senders and senders are recipients. Finally, the present invention integrates these three indicators to calculate the weights of all edges in the network, thereby providing mining precision and accuracy.

3) The method of the present invention does not require any input parameters, and has linear time complexity and fast convergence speed, and is suitable for community mining in large-scale networks.

4) Combining with the MapReduce distributed computing framework, the text data reflecting the delivery data is uploaded to the HDFS of the Hadoop cluster for storage and processing, so as to improve the scalability and time efficiency of the algorithm.

Description of drawings

Fig. 1 is the overall flowchart of the inventive method;

Fig. 2 is a flow chart of building a delivery relationship network model based on delivery data;

Figure 3 is a flow chart of mining communities in parallel using the improved label propagation algorithm.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is carried out on the premise of the technical solution of the present invention, and detailed implementation and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.

As shown in Figure 1, a parallel community discovery method based on the label propagation algorithm for delivery data is divided into the stage of building the delivery relationship network model and the mining stage, as follows:

Step S1: Preprocess the delivery data, structure it into text data according to the set format, and upload the text data to the HDFS of the Hadoop cluster for storage and processing. Specifically:

For each piece of delivery data, extract the sender's name, sender's phone number, recipient's name, recipient's phone number, sender's name, sender's phone number, recipient's name, recipient's Phone numbers correspond to four columns of information for each row of text data.

Step S2: Synthesize the posting information between nodes in the text data, standardize the weights of directed edges between nodes, and finally construct a posting directed and entitled relationship network model in the form of an adjacency list, and upload it to HDFS. As shown in Figure 2, specifically:

201: For each sender, obtain an adjacency list of the frequency of logistics exchanges between the sender and other recipients, and standardize the adjacency list. The specific instructions are as follows:

1) First, based on the MapReduce computing framework, read the standardized text data stored in HDFS in step S1 line by line in the Map stage, and use the combination of their name and phone number for the sender and recipient as their unique Mark the ID and output a key-value pair in the form of <key,value>, where key is the ID of the sender and value is the ID of the recipient.

2) When the same key value is obtained in the Reduce stage, that is, the same sender, count the frequency of logistics between the sender and different recipients. Finally, for each sender, an adjacency list that only considers the frequency of logistics transactions between the sender and other recipients is obtained.

3) Secondly, according to the adjacency list of each sender, when the sender sends express delivery frequency greater than the set frequency (500 times in this embodiment based on experience), it can be judged that the sender is a logistics transfer station or is For Taobao sellers, etc., it is necessary to delete the adjacency list of the sender, and delete the sender node from the adjacency lists of other senders at the same time.

4) Finally, according to the newly generated adjacency lists of all senders, count the sender and receiver with the largest frequency of logistics exchanges, set the maximum number of exchanges as Max, and use Max to standardize the adjacency lists of all senders: Suppose Adjacency list of a certain sender [S\tR ₁ :C ₁ \tR ₂ :C ₂ ...\tR _k :C _k ], where \t is the delimiter, S is the abbreviation of Sender, which means the sender, R is the abbreviation of Receiver, indicating the recipient, C is the abbreviation of Count, the subscript k is the sequence number of the recipient and the corresponding times, indicating the number of times, and the recipients who will have logistics with it (R ₁ , R ₂ and R _k etc.), divide the number of exchanges (C ₁ , C ₂ and C _k , etc.) by Max, and finally get the normalized adjacency list of the sender, that is, [S\tR ₁ :W ₁ \tR ₂ :W ₂ ... \tR _k :W _k ], where W _k ＝C _k /Max.

202: Obtain the number of shared sending neighbors: For any sender and receiver with logistics contacts, count the number A of the same recipients corresponding to them when they are senders, and this number A is recorded as the number of shared sending neighbors . The specific instructions are as follows:

1) First, under the MapReduce computing framework, read the adjacency list [S\tR ₁ :W ₁ \tR ₂ :W ₂ ...\tR _k :W _k ] of each sender in 1) in the Map stage , output multiple key-value pairs in the form of <key,value>: <S,+R ₁ \tR ₂ ...\tR _k >(+ is used to distinguish the following <key,value> key-value pairs) and <R ₁ ,S\tR ₂ ...\tR _k >, <R ₂ ,S\tR ₁ ...\tR _k >, ..., <R _k ,S\tR ₁ ...\tR _k-1 >, etc. .

2) Obtain <key, value> key-value pairs with the same key value in the Reduce phase, traverse each value, first obtain the value with "+", divide it into an array with "\t", and the elements in the array are When the current key user is the recipient of the sender, store these neighbor users in a HashSet data structure set_key. Secondly, use "\t" to divide each remaining value without "+" into an array and analyze it, and store the result in a map of a HashMap data structure (the key of the map is the array after being divided by "\t" The first element of the array, value is a HashSet structure used to store other elements of the array). Finally, the map is traversed, and the value and set_key of each element in the map are intersected. The size of the intersection is the number of shared sending neighbors when the key value of this element and the current Reduce key are respectively used as senders.

203: Obtain the number of shared receiving neighbors: For any sender and receiver with logistics contacts, count the number B corresponding to the same sender when they are respectively recipients, and this number B is recorded as the number of shared receiving neighbors . The specific instructions are as follows:

First, according to the adjacency list [S\tR ₁ :W ₁ \tR ₂ :W ₂ ...\tR _k :W _k ] of each sender in step 201, establish a link to the sender for each recipient Inverted index [R ₁ \tS _l \tS _p ...\tS _n ], the subscripts l, p, n indicate the serial number of the sender after inversion; secondly, similar to the solving process in step 202, any For two senders and recipients who have logistics exchanges, count the number of shared receiving neighbors when they are respectively recipients.

204: For any sender and recipient that have a logistics relationship, obtain the sum of the number of shared sending neighbors and the number of shared receiving neighbors between them, and the sum is used as the shared value between the sender and the recipient The number of neighbors, and the maximum value of the number of shared neighbors in the entire network is obtained to standardize the number of shared neighbors of each sender node and receiver node that have logistics exchanges.

205: Add the weight of the adjacency list obtained in step 201 and the number of shared neighbors obtained in step 204 according to the ratio of α: 1-α, and then obtain the effective value considering both the frequency of sending mail and the number of common sending neighbors and common receiving neighbors. The edge weight, that is, the proportion of the weight of the edge in the adjacency list is α, and the proportion of the number of co-sending neighbors and the number of co-receiving neighbors is 1-α, where 0<α<1, using the new Update the adjacency list to the edge weight, and upload the newly generated adjacency list to HDFS.

The data processing in the stage of building the delivery relationship network model is completed above, as shown in FIG. 2 . Next, the data processing in the mining stage is carried out, as shown in Figure 3.

Step S3: Use the improved label propagation algorithm to mine the community structure in the delivery network in parallel using the MapReduce framework.

The improved label propagation algorithm adopts multiple iterations, and the specific iteration process is as follows:

301: Add the unique ID of the corresponding sender node at the end of the adjacency list obtained in step S2, as the sender node label Label, complete the initialization label, and the corresponding adjacency list with node label is expressed as [S\tR ₁ : W ₁ \tR ₂ :W ₂ ...\tR _k :W _k \tLabel].

302: In the Map stage, output multiple key-value pairs in the form of <key,value> according to the adjacency list with node labels, and divide them into sender key-value pairs <S,+R ₁ :W ₁ \tR ₂ :W ₂ .. .\tR _k :W _k >(+ to distinguish <key, value> key-value pairs generated later) and recipient key-value pairs <R ₁ ,Label\tW ₁ >, <R ₂ ,Label\tW ₂ >, ..., <R _k ,+Label\tW _k >.

303: In the Reduce phase, obtain the <key, value> key-value pair with the same key value, traverse each value, and first obtain the value of the sender's key-value pair (that is, the value with "+") to represent the key value The value of the adjacency list of the recipient is stored in the variable adjacent. Secondly, for the value of the recipient key-value pair (the value without "+"), the sum of the weight values under different Labels is calculated and updated according to the proportion of different Labels The node label NewLabel of the key value, where the larger the proportion of Label, the more likely the label of the current key node is updated to this Label.

304: Add the newly generated label NewLabel of the key node to the end of adjacent, and output a new key-value pair in the form of <key,value>, that is, <S,R ₁ :W ₁ \tR ₂ :W ₂ ...\tR _k :W _k \tNewLabel>, and update the label of the adjacency list, the community structure in the delivery network corresponds to the adjacency list containing the label.

The iteration termination conditions of the improved label propagation algorithm include the following two types: 1. The labels of each node are basically stable, that is, the labels of nodes with a setting percentage greater than the previous two iterations do not change. Among them, the setting percentage in this embodiment is 90 %, 2. Reach the set number of iterations, generally 20-30 times, and 25 times in this embodiment.

Step S4: Analyze the community structure obtained in step S3, find the community in the delivery network, and save the result in HDFS. Specifically:

According to the adjacency list obtained in step S3, the nodes with the same label are regarded as the same community, so as to discover the community in the delivery network.

To sum up, the stage of building the delivery relationship network model is the data preprocessing process, and the mining stage is the iterative process. The iterative process is based on the single-machine label propagation algorithm to realize the distributed form of the algorithm. At the same time, due to the particularity of the logistics delivery data, this patent is calculating In terms of the weight of the network side, three indicators are considered: 1. The frequency of logistics exchanges between the two parties; The number of recipients corresponds to the number of identical senders. Finally, the present invention integrates these three indicators to calculate the weights of all edges in the network, thereby realizing accurate and efficient mining of communities in the delivery network.

Claims (9)

1., based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, it is characterized in that, comprising:

Step S1: pre-service consignment data, turns to text data according to setting format structure;

Step S2: consignment contact information between comprehensive text data interior joint, the weights of directed edge between standardization node, are finally built into the oriented relational network model of having the right of consignment with adjacency list form;

Step S3: utilize the label propagation algorithm improved, uses the community structure in MapReduce framework parallelization excavation consignment network;

Step S4: the analyzing step S3 community structure obtained, finds corporations in consignment network.

2. according to claim 1ly it is characterized in that based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, described text data is uploaded in the HDFS of Hadoop cluster and stores and process.

3. according to claim 1 based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, it is characterized in that, described step S1 is specially: for every bar consignment data, extract sender's name, sender telephone number, addressee's name, addressee's telephone number respectively, described sender's name, sender telephone number, addressee's name, addressee's telephone number correspond to four column informations of notebook data of often composing a piece of writing.

4. according to claim 1 based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, it is characterized in that, described step S2 is specially:

201: for each sender, obtain logistics between this sender and other addressees and to come and go the adjacency list of frequency, and standardization is carried out to adjacency list;

202: next sender and addressee are flowed to any existence, add up them and be designated as shared transmission neighbours number respectively as corresponding during sender the quantity A that there is identical addressee, this quantity A;

203: next sender and addressee are flowed to any existence, add up them and be designated as shared reception neighbours number respectively as corresponding during addressee the quantity B that there is identical sender, this quantity B;

204: next sender and addressee are flowed to any existence, obtain shared transmission neighbours number between them with share receive neighbours' number and value, and should be worth as the shared neighbours' number between this sender and addressee, and standardization is carried out to shared neighbours' number;

205: the shared neighbours' number obtained in the weights of adjacency list step 201 obtained and step 204 obtains after being added in the ratio of α: 1-α to be considered post part frequency and jointly send neighbours' number and the common directed edge weights receiving neighbours' number simultaneously, and upgrade adjacency list, wherein, 0 < α < 1.

5. according to claim 1ly it is characterized in that based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, the label propagation algorithm of described improvement adopts the mode of successive ignition, and one time iterative process is specially:

301: the unique sign ID adding corresponding sender's node in the ending of the adjacency list of step S2 acquisition, as sender's node label Label, completes init Tag;

302: the adjacency list according to band node label exports multiple <key, value> form key-value pair, is divided into sender's key-value pair and addressee's key-value pair;

303: the key-value pair obtaining identical key value, travel through each value, first the value obtaining sender key-value pair is used for the value of the adjacency list representing this key value, and be stored in variable adjacent, secondly, for the value of addressee's key-value pair, weighted value sum under statistics Different L abel, and the node label NewLabel of this key value is upgraded according to the proportion of Different L abel;

304: NewLabel is added to adjacent ending place, export a new <key, value> form key-value pair, and upgrade the label of adjacency list, the community structure in consignment network is corresponding with the adjacency list containing label.

6. according to claim 5 based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, it is characterized in that, the stopping criterion for iteration of the label propagation algorithm of described improvement comprises: the node label that twice, front and back iterative process is greater than setting number percent does not change or reaches the iterations of setting.

7. according to claim 6ly it is characterized in that based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, described setting number percent is 90%.

8. according to claim 6ly it is characterized in that based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, the iterations of described setting is 20 ~ 30 times.

9. according to claim 5 based on the parallelization Combo discovering method of label propagation algorithm towards consignment data, it is characterized in that, described step S4 is specially: the adjacency list obtained according to step S3, is considered as same corporations by the node of same label.