CN111444402A - Analysis method for community detection based on index construction and social factor control network - Google Patents

Abstract

The invention discloses an analysis method for carrying out community detection based on index construction and a social factor control network, which mainly comprises the following two steps of firstly constructing a generalized causal relationship network according to a social factor control theory, then constructing an index according to an FTV framework theory, inquiring and excavating a community structure of the network; establishing a causal relationship network by a social factor control theory, using the established network in the implementation step, then performing query work based on an FTV framework theory, and excavating a community structure in the network; and constructing a dictionary structure in the network.

Description

Analysis method of community detection based on index construction and social factor control network

technical field

The invention belongs to the field of network analysis, and is a method for querying based on index construction and analyzing social control theory. Firstly, construct a network of social factors and control relationships, and then use indexing technology to improve the speed and accuracy of query, and use this to analyze community relationships in the network.

Background technique

In recent years, with the popularization and development of social networks, more and more users have generated a large amount of data. How to analyze the possible community structure from the massive data has become a challenge in the field of network analysis. Over the years, the problems caused by massive data, due to the emergence of technologies such as hadoop, have gradually shifted from data storage to network construction and network analysis, and analyzing possible community structures from massive data is very useful in various fields. big effect. For example, analyzing various potential communities from social networks can uncover various fraudulent gangs, which is of extraordinary significance for purifying network security. This part mainly introduces the research status of community detection in network analysis.

A lot of research has been done on community detection for massive data. The composition of massive graph data often has two types. One is the super-large graph composition composed of massive data, including social networks, World Wide Web, e-commerce transaction networks and so on. In this type of network, such as social network, each node in the graph represents a person, and each edge represents the relationship between people. The query of this type of graph was first considered to be the NPC problem by KARP.R.M. It is proposed that the purpose of community detection is to find closely connected relationships and people in the network, from which possible events can be analyzed, such as the discovery of dangerous fraud gangs. KARP proposes to use the largest complete graph method to query and model community detection on massive data graphs. Another kind of graph network is composed of a large number of small-scale graphs, such as a compound network. In a network of many compounds, each atom represents a node, and each edge represents the force between atoms. This type of problem can be queried using the approximate matching method of subgraphs, but this problem is also an NPC problem. In 1976, J.Ullmannn first proposed a solvable method using the backtracking method. The patent of the present invention is aimed at the first type of graph query.

According to the method theory of KARP.R.M, although the massive data graph query problem has been transformed from the NPC problem into a solvable problem, the query speed is too slow, especially with the current surge of data, this method is more difficult to adapt to the current environment, and later V.Bonnici proposed A solution framework is proposed, the post-filtering verification framework (FTV framework), which greatly speeds up the query speed and improves the query accuracy.

This method firstly models the network according to the correlation of social factor control theory, constructs a causal network, and then queries and analyzes the constructed network according to the FTV framework theory, so as to achieve better matching, and reconstructs the community detection according to the index query technology. This method is of great significance both in terms of actual experimental effects and for subsequent extended research.

SUMMARY OF THE INVENTION

This method mainly mines the community structure in the massive data graph, and uses the FTV framework theory to speed up the query speed and improve the query accuracy, so that the community structure can be mined more quickly in the large-scale static graph data. This method has great application value in related scenarios such as fraud gang detection, same-interest group recommendation, and event outbreak warning.

This scheme is mainly divided into the following two steps. First, the causal relationship is extracted according to the dependency syntax, and then the generalized causal relationship network is constructed by using the extracted causal relationship.

This method is mainly divided into the following two steps. First, a generalized causal relationship network is constructed according to the social factor control theory, and then an index is constructed according to the FTV framework theory, and the query is carried out to mine the community structure of the network.

The social causal control theory builds a causal relationship network, and the implementation steps are as follows:

The first step is to build the network. Use the current pyspider framework to crawl the blog content in Weibo and the data of friend relationship list in Weibo, as the empirical data of this method.

Data processing is carried out on the specific content of the blog, using the word segmenter of Fudan University to divide the content of the user's blog post, and removing irrelevant particles, using FNLP keyword extraction to extract the input blog post data and extract keywords. Parts of speech are then divided, followed by semantic parsing and query abstraction (see Figure 1).

Firstly, an elementary SNA network is constructed according to the extracted data relationship, and then the text semantic content is extracted according to the semantics extracted in b, and the possible nodes and edges in the network are mined, so that the constructed network can be more dense and the network sparsity can be reduced. According to the semantic analysis model, in this method we use an example to illustrate how to perform semantic extraction. For example, in the blog post "I hate xxx", where "xxx" is a person's name, in the extraction of the blog post, "I" represents the ID of the author of the blog post in the blog, "hate" is a relational verb, and "xxx" is another object. From this, a hidden edge with the blog post author as a node can be extracted.

Use the constructed network, and then based on the FTV framework theory, perform query work to mine the community structure in the network (see Figure 2):

The second step is to construct the dictionary structure in the network. This dictionary structure is the basis for building query indexes in the future.

The dictionary structure is composed of paths in the graph, and the graph of paths not exceeding p length is used as an index, and this index is called "fingerprint".

Bitmap is performed by first splitting the fingerprint (query FQ and database FD) into a sequence of long integers (query LQ and database LD), then using bitmap operations (LQ∧) to test bitmap inclusion between each pair (LQ, LD) Include. LD=LQ).

We modify the structure of the nodes in the dictionary base by adding new fields that record edge labels, each fingerprint is constructed in exactly the same way, however, now each bit of the obtained fingerprint is appended to each corresponding bitmap's end. (see Figure 4).

And the parent node is constructed by performing a binary OR operation between the two child nodes. If the comparison between the query fingerprint and a given node of the tree returns false, the entire branch below the node can be discarded directly. Therefore, if the database is large enough, searching the dictionary base will require fewer comparisons than the database fingerprints.

To be able to apply a "fingerprint" to a set of bitmaps, distance measurement and averaging methods must be defined. The number of 1s in common between elements of the same cluster must be maximized in order to minimize 1s in binary OR. If b 1 and b 2 are two bitmaps, the distance between b 1 and b 2, denoted as d(b 1, b 2), is defined in the current proposal as follows:

According to the method in step f, a query dictionary base can be constructed, and the community structure of massive data can be directly queried according to the dictionary base, so as to speed up the query and improve the query accuracy.

beneficial effect

For the existing massive data graph mining methods, the social factor control theory is mainly used to build a causal relationship network, but only the social relationship correlation theory is used, and the complete causal syntactic relationship is not used, but the accuracy and speed of the query results are not bad. Satisfactory, this method mainly has the following gains:

First, when constructing a social network, not only the relevant information in the data grouping is used, but also semantic analysis is used to construct the network, which reduces the problems related to network sparsity.

Secondly, using the FTV theoretical framework and improving it at the same time, using the method of bitmap compression, it can effectively compress the volume of index construction and speed up the query efficiency.

Finally, the method has great compatibility and can be applied not only to graph query related fields, but also to community detection fields such as citation networks and communication networks.

Description of drawings

Figure 1 is the query abstraction diagram;

Figure 2 is the index verification query structure;

Figure 3 is the bitmap dictionary library calculation;

Figure 4 is how to build a relationship network based on data;

Figure 5 is an example of using text semantic relations to reduce network sparsity;

Figure 6 shows the filtering theoretical framework and index construction for querying after the network is established.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

The method of emotional cause mining based on dependency syntax and generalized causal network proposed in this paper is mainly used to discover the causal relationship of text and find out the law of its operation. When digging for emotional reasons, follow the steps described below.

The analysis method of community detection based on index construction and social factor control network proposed in this paper is mainly used to quickly and accurately detect communities with a certain structure in the scale of massive data, so as to make corresponding relationship prediction. The specific process of establishing the network and the process of constructing the index query are performed according to the steps described below.

Step 1: First, we first obtain the user blog post data and user data of Weibo, and we mainly use pyspider to crawl.

Step 2: Clean the crawled data, extract the user's account data, and organize the blog post data. This step is to remove a lot of invalid information, such as empty accounts and blog posts.

Step 3: Organize the blog post data, perform semantic analysis on the blog post data, and extract the keywords in the blog post.

Step 4: Analyze the semantics of keywords and remove modal particles and symbols.

Step 5: After filtering out invalid keywords and symbols, use the tokenizer to distinguish the relationship between account names, related account names, and names.

Step 6: Construct a cause-control relationship network by extracting effective keywords and relationship tables.

Step 7: Build a relationship network based on the user relationship network as the main body and blog post semantic analysis as the supplement.

Step 8: Build a dictionary library based on the FTV filtering framework theory. First, extract the fingerprint library whose length of user node is not longer than p (index path value) for matching, and then the matching result is the dictionary library.

The ninth step: compress the dictionary library for easy query, and use the bitmap compression algorithm.

Step 10: Query the community structure in the network according to the dictionary library.

Among them, the implementation process of steps 1, 2, 3, 4, and 5 corresponds to step 1 in the technical solution. After step 1, a complex network with associated semantics as a carrier can be constructed. The data processing process can crawl the data according to the framework of Figure 3, and then perform data processing, and the semantic analysis framework associated with the semantic network can be carried out through Figure 4. The implementation process of the sixth, seventh, eighth, and ninth steps corresponds to the second step in the technical solution. Through this step, the index dictionary library after the abstract query can be constructed. The data structure of the dictionary library is shown in Figure 2. Finally, in the tenth step, a candidate set, that is, a community structure of a specific structure, can be obtained from the index according to an existing query. The pattern of the obtained candidate set is shown in FIG. 6 .

Claims (4)

1. The method is characterized by mainly comprising the following two steps of firstly constructing a generalized causal relationship network according to a social factor control theory, then constructing an index according to an FTV framework theory, inquiring and excavating a community structure of the network;

a causal relationship network is constructed by the social factor control theory, and the implementation steps are as follows:

step one, constructing a network;

using the constructed network, then carrying out query work based on FTV framework theory, and mining community structures in the network;

step two, constructing a dictionary structure in the network;

bitmap inclusion is performed by first dividing the fingerprints (query FQ and database FD) into long integer sequences (query L Q and database L D), and then testing the bitmap inclusion between each pair (L Q, L D) using bitmap operations (L Q ^).

2. The index building and social cause control network based analysis method for community detection according to claim 1, wherein the structure of the nodes in the dictionary repository is modified by adding new fields recording edge tags, each fingerprint is constructed in exactly the same way, however, each bit of the fingerprint obtained now is appended to the end of each corresponding bitmap.

3. The index building and social cause control network based analysis method for community detection as claimed in claim 1, wherein the parent node is constructed by performing a binary OR operation between two child nodes.

4. The index building and social cause control network based community detection analysis method of claim 1, wherein a 'fingerprint' is applied to a bitmap set, defining a distance measurement and average calculation method.

Images