CN112131444A

CN112131444A - Low-space-overhead large-scale triangle counting method and system in graph

Info

Publication number: CN112131444A
Application number: CN202010920564.2A
Authority: CN
Inventors: 肖侬; 牟者斌; 卢宇彤; 陈志广
Original assignee: National Sun Yat Sen University
Current assignee: Sun Yat Sen University; National Sun Yat Sen University
Priority date: 2020-09-04
Filing date: 2020-09-04
Publication date: 2020-12-25

Abstract

The invention discloses a method and a system for counting triangles in a large-scale graph with low space overhead, wherein the method comprises the following steps: acquiring a data set and constructing a directed graph according to the data set to obtain graph data; traversing all vertexes in the graph data and putting the vertexes into a preset bloom sieve according to a preset rule; obtaining two vertexes corresponding to the edge according to the edge in the graph data; and judging that the two vertexes exist in the bloom screen, and calculating the intersection of the neighbor vertex sets of the two vertexes to obtain the number of the triangles corresponding to the edge. The system comprises: the system comprises a graph data module, an input module, a vertex module and an intersection module. By using the method and the device, the number of the triangles can be rapidly calculated from large-scale data under the condition of ensuring low space overhead. The method and the system for counting the triangles in the large-scale graph with low space overhead can be widely applied to the field of large-scale data processing.

Description

Low-space-overhead large-scale triangle counting method and system in graph

Technical Field

The invention relates to the field of large-scale data processing, in particular to a method and a system for counting triangles in a large-scale graph with low space overhead.

Background

With the increasing use of the internet in life, the increasing speed and the total amount of various data are increased day by day, and even the increasing and the increasing speed of the data are difficult to describe by the words of ' exponential growth ', mass ' and the like. However, the speed of analyzing and processing data cannot keep up with the increasing scale and speed of data, and the information and value hidden in the large-scale data are mined and utilized by people. In recent years, therefore, the research direction and trend of internet is towards how to analyze and process large-scale data. As one of the common and important data structures, graph data can better express the relationship between individuals and groups, and thus is also attracting more and more attention of researchers and enterprises.

In the graph data analysis, triangles can represent the incidence relation, and many solutions have been provided for the triangle counting problem. One is to look for the number of non-repeating edges that exist between all its neighboring nodes by traversing all vertices in the graph, the sum of which is three times the number of triangles of the graph. The other method is that all edges in the graph are traversed, intersection is taken for the neighbor vertex sets of two vertexes of the graph, and the sum of the number of the intersection elements of each edge is three times of the number of the triangles of the graph. How to quickly calculate all triangles from large-scale data is the current application bottleneck. Although for large-scale data sets, there are many mature methods, such as a distributed method based on a MapReduce computing platform, and the like. However, for various large-scale sparse graphs in the real world, such as social networks, traffic networks, and the like, problems of excessive space overhead (for example, a single-machine method cannot process a large-scale graph) or excessive time overhead (for example, random access to a memory or frequent communication in a distributed method) often exist.

Disclosure of Invention

In order to solve the above technical problems, an object of the present invention is to provide a method and a system for counting triangles in a large-scale graph with low space overhead, which can quickly calculate the number of triangles from large-scale data under the condition of ensuring low space overhead.

The first technical scheme adopted by the invention is as follows: a method for counting triangles in a large-scale graph with low space overhead comprises the following steps:

acquiring a data set and constructing a directed graph according to the data set to obtain graph data;

traversing all vertexes in the graph data and putting the vertexes into a preset bloom sieve according to a preset rule;

obtaining two vertexes corresponding to the edge according to the edge in the graph data;

and judging that the two vertexes exist in the bloom screen, and calculating the intersection of the neighbor vertex sets of the two vertexes to obtain the number of the triangles corresponding to the edge.

Further, still include:

traversing all edges in the graph data and obtaining the number of triangles which are formed by the corresponding participation of all the edges;

and summing the numbers of the corresponding triangles participating in the formation of the triangles on all the edges to obtain the total number of the triangles of the graph data.

Further, the step of obtaining a data set and constructing a directed graph according to the data set to obtain graph data specifically includes:

acquiring a data set and reorganizing the data set into an ordered edge set array;

constructing an index array with the size of the vertex number and stored index information according to the edge set array;

and obtaining the graph data according to the edge set array and the index array.

Further, the step of traversing all vertices in the graph data and placing the vertices into the bloom filter according to a preset rule specifically includes:

sequentially traversing all vertexes in the graph and enabling all different two vertexes in the adjacent vertexes of the vertexes to form a vertex pair;

the vertex pairs are added to the bloom screen in a front and back fashion with the vertices numbered small.

Further, the preset bloom filter is constructed by the number of vertexes in the graph data, and the expression is as follows:

in the above formula, k is the number of hash functions, m is the bloom filter length, n is the number of inserted elements, and p is the false alarm rate.

Further, the step of obtaining two vertices corresponding to the edge according to the edge in the graph data further includes sorting the two vertices corresponding to the edge in order of vertex numbers from small to large.

Further, the calculating an intersection of the sets of neighboring vertices of the two vertices specifically includes:

judging that the size ratio of the neighbor vertex sets of the two vertexes is smaller than a preset value, and obtaining the intersection of the neighbor vertex sets of the two vertexes by adopting a method of simultaneously traversing the neighbor vertex set comparison elements of the two vertexes to obtain the intersection;

and judging that the size ratio of the neighbor vertex sets of the two vertexes is larger than a preset value, firstly traversing the set with a smaller element number, sequentially carrying out binary search on the set with a larger element number, and calculating to obtain the intersection of the neighbor vertex sets of the two vertexes.

The second technical scheme adopted by the invention is as follows: a low space overhead large scale in-graph triangle count system, comprising the following modules:

the graph data module is used for acquiring a data set and constructing a directed graph according to the data set to obtain graph data;

the putting module is used for traversing all vertexes in the graph data and putting the vertexes into a preset bloom sieve according to a preset rule;

the vertex module is used for obtaining two vertexes corresponding to the edge according to the edge in the graph data;

and the intersection module is used for judging that the two vertexes exist in the bloom screen, calculating the intersection of the neighbor vertex sets of the two vertexes and obtaining the number of the triangles corresponding to the edge.

Further, still include:

the traversal module is used for traversing all the edges in the graph data and obtaining the number of the corresponding participation of all the edges in forming the triangle;

and the adding module is used for adding the numbers of the triangles which are correspondingly participated in forming the triangle by all the edges to obtain the total number of the triangles of the graph data.

The method and the system have the beneficial effects that: the method has the advantages that the form of the directed graph is adopted to store graph data, a large amount of space can be saved, in addition, whether the sides form triangles or not is detected through the bloom filter, and a large amount of unnecessary intersection fetching operation is reduced, so that the speed of traversing the sides is improved, the complexity of average time is reduced, and the calculation efficiency is improved.

Drawings

FIG. 1 is a flow chart of the steps of a low space overhead large scale graph triangle counting method of the present invention;

FIG. 2 is a block diagram of a low space overhead large scale graphical triangle counting system of the present invention;

FIG. 3 is a graph data diagram of an embodiment of the present invention;

FIG. 4 is a flow chart of the steps for constructing a bloom screen in accordance with an embodiment of the present invention;

FIG. 5 is a flowchart of the steps for traversing edges to count triangles in accordance with an embodiment of the present invention.

Detailed Description

The invention is described in further detail below with reference to the figures and the specific embodiments. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

As shown in FIG. 1, the present invention provides a large-scale graph triangle counting method with low space overhead, which comprises the following steps:

s1, acquiring a data set and constructing a directed graph according to the data set to obtain graph data;

specifically, an ordered edge set is constructed according to an existing data set, edges are represented in an ordered vertex pair mode, and the edge set is represented in an ordered edge array mode; and then constructing an index array with the size of the vertex number according to the edge set, and storing the index of the vertex on the edge set array on the corresponding position of the vertex number and the array to obtain the edge set array and a neighbor vertex set index array of the vertex, thereby obtaining the graph data of the directed graph.

S2, traversing all vertexes in the graph data and putting the vertexes into a preset bloom sieve according to a preset rule;

specifically, referring to fig. 4, a bloom sieve is constructed according to the number of vertices in the graph, then all vertices in the graph are traversed sequentially, and a vertex pair formed by all different two vertices in the neighbor nodes of the vertex is represented in a form of a front vertex with a smaller vertex number and a rear vertex with a larger vertex number, and is added into the bloom sieve. The above operations are repeated until all the vertices in the graph are traversed, and the graph data is as shown in fig. 3.

In addition, the method for calculating the number of triangles adopts the directed graph to store the graph, and compared with the method for directly storing the undirected graph, each side e1(v1, v2) and v1< v2 store more sides e2(v2, v1), so that a large amount of space can be saved.

S3, obtaining two vertexes corresponding to the edge according to the edge in the graph data;

s4, judging that the two vertexes exist in the bloom screen, and calculating the intersection of the neighbor vertex sets of the two vertexes to obtain the number of the triangles corresponding to the edge.

Further as a preferred embodiment of the method, the method further comprises:

Specifically, referring to FIG. 5, all the edges in the graph are traversed sequentially, let the currently visited edge be e, and let its vertices be v1, v2, and v1< v 2. Firstly, judging whether the vertex pair (v1, v2) is in the bloom screen, if not, proving that one vertex v does not exist, and v, v1 and two sides of v, v2 exist, namely the side does not participate in forming the triangle. On the contrary, if the vertex pair exists in the bloom filter, the edge may form at least one triangle, and then we perform an intersection operation on the edge, and calculate the size of the intersection of the sets of the vertices adjacent to the two vertices of the edge, that is, the number of triangles formed by the edge. So far, the above steps are regarded as the traversal of the edge e is completed. And repeating the steps, sequentially traversing all the edges, and summing the number of the triangles formed by all the edges, namely the number of the triangles in the graph.

Further, as a preferred embodiment of the method, the step of obtaining the data set and constructing the directed graph according to the data set to obtain the graph data specifically includes:

Specifically, if an edge e is represented in the form of a vertex pair, the two vertices of the edge e are v1, v2, and v1< v2, the edge e is represented in the form of a vertex pair (v1, v 2).

The edge sets are represented in the form of an array of ordered pairs of vertices, i.e., stored in the array sequentially by the relative size of the pairs. As long as it is in an ordered manner. If the edges e1(v1, v2), e2(v1, v3), e3(v2, v3) and v1< v2 < v3 are in ascending order, then e1, e2 and e3 are sequentially stored in the edge set array. If in descending order, the order is reversed.

The neighbor vertex set array of the vertex is mapped to the corresponding position of the edge set by constructing the index array according to the size sequence of the vertex number, namely the starting position and the ending position of each vertex adjacent edge set on the whole graph edge set. If there are edges e1(v1, v2), e2(v1, v3), e3(v2, v3) and v1< v2 < v3, (index (e1), index (e2)) is stored in the corresponding position of the vertex set array of the vertex neighbor of v1, and index (e1) and index (e2) refer to the index positions of the edge sets of e1 and e2, respectively.

In addition, the invention adopts a method of firstly traversing the array of the stored vertex neighbor set indexes to construct the bloom screen, and then traversing the edge set to take the intersection of the two vertex neighbor vertex sets to calculate the number of the triangles. The memory is ensured to be accessed in sequence as much as possible when the memory is accessed, and the time overhead brought by random access of the memory is greatly reduced. Compared with a method with a large amount of random access memories, the method has the advantage that the improvement is more obvious when the method is used on a large-scale data set.

Further, as a preferred embodiment of the method, the step of traversing all vertices in the graph data and placing the vertices into the bloom filter according to a preset rule specifically includes:

As a further preferred embodiment of the method, the predetermined bloom filter is constructed by the number of vertices in graph data, and the expression is as follows:

Specifically, for any vertex v in the figure, let v1, v2 belong to the neighbor vertex set, and v1< v2, and add all eligible vertex pairs (v1, v2) into the bloom screen without repetition. This operation is considered to be completed through point v. The above operation is repeated until all points in the graph have been traversed.

In addition, the bloom sieve constructed by the invention can detect whether the possibility of forming a triangle exists or not according to any given edge (namely two vertex pairs of the edge). If not, the intersection operation is not carried out on the neighbor vertex sets of the two vertexes of the edge, so that a large amount of unnecessary intersection operation can be reduced. Therefore, the speed of traversing edges is improved, the complexity of average time is reduced, and the calculation efficiency is improved.

Further as a preferred embodiment of the method, the step of obtaining two vertices corresponding to the edge according to the edge in the graph data further includes sorting the two vertices corresponding to the edge in order of vertex numbers from small to large.

As a further preferred embodiment of the method, the calculating an intersection of the sets of neighboring vertices of the two vertices specifically includes:

judging that the size ratio of the neighbor vertex sets of the two vertexes is larger than a preset value, firstly traversing the set with smaller element number, sequentially carrying out binary search on the set with larger element number, and calculating to obtain the intersection of the neighbor vertex sets of the two vertexes

Specifically, the invention adopts a method for dynamically allocating and taking intersection according to the sizes of two neighbor vertex sets, namely the number of elements of the two neighbor vertex sets aiming at the data organization mode of the invention. Compared with the traditional intersection taking method, the method has single applicability, and different strategies can be adopted according to the difference of the sizes of the two sets, so that the calculation efficiency is pertinently improved, and the time overhead is reduced.

Let us say that two vertices v1, v2, neb (v1), neb (v2) to be intersected respectively represent the set of neighbor vertices of the two vertices. Let | neb (v1) | be equal to or less than | neb (v2) | and | neb (v1) | represent the modulus of the neighbor vertex set of vertex v1, and this time, the purpose is to assume that the smaller number of neighbor vertex elements is v1 and the larger number is v 2.

A. When the size of the | neb (v1) | is similar to the size of the | neb (v2) | a method of simultaneously traversing the two sets to compare the element intersection is adopted, namely starting from the smallest element of the neb (v1) and the neb (v2), and simultaneously traversing the elements backwards and comparing until the traversal of the two sets is finished.

With a complexity of

O(|neb(v2)|)

B. When the | neb (v1) | is far smaller than the | neb (v2) |, the set neb (v1) with the smaller element number is traversed, and binary search is sequentially carried out on the set neb (v2) with the larger element number. With a complexity of

O(|neb(v1)|log₂|neb(v2)|)

The advantages of the invention specifically include: storing corresponding graph data in a mode of adding the edge set array and the vertex corresponding to the index array, and reducing space overhead; traversing vertexes and edges in sequence, and reducing the random access memory times; judging whether the edges participate in forming a triangle or not through the bloom screen during traversing the edges, reducing unnecessary intersection times, reducing communication overhead during distributed computation, and indirectly improving computation efficiency; and a dynamic allocation and intersection strategy is adopted, so that the complexity of average time is reduced. Finally, a high-efficiency triangle counting method with low space overhead and friendly access and storage is realized, time and space balance can be considered, and rapid triangle calculation can be carried out on the graph data with the scale as large as possible under the condition of limited memory.

As shown in fig. 2, a large-scale triangle counting system with low space overhead includes the following modules:

Further as a preferred embodiment of the present system, further comprising:

While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for counting triangles in a large-scale graph with low space overhead is characterized by comprising the following steps:

2. The method for counting triangles in a large-scale image with low space overhead according to claim 1, further comprising:

3. The method as claimed in claim 2, wherein the step of obtaining the data set and constructing the directed graph according to the data set to obtain the graph data specifically includes:

4. The method for counting triangles in a large-scale graph with low space overhead according to claim 3, wherein the step of traversing all vertices in graph data and placing the vertices into a preset bloom filter according to a preset rule specifically comprises:

5. The method for counting triangles in a large-scale map with low space overhead as claimed in claim 4, wherein the preset bloom filter is constructed by the number of vertexes in the map data, and the expression is as follows:

6. The method as claimed in claim 5, wherein the step of obtaining two vertices corresponding to the edge according to the edge in the graph data further comprises sorting the two vertices corresponding to the edge in order of vertex number from small to large.

7. The method according to claim 6, wherein the calculating the intersection of the sets of neighboring vertices of the two vertices specifically includes:

8. A low-space-overhead large-scale in-graph triangle counting system is characterized by comprising the following modules:

9. The low-space-overhead large-scale intra-map triangle counting system of claim 8, further comprising: