CN104618153B - Dynamic fault-tolerant method and system based on P2P in the processing of distributed parallel figure - Google Patents

Abstract

The invention relates to a P2P-based dynamic fault tolerance method and system in distributed parallel graph processing. The method includes: defining the data unit of the distributed graph processing problem to ensure the integrity of the recovered data during dynamic fault tolerance; forming a ring structure of the processor nodes, dividing the input graph data into several partitions and assigning each partition to In each processor node, each processor node generates a copy of its own data unit and backs it up in the adjacent processor node; after each processor node executes its own data unit, it updates it incrementally and places it in the adjacent processor node. The copy in the processor node; when a processor node fails or goes offline due to a network error, its adjacent nodes are assigned to use the data copy to replace the original data unit, complete the corresponding operation, and restore the normal execution of the graph processing. The invention can restore the graph processing operation to the normal state from abnormalities such as node failure and network error, so as to ensure the correct execution of the operation.

Description

P2P-based dynamic fault tolerance method and system in distributed parallel graph processing

technical field

The invention relates to the technical field of computer networks, in particular to a dynamic fault-tolerant method and system for distributed parallel graph processing in an open and dynamic network environment.

Background technique

In recent years, with the popularization and development of technologies such as social networks and collaborative networks, the scale of data in the Internet has become larger and larger, which has brought new challenges to the analysis of these data. In scenarios such as social networks and collaborative networks, there may be associations between data, and related research often uses graph structures to describe these data. The vertices in the graph record the attributes of the data itself, and the edges in the graph correspond to the associations between the data. In this way, the analysis of network data is transformed into graph processing. However, the above-mentioned graph structures often have millions, tens of millions of vertices and hundreds of millions of edges, which put a lot of pressure on the memory of ordinary computers. What's more, since the graph processing process will produce intermediate results proportional to the scale of the graph, this makes it difficult for a single machine to perform normal calculations on the graph due to memory constraints.

Therefore, using distributed clusters to process graphs in parallel has become the main way of current network graph data analysis. Taking the Pregel framework proposed by Google as a typical representative, most graph processing systems divide graph data into multiple subgraphs (or graph partitions) and assign them to several machines to perform parallel calculations on graph data. Parallel computing mainly uses the BSP (Bulk Synchronous Processing) model, which performs iterative operations on graph data. In each iterative step, each vertex in the graph obtains information from adjacent vertices, updates its own state, and propagates the new state to adjacent vertices. As shown in Figure 1, between iteration steps, the operations of all vertices need to be synchronized. When all vertices complete the operations of the current iteration step, the next iteration step can be triggered. When all vertices have obtained results, the graph processing computation is complete.

In an open network environment, distributed clusters are characterized by dynamic changes. Some nodes in the cluster may fail, and the network connecting the cluster may fail. These abnormal conditions will affect the normal execution of graph processing jobs. Therefore, distributed parallel graph processing needs to consider the fault tolerance in the operation process. Existing graph processing systems generally implement fault tolerance through the checkpoint mechanism. This kind of fault tolerance requires graph processing to stop the current operation when an error is found, reload the latest checkpoint data from the disk, and re-operate from the iteration point recorded by the checkpoint. This fault-tolerant mechanism is static, and the recovery cost is relatively high, so it is not suitable for an open network environment.

Contents of the invention

The purpose of the present invention is to provide a dynamic fault-tolerant method based on P2P (Peer to Peer, peer-to-peer computing/peer-to-peer network) in distributed parallel graph processing under an open and dynamic network environment, so that graph processing operations can be performed from node failure, It restores the normal state in the event of network errors and other abnormalities to ensure the correct execution of operations.

To achieve the above object, the present invention adopts the following technical solutions:

A P2P-based dynamic fault-tolerant method in distributed parallel graph processing, the steps of which include:

1) Define the data unit of the distributed graph processing problem to ensure the integrity of the recovered data during dynamic fault tolerance;

2) Form the processor nodes into a ring structure, divide the input graph data into several partitions, and assign each partition to each processor node, and each processor node generates a copy of its own data unit and backs it up in the adjacent processor node;

3) In each iterative step of the operation, after each processor node executes its own data unit, it updates its copy placed in the adjacent processor node in an incremental manner;

4) When a processor node fails or is offline due to a network error, its adjacent nodes are assigned to replace the original data unit with a data copy to complete the corresponding operation, thereby restoring the normal execution of graph processing.

Furthermore, the data unit of the distributed graph processing problem defined in step (1) is a two-tuple (P _j ,InMsg(P _j )), where P _j is a certain graph partition divided by the graph structure, that is, a certain A subgraph; InMsg(P _j ) is the set of messages received by all vertices contained in the subgraph P _j in a certain iteration step. At the beginning of the operation, InMsg(P _j ) is an empty set; 1≤j≤m, m is the number of partitions after partitioning the graph.

Further, step (2) divides the input graph data into several partitions, and distributes them to each processor node in the structure of a tuple (P _j , an empty message set).

Further, in step (2), for the data unit assigned to itself, the processor node operates on each vertex in the data unit based on the BSP model; for the data unit copy of the adjacent node, the processor node just records, and After each iteration, neighbors send data to update these replicas.

Further, in step (3), the copy is updated after each iteration step is completed; or, according to the frequency of system errors, taking into account the time cost of the copy update, the copy is updated every certain iteration step.

Further, step (3) in each iteration step of graph processing, each vertex first processes the message set received in the previous iteration step, and updates its own state according to these messages, and then propagates its own state to adjacent vertices. New state; the messages of the same vertex between different iteration steps do not need to be accumulated, and when the update value of the vertex has been calculated in a certain iteration step, the messages received before the iteration step are no longer needed.

A distributed parallel graph processing system adopting the above method, including a controller and processor nodes, the controller is responsible for partitioning input graph data, assigning each partition to each processor node, and monitoring each processor The running status of the node; each processor node forms a ring structure, and each processor node generates a copy of its own data unit for backup in the adjacent processor node, and after each processor node executes its own data unit, it increments Update the copy placed in the adjacent processor node; when a processor node fails or goes offline due to network errors, the controller uses the data copy of its adjacent node to replace the original data unit, and restores the graph processing. Execute normally.

Compared with the prior art, the beneficial effects of the present invention are as follows:

(1) The traditional checkpoint mechanism generally needs to read and write the disk, but the present invention caches the copy required for fault tolerance in the memory. Compared with the traditional checkpoint mechanism, reading and writing memory can realize copy recording and error recovery faster;

(2) The P2P-based copy assignment of the present invention avoids the single hotspot phenomenon in the copy recording and error recovery process, makes the network traffic between the distributed processors more average, and reduces the overall communication time in each iteration step;

(3) The present invention supports dynamic fault tolerance, that is, when an exception occurs during the operation of graph processing, the operation can be resumed from the copy of the data unit closest to the abnormal point, without having to recalculate the data that has been completed correctly, and without having to re-execute the entire graph from the beginning. image processing applications.

Description of drawings

Figure 1 is a workflow flowchart of distributed parallel graph processing based on the BSP model.

Fig. 2 is a schematic diagram of P2P-based data unit copy generation.

Fig. 3 is a schematic diagram of two adjacent processor nodes performing replica update in adjacent iteration steps.

Figure 4 is a schematic diagram of error recovery and reconstruction of P2P replicas.

Fig. 5 is a schematic diagram of UniAS architecture in a specific example.

Detailed ways

In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described below through specific embodiments and accompanying drawings.

The P2P-based dynamic fault-tolerant method in the distributed parallel graph processing of the present invention comprises the following steps:

(1) In order to ensure that data can be recovered completely during the dynamic fault-tolerant process, the present invention defines the data unit for distributed graph processing.

In graph processing, the parameters required for each vertex operation include not only the original state of the vertex itself, but also the messages propagated by its adjacent vertices. Therefore, when defining the data unit of the graph processing problem, not only the dependency on the data in the vertices of the previous iteration should be considered, but also the message dependencies of adjacent vertices need to be considered. In order to describe the data unit more intuitively, the present invention defines the graph structure as follows: G=(V, E), where V represents the set of vertices {v ₁ , v ₂ , v ₃ ...v _n } in the graph, and E represents the vertices in the graph A collection of directed edges. The initial graph structure is divided into a set of several graph partitions, expressed as P={P ₁ , P ₂ , P ₃ . . . P _m }, and these partitions will be assigned to each processor node during operation. For any vertex v _i , the set of messages it receives in a certain iteration step is InMsg(v _i ). Further, for a certain subgraph P _j , the set of messages received by all vertices contained in it at a certain iteration step is InMsg(P _j )={InMsg(v _i )|all v _i in P _j }. Therefore, the present invention defines the data unit for distributed graph processing as a tuple (P _j , InMsg(P _j )), where 1≤j≤m, and InMsg(P _j ) is an empty set at the beginning of the operation. In the following fault-tolerant techniques, it is required to treat the binary as a whole and perform replication or recovery at the same time.

(2) The processor nodes are formed into a ring structure, and each node generates a copy of its own data unit for backup in adjacent processor nodes.

In order to achieve high-efficiency fault tolerance, the present invention does not save the copy of the operation process in the disk, but records it in the memory. However, due to the large scale of the graph in distributed graph processing, if the generated copy of the graph after each iteration operation is stored in a certain processor node, this node may become a data transmission hotspot and further affect the efficiency of the overall operation . Therefore, the present invention distributes the storage of copies to each processor node, so as to balance the amount of data transmission in the process of generating copies. The present invention numbers all processor nodes participating in the operation process to form a ring structure. According to the definition of data unit in (1), the input graph data is divided into several partitions, and assigned to each processor node in the structure of a tuple (P _j , an empty message set). After the allocation is completed, each data unit transmits its copy to two processor nodes with adjacent numbers for storage.

Therefore, each processor node not only saves its assigned data unit (hereinafter referred to as the operation unit), but also caches a copy of the data unit in the adjacent node (hereinafter referred to as the mirror unit), as shown in Figure 2, where A ~H represents the operation unit, and A'~H' represents the mirror image unit. For the data unit assigned to itself, the processor will operate on each vertex in the data unit based on the BSP model; for the copy of the data unit of the adjacent node, the processor just records and waits for each iteration step, the adjacent node Send data to update these replicas.

(3) In operation, after each processor node executes its own data unit, it updates its copies placed in adjacent processor nodes in an incremental manner.

In a distributed graph processing system, each vertex of the graph is continuously updated with iterative operations, which makes the copy set in step (2) need to be continuously updated. And because the operation of each vertex requires the information propagated by its adjacent vertices in the previous iteration step, this makes it necessary to consider the relationship between the iteration steps where multiple data unit copies are located in the data recovery process.

In order to simplify this problem, the update of the copy is guaranteed to be consistent with the iteration step of the copy. The present invention is based on the BSP calculation model, adding a copy updating stage between two iteration steps, and at this stage, all data units in the graph processing are required to update their copies according to the current value. The replica update phase can occur after each iteration step is completed, or it can be performed every certain iteration step according to the frequency of system errors and the time cost of replica update.

In order to reduce the time consumed by each copy transmission and the cost of network resources, and reduce the memory space occupied by the copy cache, the present invention proposes an incremental copy update. Each update of the state of the graph partition in the copy only transmits the changed part, and the adjacent processor nodes accumulate the changed part into the copy of the data unit to realize the update of the state of the graph partition in the copy. In addition, in each iterative step of graph processing, each vertex will perform the following operations, first process the message set received in the last iterative step, and update its own state according to these messages, and then propagate its own state to adjacent vertices new state of . Therefore, the messages of the same vertex between different iteration steps do not need to be accumulated. When the update value of a vertex has been calculated in a certain iteration step, the messages received before the iteration step are no longer needed. Taking a certain graph partition P _j as an example, a copy of (P _j ⁰ , null) is initially cached in an adjacent processor node. When the first iteration step operation is completed, the state of the partition is updated to P _j ¹ , and the partition increment ΔP ¹ is the difference between P _j ¹ and P _j ⁰ . Then the processor unit where P _j is located sends (ΔP ¹ , InMsg(P _j )[1]) to the adjacent processor node. Adjacent processor nodes accumulate △P ¹ into the replica unit (P _j ⁰ , null) to generate P _j ¹ , and replace the message set as InMsg(P _j )[1]. By analogy, when the operation of the kth iteration step is completed, let the partition increment ΔP ^k be the difference between P _j ^k and P _j ^k-1 . In the replica update phase, (△P ^k , InMsg(P _j )[k]) is sent to adjacent processor nodes. Adjacent processor nodes accumulate △P ^k into the replica unit (P _j ^k-1 ,InMsg(P _j )[k-1]) to generate P _j ^k , and delete InMsg(P _j )[k-1], The set of replacement messages is InMsg(P _j )[k]. Figure 3 shows the process of two adjacent processor nodes performing replica updates in adjacent iterations.

(4) When a processor node fails or is offline due to a network error, its adjacent nodes are assigned to replace the original data unit with a copy of the data to complete the corresponding operation, thereby restoring the normal execution of graph processing.

When the graph processing runs to a certain iteration step k, if a processor node fails or goes offline due to a network error, the data units in the offline node cannot continue to operate, but their copies (P _j ,InMsg(P _j )) are cached in adjacent nodes. Based on the method of incremental storage of replica data mentioned in step (3), the present invention can restore these lost data units from the replicas of adjacent nodes, as shown in FIG. 4 .

For this reason, the present invention has designed following dynamic fault tolerance algorithm:

Recovery_scheduling (offline node ID, data unit collection Set in offline nodes) {

Obtain the two adjacent nodes L, R of the offline node;

if(L online && R online){

According to the load of nodes L and R, Set is divided into two subsets SetL and SetR;

Node L sets the mirror unit corresponding to SetL in its own cache as an operation unit, and completes the operation of all vertices in these SetL in the current iteration step;

Node R sets the mirror unit corresponding to SetR in its own cache as an operation unit, and completes the operation of all vertices in these SetR in the current iteration step;

Set nodes L and R as adjacent nodes;

Comparing the computing units in nodes L and R, if there is no mirror unit in the other node, generate a copy of the data unit and send it to the other node;

resume normal execution;

}else if(L online || R online){

N = online nodes;

The node N sets the mirror unit corresponding to the Set in its own cache as the operation unit, and completes the operation of all the vertices in the Set in the current iteration step;

Find the adjacent nodes of N;

Comparing the computing units in the node N and its adjacent nodes, if there is no mirror unit in the other node, generate a copy of the data unit and send it to the other node;

resume normal execution;

}

}

Since the above solution is an extension of the original BSP model, it does not affect the execution of the graph processing system in a normal environment. However, using the solution provided by the present invention, the fault tolerance of the graph processing system does not need to stop the current operation, and all roll back to the nearest checkpoint point to read the copy data from the disk. When the processor node is offline, the graph processing system only needs to adjust and restore the error local node, and the calculation results completed by other normal nodes can continue to be used in the next iteration step.

So far, the present invention completes dynamic fault tolerance in distributed parallel graph processing.

An implementation example of using the present invention to construct distributed parallel graph processing supporting dynamic fault tolerance on a distributed parallel computing platform UniAS is given below.

UniAS is a distributed parallel computing platform independently developed by the Institute of Software, School of Information Science and Technology, Peking University. It currently supports big data processing applications in various modes, including batch processing, graph processing, and stream processing. The following describes the implementation process of the present invention around the graph processing framework in UniAS.

As shown in Figure 5, the graph processing framework in UniAS is implemented based on the Master-Slave structure. The controller is responsible for partitioning the graph, assigning each partition to processor nodes, and monitoring the operation of each node. Each processor node has a data unit queue module, which is responsible for copy maintenance and dynamic fault tolerance during operation. According to the technical points of the present invention, the dynamic fault-tolerant distributed graph processing is realized through several steps below:

1. Start the processor node and register with the controller. The controller numbers all the online processor nodes and arranges them into a ring structure;

2. In the initialization phase, the controller partitions the input graph structure and assigns each partition to the processor node. After the allocation is completed, the copy generation operation of each node is triggered, and each node makes a copy of its own data unit and sends it to the adjacent node;

3. Start the graph processing operation. According to the requirements of the technical point 3 of the present invention (i.e. the above-mentioned step (3)), when each iterative step is completed, the difference between the previous iterative step and the current iterative step result on each vertex is calculated. If the state of the vertex changes, the difference is added to the partition increment; if the vertex is not modified in this iteration, the difference is not added to the partition increment. Send the partition increment and message set of each partition to adjacent nodes for replica update of adjacent nodes;

4. Each processor node periodically sends a heartbeat message to the controller. When the controller fails to receive the heartbeat message of a processor node in time, the controller confirms that the processor node is offline and enters the error recovery stage;

5. The controller invokes the algorithm designed by the technical point 4 of the present invention (i.e. the above-mentioned step (4)), utilizes the copies of adjacent processors to restore lost data, and resets the adjacency state between processor nodes, thereby restoring the program normal operation;

6. When in a certain iteration step, all vertices are processed and all data units do not need to be updated, the entire graph processing job is successfully completed.

So far, the present invention has constructed a dynamic fault-tolerant mechanism of a distributed graph processing framework on the UniAS platform.

The above embodiments are only used to illustrate the technical solution of the present invention and not to limit it. Those of ordinary skill in the art can modify or equivalently replace the technical solution of the present invention without departing from the spirit and scope of the present invention. The scope of protection should be determined by the claims.

Claims (6)

1. A dynamic fault-tolerant method based on P2P in distributed parallel graph processing, its steps comprising: 1) Define the data unit of the distributed graph processing problem to ensure the integrity of the recovered data during dynamic fault tolerance; the data unit is a two-tuple (P _j ,InMsg(P _j )), where P _j is the The structure is divided into a certain graph partition, that is, a certain subgraph; InMsg(P _j ) is the set of messages received by all vertices contained in the subgraph P _j in a certain iteration step. At the beginning of the operation, InMsg(P _j ) is an empty set; 1≤j≤m, m is the number of partitions after partitioning the graph; 2) Number the processor nodes and form a ring structure, divide the input graph data into several partitions according to the definition of data units, and assign each partition to each processor node, and each processor node divides its data unit Generate a copy backup in two processor nodes with adjacent numbers; the copy is not saved in the disk, but recorded in the memory; 3) In each iterative step of the operation, after each processor node executes its own data unit, it updates its copy placed in the adjacent processor node in an incremental manner, and each time the state of the graph partition in the copy is The update of the graph only transmits the changed part, and the adjacent processor nodes accumulate the changed part into the copy of the data unit to update the state of the graph partition in the copy; in each iterative step of graph processing, each vertex is first processed in The set of messages received in the last iteration step, and update its state according to these messages, and then propagate its new state to adjacent vertices; the messages of the same vertex between different iteration steps do not need to be accumulated. After calculating the updated value of the vertex, the messages received before this iteration step are no longer needed; 4) When a processor node fails or is offline due to a network error, its adjacent nodes are assigned to replace the original data unit with a data copy to complete the corresponding operation, thereby restoring the normal execution of graph processing. 2. The method according to claim 1, characterized in that: step 2) divides the input graph data into several partitions, and assigns them to each processor node with the structure of a binary group (P _j , an empty message set) middle. 3. The method according to claim 1, characterized in that: in step 2), for the data unit assigned to itself, the processor node operates on each vertex in the data unit based on the BSP model; For data unit copies, processor nodes just record and wait for each iteration step, when neighboring nodes send data to update these copies. 4. The method according to claim 1, characterized in that: step 3) performs copy update after each iterative step is completed; or according to the frequency of system errors, considering the time cost of copy update comprehensively, every certain The iteration step performs a replica update. 5. A distributed parallel graph processing system adopting the method according to claim 1, characterized in that it comprises a controller and a processor node, the controller is responsible for partitioning the input graph data, and assigning each partition to each In the processor node, and monitor the operation of each processor node; each processor node forms a ring structure, each processor node generates a copy of its own data unit and backs it up in the adjacent processor node, and each processor node executes After completing its own data unit, it incrementally updates its copies placed in adjacent processor nodes; when a processor node fails or goes offline due to a network error, the controller uses the data copies of its adjacent nodes Replaces the original data unit, resuming normal execution of graph processing. 6. The system according to claim 5, wherein each processor node sends a heartbeat message to the controller periodically, and when the controller fails to receive a heartbeat message from a certain processor node in time, the controller confirms the heartbeat message of the processor node. The processor node goes offline and enters the error recovery phase.