JP6535304B2 - Distributed synchronous processing system and distributed synchronous processing method - Google Patents

Description

  The present invention relates to a distributed synchronization processing system and a distributed synchronization processing method that execute processing by synchronizing a plurality of distributed servers.

  Non-Patent Document 1 discloses MapReduce as a framework of a distributed processing system in which a plurality of servers are distributed and arranged on a network. However, since this MapReduce requires reading out input data from an external data store and writing out results every time processing, it is an iterative (iteration that uses the result of a certain processing in the next processing) Yes) not suitable for processing. For this type of processing, BSP (Bulk Synchronous Parallel) disclosed in Non-Patent Document 2 is suitable.

  The BSP executes data processing in a distributed environment by repeatedly executing a processing unit called "superstep (SS)". FIG. 1 is a diagram for explaining a BSP calculation model.

As shown in FIG. 1, one super step includes the following three phases (PH: phase), “local computation (LC: Local computation) (phase PH1),“ data exchange (Com: Communication) (phase: PH2), “Sync” (phase PH3).
Specifically, when any of a plurality of nodes (node 1 to node 4) receives data, that node (for example, node 1) performs calculation processing (local calculation) for the data in phase PH1. Execute (LC)). Subsequently, in phase PH2, data exchange between nodes is performed on data which is the result of the local calculation held by each node. Next, in phase PH3, synchronization processing is performed, more specifically, waiting for the end of data exchange between all nodes.
Then, when the series of super step processing (PH1 to PH3) is finished as the super step SS1, each node holds the calculation result, and proceeds to the next series of processing, super step SS2.

  Non-Patent Document 3 discloses Pregel as a distributed processing framework adopting this BSP. In the framework such as Pregel, the entire processing is expressed as a graph G = (V, E), which is applied to BSP and executed. Here, V is a “set of vertexes”, and E is a “set of edges”.

Here, with reference to FIG. 2, an example in which BSP is applied to traffic simulation will be described.
In FIG. 2, each intersection (v) is associated with a vertex (v ₁ to v _{4 in} FIG. 2). In addition, the road (e) connecting each intersection is associated with an edge (e ₁ to e _{6 in} FIG. 2). Here, the edge is one-way, and a bidirectional road is associated with two edges. Also, an edge in the direction in which the vehicle exits is referred to as an "outgoing edge" as viewed from a certain vertex, and an edge in the direction in which the vehicle flows in is referred to as an "incoming edge". . For example, in FIG. 2, when viewed from the vertex _{v 2,} the edge _{e 1} is input edge, the edge _{e 2} is the output edge. Conversely, when viewed from the vertex v _1, the edge e ₁ is the output edge, the edge e ₂ is the input edge.

In the super step shown in FIG. 1, in phase PH1 (local calculation), for each vertex (vertex), the state of the intersection associated with each vertex v _{1 to} v ₄ (e.g. It simulates the color of the signal (blue, yellow, red), the movement of the vehicle in the intersection, etc., and the state in the road (movement of vehicles (average number, etc.)) as an output edge accompanying it. In phase PH2 (data exchange), one vertex transmits information (such as the number of vehicles) of a vehicle moving out through the output edge to another vertex contacting via the output edge, and also inputs the input edge. Receive information (number etc.) of incoming vehicle movement via. In phase PH3 (synchronization), simulation time t is synchronized between vertices. That is, it waits for completion of data exchange among all the vertices.
In this traffic simulation, it is possible to reduce the calculation time by performing parallel processing in intersection units (vertex) in this way.

Dean, J., et al., “MapReduce: Simplified Data Processing on Large Clusters,” OSDI '04, 2004, p. Valiant, L., et al., “A bridging model for parallel computation,” Communications of the ACM, 1990, vol. 33, No. 8, p. 103-111. Malewicz, G., et al., “Pregel: A System for Large-Scale Graph Processing,” Proc. Of ACM SIGMOD, 2010, p. 136-145.

  The master / worker configuration is adopted as an architecture for realizing the distributed processing framework adopting BSP as described above. As shown in FIG. 3, in the master / worker configuration, a plurality of workers (processing server 30a) having a plurality of vertexes 20a serving as a processing unit and a master (management server 10a) that manages the progress of the processing of the worker, etc. It consists of one.

Here, the role of master (management server 10a) is allocation of processing (vertex 20a) to worker (processing server 30a) (partitioning of graph G), management of progress of processing of worker, common to all workers Management of supersteps as a whole, and management of graph topology accompanying addition and deletion of vertices and edges.
The role of the worker (processing server 30a) is local calculation of the phase PH1 in each superstep, transmission / reception of data between adjacent vertices in the phase PH2, and reporting to the master.

  Many of the architectures in the existing framework adopt this master / worker configuration, and when BSP is applied, the worker will be master when processing of all vertices that he / she has (phase PH1, 2) is completed. Report to When the master receives a report from all the workers, it instructs the workers to "+1" the super step and to shift to the next super step.

However, in the above configuration, since all vertices are synchronized every super step, it is matched with the latest processing vertex. Therefore, if there is only one vertex that is extremely slow from the whole, the influence will be overall. That is, the whole is significantly delayed according to the slowest processing vertex.
Also, when processing a large graph G, that is, when dealing with a calculation object having a large number of vertices and edges, in the master / worker configuration, one master manages the entire graph; If the scale is large, the master will be a bottleneck.

  Therefore, it is an object of the present invention to provide a distributed synchronization processing system and a distributed synchronization processing method capable of solving the above-mentioned problems and reducing the processing delay of the entire system involved in synchronization processing.

  In order to solve the problems described above, the invention according to claim 1 is necessary for a plurality of processing servers performing processing in parallel, a plurality of distributed processing units operating on the processing server, and target calculation processing. A distributed synchronous processing system, comprising: a management server that allocates a plurality of distributed processing units to a plurality of processing servers, wherein the processing server performs calculation processing and calculation in a predetermined calculation step by the distributed processing units. It detects completion of calculation / transmission processing indicating transmission processing to the distributed processing unit connected as a result output destination, generates a completion report indicating completion of the calculation / transmission processing, and transmits it to the management server A distributed process that receives from the management server an instruction to move to the next step, which is an instruction to move to the next calculation step, and outputs the instruction to the distributed processing unit that has completed the calculation and transmission process. Whether the distributed processing unit that has a management unit, the management server has received the completion report, and has completed the calculation and transmission processing has completed acquisition of the calculation result required in the next calculation step The determination is made based on whether or not the completion report from the distributed processing unit connected as the input source of the calculation result is received, and when the acquisition of the calculation result is completed, the next step shift instruction is completed According to another aspect of the present invention, there is provided a distributed synchronization processing system comprising: an adjacent synchronization processing unit that transmits a report to a processing server that has transmitted the report.

  The invention according to claim 3 is that the plurality of processing servers performing processing in parallel, the plurality of distributed processing units operating on the processing server, and the plurality of distributed processing units necessary for the target calculation processing A distributed synchronous processing method of a distributed synchronous processing system, comprising: a management server that assigns the plurality of processing servers to the plurality of processing servers, wherein the processing server performs calculation processing and calculation results in predetermined calculation steps by the distributed processing unit. Detecting completion of calculation / transmission processing indicating transmission processing to the distributed processing unit connected as an output destination of the process, generating a completion report indicating completion of the calculation / transmission processing, and transmitting it to the management server Receiving an instruction to move to the next step, which is an instruction to shift to the next calculation step, from the management server, and outputting the instruction to the distributed processing unit that has completed the calculation and transmission process. The management server receives the completion report, and the distributed processing unit that has completed the calculation / transmission process calculates whether or not the acquisition of the calculation result required in the next calculation step is completed. The determination is made based on whether or not the completion report from the distributed processing unit connected as the input source of is received, and when the acquisition of the calculation result is completed, the completion report is transmitted with the next step shift instruction The distributed synchronous processing method is characterized in that the procedure for transmitting to the processing server which has been performed is executed.

  As described above, the distributed synchronous processing system can determine, for each distributed processing unit, whether the management server may shift to the next calculation step. Therefore, since it is not necessary to wait until the end of the calculation / transmission processing of all the distributed processing units, it is possible to reduce the processing delay of the entire system accompanying the synchronization processing.

  The invention according to claim 2 is a distributed synchronous processing system having a plurality of processing servers performing processing in parallel, and a plurality of distributed processing units operating on the processing server, wherein the processing server is Calculation processing and transmission processing completion indicating the transmission processing to the distributed processing unit connected as an output destination of the calculation result in a predetermined calculation step by the distributed processing unit is detected, and the distribution processing completed the calculation and transmission processing The processing unit determines from the distributed processing unit connected as an input source of the calculation result whether acquisition of the calculation result necessary in the next calculation step is completed and acquisition of the calculation result is completed. And an adjacent synchronization and dispersion management unit for outputting, to the distributed processing unit that has completed the calculation / transmission processing, an instruction for moving to the next calculation step, which is an instruction to move to the next calculation step. It was distributed synchronization system that.

  The invention according to claim 4 is a distributed synchronous processing method of a distributed synchronous processing system including a plurality of processing servers performing processing in parallel and a plurality of distributed processing units operating on the processing server, A procedure in which the processing server detects the completion of the calculation processing and transmission processing indicating the calculation processing and the transmission processing to the distributed processing unit connected as an output destination of the calculation result in the predetermined calculation step by the distributed processing unit; The distributed processing unit that has completed the calculation / transmission processing determines whether acquisition of the calculation result necessary in the next calculation step has been completed from the distributed processing unit connected as an input source of the calculation result, When acquisition of the calculation result is completed, an instruction to output a next step shift instruction which is an instruction to shift to the next calculation step to the distributed processing unit which has completed the calculation / transmission processing When, it was distributed synchronization processing method, characterized by the execution.

As described above, the distributed synchronous processing system can determine, for each distributed processing unit, whether the processing server may shift to the next calculation step. Therefore, since it is not necessary to wait until the end of the calculation / transmission processing of all the distributed processing units, it is possible to reduce the processing delay of the entire system accompanying the synchronization processing.
Furthermore, since each processing server autonomously determines the shift to the next calculation step in a distributed manner, the processing delay of the entire system is reduced even in a large-scale system in which the number of processing servers and distributed processing units are large. It becomes possible.

  According to the present invention, it is possible to provide a distributed synchronization processing system and a distributed synchronization processing method that reduce the processing delay of the entire system involved in synchronization processing.

It is a figure for demonstrating a BSP calculation model. It is a figure for demonstrating the example which applied the BSP calculation model to traffic simulation. It is a figure for demonstrating the master / worker structure of the distributed synchronous processing system which concerns on a comparative example. It is a figure for demonstrating the definition of the component of a vertex. It is a figure which illustrates the component of one vertex. It is a figure which illustrates the graph of calculation object in a BSP calculation model. It is a figure for demonstrating the flow of the process in the distributed synchronous processing system of a comparative example. FIG. 8 is a view for explaining the flow of processing of the distributed synchronous processing system according to the comparative example (FIG. 8A) and the flow of processing of the distributed synchronous processing system according to the present embodiment (FIG. 8B). is there. FIG. 1 is a diagram showing an overall configuration of a distributed synchronous processing system according to an embodiment of the present invention. It is a sequence diagram which shows the flow of a process of the distributed synchronous processing system which concerns on this embodiment. It is a figure which shows the whole structure of the distributed synchronous processing system which concerns on the modification of this embodiment. It is a flowchart which shows the flow of a process of the distributed synchronous processing system which concerns on the modification of this embodiment.

<Details of contents and problems of distributed processing method of comparative example>
First, in order to describe the characteristic configurations of the distributed synchronization processing system 1 and the distributed synchronization processing method according to the present embodiment, the distributed synchronization processing system 1a and the distributed synchronization processing method in the related art will be described in detail as comparative examples.

  The distributed synchronous processing system 1a of the comparative example adopts a master / worker configuration as shown in FIG. 3, and each of a plurality of workers has a plurality of vertices. Then, when applying BSP to this master / worker configuration, each worker reports to master that all vertex processing (phase PH1, 2) that it has is completed, and master receives reports from all workers. And move the superstep to the next superstep.

Here, focusing on vertices, each vertex executes the following processing.
Vertexes are information obtained in the phase PH1 of the BSP, the state of the current vertex, the state of the output edge, and the input message of the previous super step (hereinafter may be simply referred to as "step") (the input edge's The calculation is performed with the state) as a parameter, and the state of the vertex and the state of the output edge are updated. Then, in phase PH2, the vertex transmits the updated output edge state as an output message to the vertex adjacent to the output edge. The "vertex adjacent to the output edge" means "vertex connected as an output destination of the calculation result".

The above process (calculation and transmission process) can be expressed as the following equation (1).
f (v _{vid, n} , E _{out, n} , M _{in, n-1} ) = (v _{vid, n + 1} , E _{out, n + 1} , M _{out, n} ) (1)
Here, the definition of each component of vertex is shown in FIG.

As shown in FIG. 4, “vid” indicates “vertex ID”. "V _{vid, n} " indicates "state of vertex". "N" indicates the current step (super step). “E _{out, n} ” indicates a set of output edge states. “M _{in, n} ” indicates information (for the current step) stored in the buffer of the input message indicating the state of the input edge. “M _{in, n−1} ” indicates the information (for the previous step) stored in the buffer of the input message. "S _n " indicates the "status flag (active / inactive)" of the current step. “S _{n + 1} ” indicates the “status flag (active / inactive)” of the next step. f (v _{vid, n} , E _{out, n} , M _{in, n-1} ) = (v _{vid, n + 1} , E _{out, n + 1} , M _{out, n} ) is as shown in the equation (1) Shows the calculation and transmission process. Here, hereinafter, the calculation / transmission process in step (super step) n will be referred to as “calculation / transmission process f _n ”.
The status flag “s _n ” is in the “active” status while the vertex is executing the processing of phases 1 and 2 of the BSP, and the processing of other vertexes in the synchronous processing of the phase PH 3 is waiting for processing When it is in the "inactive" state. In addition, “s _{n + 1} ” is set to “active” in the case of setting to shift to the process of the next step, setting that the process is not performed in the next step due to the end of the setting time of the simulation process, etc. In the case of, the state of "inactive".

FIG. 5 is a diagram illustrating components in the case of focusing on one vertex.
As shown in FIG. 5, the vertex "1" with "vertex ID""1" in the current step "n" holds the state "v _{1, n} " of the vertex in the step "n". Also, vertex "1" outputs "e _{1,3, n} " as vertex "3" and "e _{1,4, n} " as vertex "4" as the output edge state. Then, vertex “1” receives “m _{2,1, n} ” from vertex “2” as information (state of input edge) of the input message, and vertex “3” to “m _{3,1, n} ” Receive

  The worker (see FIG. 3) manages the status flag (active / inactive) of the current step (superstep) and the status flag (active / inactive) of the next step (superstep), for each vertex that the worker has. Also, when a worker sends an output edge status as an output message from a vertex that belongs to itself to a vertex that belongs to another worker, it sends messages collectively by buffering messages to a vertex that belongs to the same worker. You may By doing this, communication costs can be reduced.

Next, with reference to FIG. 7, the flow of processing executed by the distributed synchronous processing system 1a of the comparative example will be described. Here, it is assumed that the calculation target of the graph G is the graph topology shown in FIG. Further, as shown in FIG. 7, consists of one master and two worker (worker1, worker2), among the vertex _v 1 to v _6, the vertex _v 1 to v ₃ worker1 is responsible, vertex v _It is assumed that worker 2 is in charge of _{4 to 6} v. Hereinafter, the entire processing flow will be described.

First, master may each vertex of a graph G shown in FIG. 6 (Vertex _v 1 to v _6), and set as a processing target allocated to the worker (step S101), i.e., executes a partitioning of the graph G.
Here, as shown in FIG. 6, of the vertex _v 1 to v _6, allocate vertex _v 1 to v ₃ in worker1, it shall allocate vertex _v 4 to v ₆ to worker2.

  Subsequently, each worker (worker1, worker2) executes the superstep of the vertex in charge (step S102). Specifically, the local calculation of phase PH1 is performed, and the processing of the superstep is started.

  Next, each worker monitors the progress of processing of the vertex that he is in charge of, and determines whether or not each vertex is completed until the data exchange of the phase PH2. Then, each worker, when confirming that all the vertexes in charge have completed the processing up to the phase PH2, reports (sends) the status flag in the next super step of each vertex to the master (step S103). Here, the worker reports “active” (setting to shift to the processing of the next super step) as a status flag in the next super step of each vertex.

Then, the master confirms whether or not a status flag indicating the completion of the process has been received from all the workers (worker1, worker2). When the master receives a report from all the workers, it updates the super step to "+1" (step S104).
Here, if there is a change in graph topology, for example, if there is a vertex or edge addition or deletion, the master notifies each worker of the change in graph topology.

  Subsequently, the master instructs all the workers (worker1, worker2) to shift to the super step next (step S105). And each worker repeats step S102-S105.

In the distributed synchronous processing system 1a of the comparative example, all vertices to be calculated are synchronized at each superstep, specifically, in order to synchronize at the entire synchronization point shown in FIG. It becomes. For example, the Super step SS1 in FIG. 7, of the vertex _v 1 to v _6, and thus to match the slowest vertex _{v 2.} Furthermore, the Super step SS2, and thus fit the slowest vertex _{v 6.} Thus, if there is a significantly slower vertex, then the overall processing of the vertex will be significantly delayed to match that vertex.
Also, in the master / worker configuration, one master manages the whole, so when the scale of the graph G becomes large, that is, when the number of vertices and the number of workers increase, the master becomes a bottleneck It becomes.

It solves the above-mentioned delay of the processing speed as a whole and the problem of not performing processing in phase PH3 but having many synchronization waits (processing efficiency) (hereinafter referred to as the "processing speed / efficiency" problem). For this purpose, an asynchronous distributed processing framework has been proposed (see, for example, Non-Patent Document 4).
Here, Non-Patent Document 4 is “Low, Y., et al.,“ Distributed Graph Lab ”, Proc. Of the VLDB Endowment, 2012.”.

However, in the asynchronous distributed processing framework described in Non-Patent Document 4, there is a trade-off between processing speed / efficiency and calculation accuracy, so the programmer's burden in designing processing (complexity of program) Will increase.
Specifically, in the asynchronous type, since each vertex does not guarantee that the same superstep is executed, it is necessary for the programmer to consider overtaking or overwriting of processing between the vertices. Passed over iterations will be invalidated, which will lead to a loss of accuracy. In addition, the theoretical guarantee of accuracy becomes difficult as the number of superstep overtakings increases without limit.

The distributed synchronous processing system 1 (refer to FIG. 9) and the distributed synchronous processing method according to the present embodiment are synchronous type and programmer-friendly for these problems (that is, no consideration of overtaking or overwriting of processing becomes necessary). The task is to improve the processing speed / efficiency that has been synchronous and problematic while providing a simple framework.
Furthermore, it is an object to prevent the bottleneck of the master and secure processing speed / efficiency even for a large graph G.

  Of the management of graph topology that the master originally executes, “Dynamic addition of elements (vertex and edges)” requires a change in the configuration as a system, so this is not applicable to the present invention. The subject of the present invention is the case where there is no need for 'dynamic addition of elements'.

<Overview of this embodiment>
Next, an outline of processing executed by the distributed synchronous processing system 1 according to the present embodiment will be described.
In the distributed synchronous processing system 1 (FIG. 9 described later) according to the present embodiment, synchronization processing is not performed on all vertices (the “distributed processing unit 20” described later) by the master (the “management server 10” described later). It is characterized by judging the transition to the next super step every time. As a result, the distributed synchronous processing system 1 significantly reduces the influence of vertices that are slow in processing.

Specifically, in the distributed synchronous processing system 1, the transition condition to the next super step is set as "the calculation / transmission processing f _{n of} all vertices contacting at the own vertex and the input edge is completed". Note that "all vertices in contact with input edge" mean all vertices connected as input sources of calculation results. Hereinafter, this “transition condition to the next super step” is referred to as “adjacent synchronization”. The details of this adjacent synchronization will be described with reference to FIG.

8 shows a process (see FIG. 8A) executed by the distributed synchronous processing system 1a of the comparative example shown in FIG. 7 and a process executed by the distributed synchronous processing system 1 according to the present embodiment (FIG. 8 (b) )).
In the distributed synchronous processing system 1 according to the present embodiment, as described above, “the calculation and transmission processing f _{n of} all vertices in contact with the own vertex and the input edge is completed” (“adjacent synchronization”), Move to the next super step.

For example, focusing on the vertex _{v 2} in FIG. 8 (b), the vertex _{v 2} is the vertex _{v 1,} v 3, _v calculation and transmission processing of _{4 f n} and their calculation and transmission processing _{f n} in contact with the input edge The end point is the adjacent sync point adjacent sync. Here vertex v _2, when the super step SS1, since the end of calculation and transmission processing f ₁ itself was later slowest from vertex _{v 1, v 3, v 4} , and that point adjacent synchronization points It has become.
Focusing on the vertex v _3, Vertex v ₃ is an adjacent sync point when the vertex v _2, v ₄ of calculation and transmission processing f _n and their calculation and transmission processing f _n in contact with input edge is finished is adjacent sync Become. Here vertex v _3, when the super step SS1, at the time the end of the calculation and transmission processing f ₁ itself, calculation and transmission processing f ₁ Vertex v ₄ is ending, the vertex v ₂ calculation and transmission since the process f ₁ is not finished, waiting in the state of "inactive" (reference numeral α in FIG. 8 (b)), and neighboring synchronization point when the calculation and transmission processing f ₁ vertex v ₂ is finished is adjacent sync Become.
Further, when focusing on the vertex v _1, vertex v _1, the vertex in contact with the input edge is not present, therefore, when the super step SS1, adjacent synchronization when the calculation and transmission processing f ₁ of the own vertex is completed adjacent sync It becomes a point.

As shown in FIG. 8 (b), there is a possibility that the superstep may be shifted between the vertices when viewed at a certain point in the entire process. Therefore, when transmitting and receiving messages between vertices, management is performed for each super step without overwriting. That is, super step information (step number) is stored together. Therefore, in addition to the elements of the vertices shown in FIG. 4, each vertex in the present embodiment stores “M _{in, n + m} ” in the buffer of the input message. Here, “M _{in, n + m} ” indicates the information stored in the buffer as the state of the input edge at the step number n + m (“m” is a positive integer). When each vertex acquires the state of the input edge from the vertex which has shifted to the next super step before its own super step (for example, step number “n” (current step)), the step number n + 1, The state of the input message is stored together with the step numbers n + 2,..., n + m.

As described above, by shifting to the next super step based on the adjacent synchronization, logically, when focusing on each of the vertices, synchronization within the same super step is established. Therefore, the programmer can eliminate the need to consider the trade-off between processing speed / efficiency and calculation accuracy such as asynchronous processing.
Further, compared to the comparative example shown in FIG. 8A, the time for waiting for synchronization as inactive is significantly reduced (symbol β in FIG. 8B), so that processing speed / efficiency can be improved. It becomes. That is, it is possible to solve the problem of the delay of the processing speed of the whole system and the fact that the processing is not performed in the phase PH3 and the number of synchronization waits is large (processing efficiency).

<< Configuration of Distributed Synchronous Processing System >>
Next, the configuration of the distributed synchronous processing system 1 according to the present embodiment will be specifically described.
As shown in FIG. 9, the distributed synchronous processing system 1 operates on the management server 10 (master), a plurality of processing servers 30 (workers) connected to the management server 10 and performing processing in parallel, and the processing server 30. And a plurality of distributed processing units 20 (vertex).

  The management server 10 and the processing server 30 include hardware as a general computer, such as a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), and a hard disk drive (HDD). The HDD stores an operating system (OS), application programs, various data, and the like. The OS and application programs are expanded in the RAM and executed by the CPU. In FIG. 9, the inside of the management server 10, the distributed processing unit 20, and the processing server 30 is shown as a block the function (feature configuration) realized by the application program etc. expanded in the RAM.

The management server 10 functions as a master that manages the entire system. The management server 10 allocates, to the processing server 30 functioning as a worker, a plurality of calculation processes divided into predetermined units for the entire target calculation process. Each calculation process includes data input, calculation, message transmission / reception, and the like. On a plurality of processing servers 30 (workers) performing processing in parallel, a plurality of distributed processing units 20 respectively corresponding to individual calculation processing operate. When the target calculation process is expressed as a graph G = (V, E), individual calculation processes required for the calculation process are expressed as individual vertices (vertex) in the graph G. That is, the distributed processing unit 20 functions as a vertex (vertex).
Hereinafter, each device constituting the distributed synchronous processing system 1 will be described in detail.

<Management server (master)>
The management server 10 performs setting of individual calculation processing (vertex) necessary for target calculation processing and allocation of the individual calculation processing (vertex) to each processing server 30 (worker). Also, the management server 10 manages the entire target calculation processing by performing processing of determining whether to shift to the next super step in the BSP for each vertex set in the system. Do.
The difference between the conventional distributed synchronous processing system 1a shown in FIG. 3 and the master is that the transition to the next super step is not judged by the completion of processing of all vertices, and this embodiment In the management server 10 (master) according to the present invention, the determination is made based on the “adjacent synchronization” described above for each vertex.

The management server 10 includes an adjacent synchronization processing unit 11 as its characteristic configuration.
The adjacent synchronization processing unit 11 performs the calculation and transmission processing f _n from each processing server 30 (worker) for each distributed processing unit 20 (vertex), that is, the phase PH1 (local calculation) and the phase PH2 (data When the exchange) is completed, a completion report of calculation / transmission processing f _n (hereinafter referred to as “calculation / transmission processing completion report”) is received.
Then, the adjacent synchronization processing unit 11 performs the next superstep on the distributed processing unit 20 (vertex) indicated by the received calculation / transmission processing completion report, that is, the distributed processing unit 20 (vertex) whose calculation / transmission processing is completed. Judgment of transition to the network is made based on the above-mentioned "adjacent synchronization" condition. That is, the adjacent synchronization processing unit 11 determines whether or not the condition of “the calculation and transmission processing f _{n of} all vertices in contact with the own vertex and the input edge is completed” (adjacent synchronization) is satisfied. It should be noted that the determination of the adjacent synchronization is made by receiving a calculation / transmission processing completion report from the adjacent distributed processing unit 20 (vertex) as to whether or not acquisition of the necessary calculation result is completed in the next super step. It means to make a decision based on the presence or absence.

  When the distributed processing unit 20 (vertex) indicated by the received calculation / transmission processing completion report satisfies the adjacent synchronization condition, the adjacent synchronization processing unit 11 determines the next superimposition of the distributed processing unit 20 (vertex). The instruction is transmitted to the processing server 30 (worker) in charge of the distributed processing unit 20 (vertex) so as to shift to the step (make the super step “+1”). The instruction to shift to the next super step by the adjacent synchronization processing unit 11 is hereinafter referred to as “next step shift instruction”.

  Further, when the adjacent synchronization processing unit 11 receives a calculation / transmission processing completion report of a certain distributed processing unit 20 (vertex), an output edge of the distributed processing unit 20 (vertex) indicated by the calculation / transmission processing completion report is output. Of the distributed processing units 20 (vertex) in contact with each other, waiting for the input message from only the distributed processing unit 20 (vertex) (waiting for acquisition of the state of the input edge). If there is 20 (vertex), the next step shift instruction is transmitted so that the distributed processing unit 20 (vertex) can shift to the next super step.

Specifically, this will be described with reference to FIG. Focusing on the vertex v ₃ when the super step SS1, Vertex v ₃ is a neighboring synchronization point time when the vertex v _2, v ₄ and its own calculation and transmission processing f ₁ in contact with input edge finished is adjacent sync . Here vertex v ₃ is the time when the calculation and transmission processing f ₁ itself is finished, calculation and transmission processing f ₁ Vertex v ₄ is ending, not end calculation and transmission processing f ₁ Vertex v ₂ is Because there is no, it is waiting in the state of "inactive". In this state, when the adjacent synchronization processing unit 11 of the management server 10 receives, from the processing server 30 (worker 1), the calculation / transmission processing completion report indicating that the calculation / transmission processing f _{1 of the} vertex v ₂ is completed, The next step shift instruction is sent to the vertex v ₃ which has been waiting for an input message from the vertex v ₂ only (waiting to obtain the state of the input edge).
By doing so, ends the calculation and transmission processing f _n itself, the distributed processing unit was waiting in inactive state 20 (vertex), it can be then proceeds to step.

<Distributed processing unit (vertex)>
Returning to FIG. 9, the distributed processing unit 20 (vertex) executes calculation processing divided into predetermined units, and includes a numerical calculation unit 21 and a message transmission / reception unit 22.

The numerical calculation unit 21 executes the process of phase PH1 (local calculation) in BSP. The numerical value calculation unit 21 shifts to the next super step in accordance with the next step shift instruction received via the message transmission / reception unit 22. The numerical calculation unit 21, after the calculation and transmission processing f _n itself is completed, until it receives the next step change instruction waits in inactive state.

The message transmission / reception unit 22 transmits / receives information to / from another distributed processing unit 20 or processing server 30 (worker). Specifically, in phase PH2 (data exchange) in the BSP, the message transmitting / receiving unit 22 transmits the state of its output edge as an output message toward a vertex connected at the output edge. The output message is assigned the step number of the superstep at that time in association with the state of the output edge. Further, the message transmitting / receiving unit 22 receives the state of the input edge as an input message from the vertex connected at the input edge. Also, in this input message, the step number of the superstep at that time is attached to the input edge in association with the state. The message transmission / reception unit 22 transmits / receives the output message and the input message via the message processing unit 32 of the processing server (worker) 30.
Further, the message transmitting / receiving unit 22 receives an instruction to shift to the next step from the processing server 30 (worker) to which the message transmission / reception unit 22 belongs, and outputs the instruction to the numerical value calculation unit 21.

<Processing server (worker)>
The processing server 30 (worker) (see FIG. 9) is connected to the management server 10 (master) and other processing servers 30 (worker). The processing server 30 (worker) includes a plurality of distributed processing units 20 (vertex) which are processing units, and manages the progress of the processing of the distributed processing unit 20 (vertex) provided in itself, and the other processing servers 30. It exchanges information with (worker) and the management server 10 (master). The processing server 30 (worker) includes a virtualization control unit 31, a message processing unit 32, and a vertex management unit 33 (distributed processing management unit).

  The virtualization control unit 31 constructs a virtualization platform on the processing server 30 based on virtualization technology, and performs control of arranging a plurality of distributed processing units 20 (virtual machines).

  The message processing unit 32 receives an output message indicating the state of the output edge at each phase PH2 (data exchange) in the BSP from each distributed processing unit 20 (vertex) belonging to itself, and the graph topology of the graph G to be calculated And outputs the received output message as an input message indicating the state of the input edge to the vertex connected at the output edge based on Hereinafter, when the output message and the input message are not particularly distinguished, they may be simply referred to as a "message".

The message processing unit 32 can transmit a message to the distributed processing unit 20 (vertex) adjacent at the output edge, for example, at two timings shown below.
(Timing 1)
It transmits immediately after the end of the calculation of the self-distributed processing unit 20 (vertex).
Specifically, when the message processing unit 32 receives an output message from the self-distributed processing unit 20 (vertex), the distributed processing unit 20 (vertex) connected by the output edge belongs to the distributed processing unit 20 belonging to itself. When it is (vertex) and when it is the distributed processing unit 20 (vertex) belonging to another processing server 30, it is immediately transmitted to the distributed processing unit 20 (vertex).
By doing so, the influence of communication delay can be reduced.

(Timing 2)
The distributed processing unit 20 (vertex) (adjacent vertex) connected at the output edge buffers until immediately before moving to the next super step.
Specifically, when the message processing unit 32 receives an output message from the self-distributed processing unit 20 (vertex), the distributed processing unit 20 (vertex) connected by the output edge belongs to another processing server 30. When the distributed processing unit 20 (vertex) is in the state of receiving information (instruction to shift to the next step) when it is the distributed processing unit 20 (vertex), the management server 10 The information to the effect that the next step shift instruction is issued is acquired in advance. Then, the messages buffered immediately before the transition are collectively sent to the distributed processing unit 20 (vertex) connected at the output edge.
By doing this, it is possible to reduce the number of transmissions (the number of communication) to the distributed processing unit 20 (vertex) belonging to another processing server 30.
When the message processing unit 32 receives an output message from the self-distributed processing unit 20 (vertex), the distributed processing unit 20 (vertex) connected by the output edge belongs to the distributed processing unit 20 (vertex) belonging to itself. In this case, since the reduction effect of the number of communications as described above can not be obtained, it is transmitted without buffering.

The vertex management unit 33 (distributed processing management unit) monitors the distributed processing unit 20 (vertex) belonging to itself, and when each distributed processing unit 20 (vertex) completes the calculation and transmission processing f _n , that is, the phase When PH1 (local calculation) and phase PH2 (data exchange) are completed, a completion report (calculation / transmission process completion report) of calculation / transmission process f _n is generated and transmitted to the management server 10 (master).
Then, the vertex management unit 33 receives the next step shift instruction from the management server 10 (master) as a response to the calculation / transmission processing completion report, the distributed processing unit 20 (targeted to the next step shift instruction) Output to vertex).

<< Operation of distributed synchronous processing system >>
Next, the operation of the distributed synchronous processing system 1 will be described.
FIG. 10 is a sequence diagram showing a flow of processing of the distributed synchronous processing system 1 according to the present embodiment.
Here, the management server 10 (master) sets individual calculation processes (vertex) required for the target calculation process and sets the individual calculation processes (vertex) to each processing server 30 (worker). Explain that allocation has already been completed.

First, the vertex management unit 33 (distributed processing management unit) of the processing server 30 (worker) monitors and calculates the distributed processing unit 20 (vertex) by monitoring the distributed processing unit 20 (vertex) belonging to the processing server 30 (worker). It is detected that f _n has been completed (step S10). Then, the vertex manager 33 transmits to the management server 10 a calculation / transmission processing completion report with the identification number of the distributed processing unit 20 (vertex) and the step number (n) of the superstep at that time. (Step S11).

Next, the management server 10 (master) that has received the calculation / transmission processing completion report determines that the adjacent synchronization processing unit 11 performs adjacent synchronization for the distributed processing unit 20 (vertex) indicated by the received calculation / transmission processing completion report. It is determined whether the condition is satisfied (step S12). Specifically, the adjacent synchronization processing unit 11 determines whether or not “the calculation and transmission processing f _{n of} all the vertices in contact with the own vertex and the input edge is satisfied”.

  Then, when the condition for adjacent synchronization is not satisfied (step S12 → No), the adjacent synchronization processing unit 11 of the management server 10 (master) receives the next calculation / transmission processing completion report from the processing server 30 (worker). Wait until you do.

  On the other hand, in step S12, the adjacent synchronization processing unit 11 of the management server 10 (master) has sent the calculation / transmission processing completion report when the adjacent synchronization condition is satisfied (step S12: Yes) The next step shift instruction is sent to the distributed processing unit 20 (vertex) so as to shift to the next super step (step S13).

  Further, in step S12, the adjacent synchronization processing unit 11 of the management server 10 (master) is a distributed processing unit 20 (vertex) in which the distributed processing unit 20 (vertex) indicated by the received calculation / transmission processing completion report contacts at the output edge. Whether there is a distributed processing unit 20 (vertex) waiting in the inactive state due to the reason of waiting for an input message from the distributed processing unit 20 (vertex) alone (waiting for acquisition of the state of the input edge) ”. Determine Then, when there is the corresponding distributed processing unit 20 (vertex), the adjacent synchronization processing unit 11 instructs the processing server 30 (worker) to which the distributed processing unit 20 (vertex) belongs to the next step migration. Send.

The vertex management unit 33 of the processing server 30 (worker) that has received the instruction to shift to the next step is in the inactive state in the distributed processing unit 20 (vertex) in which the calculation and transmission processing f _n is completed and in step S12 above. An instruction to shift to the next step is output to the distributed processing unit 20 (vertex) determined to have been on standby (step S14). As a result, the numerical value calculation unit 21 of the distributed processing unit 20 (vertex) that has received the instruction to shift to the next step executes the calculation / transmission processing f _{n + 1} of the next super step (n + 1).

As described above, according to the distributed synchronous processing system 1 and the distributed synchronous processing method according to the present embodiment, the delay in the processing speed as a whole system, the phase PH 3, which was a problem in the distributed synchronous processing system of the comparative example. Solve the problem of many synchronization waits without processing, that is, the problem of processing speed / efficiency, and reduce the influence of the distributed processing unit 20 (vertex) whose processing is extremely slow.
Also, the programmer does not have to consider overtaking or overwriting of processing between vertices, and can create a program of this system as a simple framework.

[Modification of this embodiment]
Next, a modification of the distributed synchronous processing system 1 according to the present embodiment will be described.
FIG. 11 is a diagram showing an entire configuration of a distributed synchronous processing system 1A according to a modification of the present embodiment.
In the distributed synchronous processing system 1 according to the present embodiment shown in FIG. 9, the management server 10 (master) determines whether or not the condition for adjacent synchronization is satisfied for each distributed processing unit 20 (vertex). . That is, it is a "master centralized type" in which the management server 10 determines the adjacent synchronization. On the other hand, the distributed synchronous processing system 1A shown in FIG. 11 does not include the management server 10 (master), and each distributed processing unit 20 (vertex) determines whether or not the adjacent synchronous condition is satisfied. The processing is performed autonomously in the processing server 30 (worker). That is, the distributed synchronous processing system 1A is characterized in performing adjacent synchronization in an autonomous distributed type (master-less type).
Specifically, in the distributed synchronous processing system 1A according to the modification of the present embodiment, the management server 10 (master) in the distributed synchronous processing system 1 shown in FIG. 9 is not provided, and each processing server 30A (worker , And instead, as shown in FIG. 11, the processing server 30A is provided with the adjacent synchronization vertex management unit 34 (adjacent synchronization and dispersion management unit).
In addition, about the structure provided with the same function as the structure shown in FIG. 9, the same name and code | symbol are attached | subjected and description is abbreviate | omitted.

The adjacent synchronization vertex management unit 34 (adjacent synchronization distribution management unit) monitors the distributed processing units 20 (vertex) belonging to itself, and when each distributed processing unit 20 (vertex) completes the calculation and transmission processing f _n , That is, when the phase PH1 (local calculation) and the phase PH2 (data exchange) are completed, whether or not the input messages (state of the input edge) from all the distributed processing units 20 (vertex) in contact with the input edge are aligned. Determine The adjacent synchronization vertex management unit 34 refers to the buffer (“M _{in, n} ” (for the current step) _in FIG. 4) of the input message (incoming message) of each distributed processing unit 20 (vertex) and performs the input. It is determined whether the messages (the state of the input edge) are aligned.
Also in the modification of the present embodiment, as in the present embodiment, there is a possibility that supersteps may be shifted between the respective vertices when viewed at a certain point in the entire processing. Therefore, when transmitting and receiving messages between vertices, management is performed for each super step without overwriting. Therefore, each vertex uses the state of the input edge acquired from the vertex that is in contact with the input edge that has shifted to the next super step prior to its own super step as “M _{in, n + m} ” in the buffer of the input message. Remember.

Then, when the input message (the state of the input edge) is complete, the adjacent synchronization vertex management unit 34 completes the acquisition of the calculation result necessary for the next calculation step from the adjacent distributed processing unit 20 (vertex). In this case, it is assumed that the condition of the adjacent synchronization in the present embodiment "the calculation and transmission processing f _{n of} all the vertices in contact with the own vertex and the input edge is completed". In this case, the adjacent synchronization vertex management unit 34 proceeds to the next super step (increases the super step by “+1”), the “next step shift instruction” is distributed and the calculation / transmission process f _n is completed. Output to the processing unit 20 (vertex).

  The adjacent synchronous vertex manager 34 receives an input from the distributed processing unit 20 (vertex) from any of the distributed processing units 20 (vertex) in contact with the distributed processing unit 20 (vertex) waiting in the inactive state at the input edge. When a message (the state of the input edge) is received, it is determined whether or not the input message (the state of the input edge) is aligned again, that is, whether or not the condition for adjacent synchronization is satisfied. Run.

  When the message processing unit 32 of the processing server 30A (worker) does not have data as the state of the input edge because the adjacent synchronization vertex management unit 34 determines the above-mentioned adjacent synchronization (for example, the data is "0"). Even when the phase PH1 (local calculation) is finished, the input message is transmitted to the distributed processing unit 20 (vertex) that is in contact with the output edge. Further, the message processing unit 32 of the processing server 30A (worker) immediately transmits the output message without buffering it, for the determination based on the condition of the adjacent synchronization of the adjacent synchronization vertex management unit 34.

<< Operation of the distributed synchronous processing system of the modified example >>
Next, the operation of the distributed synchronous processing system 1A according to the modification will be described.
FIG. 12 is a flowchart showing the process flow of the distributed synchronous processing system 1A according to the modification of the present embodiment.
Here, it is assumed that setting of individual calculation processing (vertex) necessary for target calculation processing and allocation of each calculation processing (vertex) to each processing server 30A (worker) have been completed in advance. explain. The setting of the individual calculation processing (vertex) and the allocation to each processing server 30A (worker), for example, include the management server of the entire system or the representative server in the processing server 30A. It may be performed in advance by preparing for

First, the adjacent synchronization vertex management unit 34 (adjacent synchronization distribution management unit) of the processing server 30A (worker) calculates about a certain distributed processing unit 20 (vertex) by monitoring the distributed processing unit 20 (vertex) belonging to itself. The completion of the transmission process f _n is detected (step S20).

  Subsequently, with respect to the distributed processing unit 20 (vertex), the adjacent synchronous vertex management unit 34 determines whether or not the input messages (states of the input edge) from all the distributed processing units 20 (vertex) in contact with the input edge are aligned. Is determined (step S21).

Here, when the adjacent synchronization vertex management unit 34 determines that the input message (the state of the input edge) is aligned (step S21 → Yes), the condition of the adjacent synchronization “all vertex and input edge contact with each other” It is satisfied that “vertex calculation / transmission processing f _n is completed”, and the process proceeds to step S24 (output of “next step shift instruction”) described later.

On the other hand, when the adjacent synchronization vertex management unit 34 determines that the input messages (the state of the input edge) are not aligned (step S21 → No), the “next step shift instruction” is not output. Therefore, the distributed processing unit 20 (vertex) which has completed the calculation / transmission process f _n becomes a standby in the inactive state (step S22).

  Subsequently, the adjacent synchronization vertex management unit 34 sends the distributed processing unit 20 (vertex), which is in contact with the input edge, to the distributed processing unit 20 (vertex) waiting in the inactive state. It is determined whether the vertex (vertex) receives the input message (the state of the input edge) (step S23). Then, if the distributed processing unit 20 (vertex) waiting in the inactive state has not received an input message (step S23 → No), the adjacent synchronization vertex management unit 34 waits until it is received. On the other hand, when the distributed processing unit 20 (vertex) waiting in the inactive state receives the input message (step S23 → Yes), the adjacent synchronization vertex manager 34 returns to step S21 triggered by that. .

On the other hand, when the condition for adjacent synchronization is satisfied (step S21 → Yes), the adjacent synchronization vertex management unit 34 causes the distributed processing unit 20 (vertex) to shift to the next super step in step S24. Outputs the "next step shift instruction". Accordingly, the numerical value calculation unit 21 of the distributed processing unit 20 (vertex) that has received the “next step shift instruction” executes the calculation / transmission process f _{n + 1} of the next super step (n + 1).

  As described above, according to the distributed synchronization processing system 1A and the distributed synchronization processing method according to the modification of the present embodiment, in addition to the effects of the distributed synchronization processing system 1 according to the present embodiment, an autonomous distributed type is adopted. Thus, the bottleneck of the management server 10 (master) can be avoided, and the processing speed / efficiency can be secured even in the large graph G.

1,1A Distributed synchronous processing system 10 Management server (master)
11 adjacent synchronization processing unit 20 distributed processing unit (vertex)
21 Numerical calculation unit 22 Message transmission and reception unit 30, 30A Processing server (worker)
31. Virtualization control unit 32. Message processing unit 33. Vertex management unit (distributed processing management unit)
34 Adjacent sync vertex manager (adjacent sync distribution manager)

Claims (4)

A management that assigns a plurality of processing servers performing processing in parallel, a plurality of distributed processing units operating on the processing server, and a plurality of distributed processing units required for the target calculation processing to the plurality of processing servers A distributed synchronous processing system having a server;
The processing server is
It detects the completion of calculation / transmission processing indicating the transmission processing to the distributed processing unit connected as the output destination of the calculation processing and the calculation result in the predetermined calculation step by the distributed processing unit, and completes the calculation / transmission processing Generating a completion report to be sent to the management server,
The distributed processing management unit is configured to receive, from the management server, an instruction to move to the next step, which is an instruction to shift to the next calculation step, and output the instruction to the distributed processing unit that has completed the calculation and transmission process.
The management server is
The distributed processing unit that has received the completion report and completed the calculation / transmission process is connected as an input source of the calculation result whether acquisition of the calculation result necessary in the next calculation step is completed. It determines based on whether the completion report from the distributed processing unit has been received, and transmits the next step shift instruction to the processing server that has sent the completion report when acquisition of the calculation result is completed. A distributed synchronization processing system comprising: an adjacent synchronization processing unit. A distributed synchronous processing system, comprising: a plurality of processing servers performing processing in parallel; and a plurality of distributed processing units operating on the processing server,
The processing server is
Detection of completion of calculation / transmission processing indicating transmission processing to the distributed processing unit connected as the output destination of the calculation processing and the calculation result in a predetermined calculation step by the distributed processing unit;
The distributed processing unit that has completed the calculation / transmission processing determines whether acquisition of the calculation result necessary in the next calculation step has been completed from the distributed processing unit connected as an input source of the calculation result, The adjacent synchronization and distribution management unit which outputs, to the distributed processing unit that has completed the calculation / transmission processing, an instruction to move to the next step, which is an instruction to shift to the next calculation step, when acquisition of the calculation result is completed. A distributed synchronous processing system comprising: A management that assigns a plurality of processing servers performing processing in parallel, a plurality of distributed processing units operating on the processing server, and a plurality of distributed processing units required for the target calculation processing to the plurality of processing servers A distributed synchronous processing method of a distributed synchronous processing system having a server;
The processing server is
It detects the completion of calculation / transmission processing indicating the transmission processing to the distributed processing unit connected as the output destination of the calculation processing and the calculation result in the predetermined calculation step by the distributed processing unit, and completes the calculation / transmission processing A procedure for generating a completion report to be sent and sending it to the management server;
Receiving from the management server an instruction to move to the next step, which is an instruction to shift to the next calculation step, and outputting the calculation / transmission processing to the distributed processing unit that has completed the processing;
The management server is
The distributed processing unit that has received the completion report and completed the calculation / transmission process is connected as an input source of the calculation result whether acquisition of the calculation result necessary in the next calculation step is completed. It determines based on whether the completion report from the distributed processing unit has been received, and transmits the next step shift instruction to the processing server that has sent the completion report when acquisition of the calculation result is completed. A distributed synchronous processing method characterized in performing a procedure. A distributed synchronous processing method of a distributed synchronous processing system, comprising: a plurality of processing servers performing processing in parallel; and a plurality of distributed processing units operating on the processing server,
The processing server is
A procedure for detecting completion of calculation / transmission processing indicating transmission processing to a distributed processing unit connected as a calculation processing and output destination of calculation results in a predetermined calculation step by the distributed processing unit;
The distributed processing unit that has completed the calculation / transmission processing determines whether acquisition of the calculation result necessary in the next calculation step has been completed from the distributed processing unit connected as an input source of the calculation result, Executing a procedure of outputting a next step shift instruction, which is an instruction to shift to the next calculation step, to the distributed processing unit that has completed the calculation / transmission process when the acquisition of the calculation result is completed Distributed synchronous processing method characterized by:

Images