JP2013069189A - Parallel distributed processing method and parallel distributed processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently move an entry without making a user be conscious of entry moving time in a parallel distributed processing system.SOLUTION: In a parallel distributed processing method and parallel distributed processing system 100 for performing parallel processing in a plurality of entries 22 which have a plurality of nodes, which are virtual calculation units operating in each node, and which are movable among the nodes, calculation processing is performed in each entry 22, and movement of entries 22 is effected by moving data of entries 22 whose nodes are to be changed to the change destination nodes of the entries 22.

Description

  The present invention relates to a technique of a parallel distributed processing method and a parallel distributed processing system for performing parallel processing in entries arranged in each node.

  One of the frameworks of parallel distributed processing is Pregel (see Non-Patent Document 1). This is a framework for executing processing based on the graph structure. In this pregel, each entry (or vertex: graph vertex) which is a logical calculation unit holds data, and messages are exchanged by tracing an edge (edge) which is a virtual connection relationship between entries. Do. When each entry receives a message from another entry, each entry performs a predetermined process (calculation process) based on the received message and the data held by itself, and updates the data held by itself. If necessary, a message is transmitted to another entry connected at the edge. If the identifier of the destination entry is known in advance, a message can be transmitted to an entry for which no edge is defined.

  Each entry is distributed in a plurality of nodes (logical computers). Each entry performs calculation and message transmission processing for each entry according to a BSP (Bulk Synchronous Parallel) model. That is, calculation processing and message transmission are performed synchronously and alternately between the entries. After the calculation process (parallel calculation process) is completed in all entries, each entry transmits a message to the next entry (message transmission). When transmission is completed for all messages, each entry performs parallel computation again. Therefore, a message transmitted as a result of a certain parallel calculation process can be used for the first time in the next parallel calculation process. Such a system can prevent data access conflicts and guarantee the uniqueness of processing (which always results in the same result).

  As another technique for equalizing the load on the node, there is a technique described in Patent Document 1. The distributed database management system according to this technique detects a processing load bias in each processor, and changes the data distribution arrangement configuration so as to equalize the processing load bias in each processor. In this technique, each node performs processing (calculation) on the data held by itself, so that the load on the nodes can be equalized by adjusting the arrangement of the data.

  As a technique for realizing such node load balancing and fault tolerance improvement in such a parallel distributed processing system, there is a technique described in Patent Document 2. The distributed parallel processing system in this technology creates a copy of each data at a node other than the node that originally owns the data. When a node stops failure or when the load on a node increases, the distributed parallel processing system is a process in which a node that holds a copy of the data owned by that node is executed as a proxy. It is.

JP-A-9-218858 JP 2001-101149 A

Pregel: a system for large-scale graph processing, SIGMOD 2010

  The technique described in Non-Patent Document 1 uses checkpoints as a countermeasure against node failures. Checkpoint is a method of recording a system state (entry state) at a certain moment in a storage device, reading data from the storage device when a failure occurs, and returning to the arrangement state at the time of checkpoint acquisition.

  Since the checkpoint needs to be acquired in a state where the entry arrangement is fixed, the system acquires the checkpoint immediately before the message transmission between the entries is completed and the parallel calculation processing is performed for each entry. There is a need. In other words, there is a problem that processing time only for obtaining checkpoints is required, and the overall processing time increases.

Further, the technique described in Non-Patent Document 1 determines the arrangement method of entries to nodes using a partition function in order to realize load balancing of nodes. The partition function is a function that, when an entry identifier is given as an argument, outputs the identifier of the node where the entry is to be placed. In this method, the relationship between entries and nodes is fixed.
On the other hand, in a system that can execute multiple jobs at the same time, other jobs may already be executed before the job is executed, or other jobs may be executed during job execution. There is a characteristic that the ratio of the number changes dynamically. In such a computer environment, there is a case where the load of each node cannot be equalized by the method of fixedly determining the entry arrangement as in the technique described in Non-Patent Document 1.

  In addition, as described above, the technique described in Patent Document 1 is a method of changing the data arrangement in a node in order to equalize the processing load of each node. If this technique is to be applied to Non-Patent Document 1, the technique described in Non-Patent Document 1 needs to change the data arrangement in a state where the entry arrangement is fixed. The processing time only for changing the data arrangement is required, and there is a problem that the processing time increases as a whole.

  As described above, the technology described in Patent Document 2 creates a copy of data in a different node in advance, and when a load on a certain node becomes high, a node having the copy data performs proxy processing. , To achieve load equalization of nodes. However, if such a technique is applied to the technique described in Non-Patent Document 1, the technique described in Non-Patent Document 1 needs to create duplicate data in a state where the entry arrangement is fixed. As in the case of checkpoint acquisition, processing time only for creating a copy of data is required, and the processing time increases as a whole.

  Therefore, a technique for efficiently moving entries in a parallel distributed processing system is required.

  One means in the present invention is a virtual distributed processing method having a plurality of nodes and operating in each node, and performing parallel processing in a plurality of entries that can move between the nodes. The node performs a calculation process on the entry and moves data of the entry whose node is changed to the node to which the entry is changed.

Furthermore, the other means of the present invention is a parallel distributed processing method for performing parallel processing in a plurality of entries which are a plurality of nodes and which operate in each node and can be moved between the nodes. And there is a first node and a second node that are in a relation between the backup source of the entry, the backup destination of the entry, and the first node performs a calculation process, and the calculation process The calculation result that is the result of the above is copied to the second node, the second node performs the calculation process, the calculation result that is the result of the calculation process is copied to the first node, and the first node When all the calculation results are obtained at the node, but the calculation result is not obtained at the second node, the first node obtains the calculation result at the second node. When the next calculation process is started, the first node and the second node perform the calculation process and copy the calculation result, and the first node The held calculation result is copied to the second node.
Other solutions will be described in the embodiments.

  According to the present invention, entries can be moved efficiently in a parallel distributed processing system.

It is a figure which shows the structural example of the parallel distributed processing system which concerns on this embodiment. It is a hardware block diagram of the computer which concerns on this embodiment. It is a figure which shows the model example of the parallel processing which concerns on this embodiment. It is a figure which shows the operation | movement model example of the parallel processing which concerns on this embodiment. It is a figure which shows the example of the partition table which concerns on this embodiment. It is a figure which shows the example of the data table which concerns on this embodiment. It is a figure which shows the example of the rearrangement of the entry which concerns on this embodiment. It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 1st Embodiment (the 1). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 1st Embodiment (the 2). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 1st Embodiment (the 3). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 1st Embodiment (the 4). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 1st Embodiment (the 5). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 1st Embodiment (the 6). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 2nd Embodiment (the 1). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 2nd Embodiment (the 2). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 2nd Embodiment (the 3). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 2nd Embodiment (the 4). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 2nd Embodiment (the 5). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 1). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 2). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 3). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 4). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 5). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 6). It is a figure which shows the operation | movement in the rearrangement of the entry which concerns on 3rd Embodiment (the 7). It is a figure which shows the operation | movement model example of the parallel processing which concerns on 4th Embodiment.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described in detail with reference to the drawings as appropriate.

(System Configuration)
FIG. 1 is a diagram illustrating a configuration example of a parallel distributed processing system according to the present embodiment.
The parallel distributed processing system 100 includes one master node 1, a plurality of slave nodes 2, and client nodes 3.
Each of the master node 1 and the slave node 2 is a logical computer, and one server may execute a plurality of nodes (master node 1 and slave node 2), or a single node may be executed. May be.
Each node is connected to each other by a virtual network 4.

The master node 1 is a logical computer that manages the operation status of the slave node 2, and actual calculation processing is performed on the slave node 2. The master node 1 can also serve as the slave node 2.
Each job executed by the parallel distributed processing system 100 is divided into a plurality of tasks 21 and executed.

The master node 1 executes a job manager 11 and a distributed file system 12.
The job manager 11 manages the job in the slave node 2 and the execution state of the task 21. Specifically, the correspondence between the job and the task 21, the correspondence between the task 21 and the slave node 2 that executes the task 21, the progress rate of each task 21, and the like are managed.
The distributed file system 12 will be described later.
The slave node 2 is a logical computer that performs actual calculation processing, and at least one task 21, a task manager 23, and a distributed file system 24 are executed.
In the task 21, a plurality of entries 22 are executed. The entry 22 can move between the task 21 and the nodes.
The task manager 23 manages the execution state of the task 21 being executed in each slave node 2 (such as the progress rate of the task 21 being executed). When the task manager 23 executes the task 21, it allocates and executes a plurality of threads for one task 21, thereby effectively utilizing the CPU resources even when the CPU (Central Processing Unit) 201 (FIG. 2) is multi-core. it can.

When the file size is large, one file may be divided and arranged on a plurality of nodes.
In such a case, the distributed file systems 12 and 24 have the same path name and all files from any node when the file body is distributed and arranged in a plurality of nodes as in the parallel distributed processing system 100. It is a system to make it accessible. When the distributed file systems 12 and 24 are used, if the data to be accessed is not stored in its own node but is stored in another node, the data is accessed via the distributed file system 12 or 24 of the other node. , Access the data.
Since the execution slave node of the generated task 21 is dynamically determined according to the load status of the slave node 2 group, it is not determined in advance which slave node 2 to actually execute the calculation process. By using the distributed file systems 12 and 24, when the task 21 is executed on any slave node 2, the task 21 can access the designated data position of the designated file.

  The client node 3 is a computer terminal that provides a user interface. When the user operates the client node 3 to notify the parallel distributed processing system 100 of a job execution instruction, the client node 3 transmits the job execution instruction to the job manager 11 of the master node 1. The job manager 11 divides the designated job into a plurality of tasks 21, distributes each task 21 to the task manager 23 of the slave node 2, and instructs the task manager 23 to execute it.

  When each slave node 2 executes a plurality of jobs simultaneously, a plurality of tasks 21 may be generated in one slave node 2 as shown in FIG.

  The job manager 11 and the task manager 23 are always resident in the parallel distributed processing system 100 and operate as service processes. That is, when the job is not being executed, the job manager 11 and the task manager 23 are in a standby state waiting for a job execution request from the user. The task 21 is a process that is dynamically generated every time the job is executed. When the job execution is started, the task manager 23 dynamically generates a process corresponding to the task 21. When different jobs influence each other, the performance and stability of job execution may be lowered. Therefore, the task manager 23 reduces the influence between jobs by separating different jobs into different processes.

(Hardware configuration)
FIG. 2 is a hardware configuration diagram of a computer that executes a master node and a slave node according to the present embodiment.
In the computer 200, a CPU 201, a LAN I / F (Interface) 202 connected to a LAN (Local Area Network) 250, an input / output I / F 203, a memory 204, and a disk I / F 205 are internally connected via a bus 206. Yes. A keyboard 210, a mouse 220, and a display 230 are connected to the input / output I / F 203, and the user uses these to instruct execution of a job. Connected to the disk I / F 205 is an OS (Operating System), a service process program operating on each node, and an HDD (Hard Disk Drive) 240 in which data to be analyzed for processing is recorded. . When the parallel distributed processing system 100 (FIG. 1) is started, each program is read from the disk device 109 into the memory 204, and the CPU 201 executes the program, thereby realizing the job manager 11, the task 21, the entry 22, the task manager 23, and the like. To do. Note that the computer 210 executing the master node 1 (FIG. 1) and the slave node 2 (FIG. 2) omits the keyboard 210, the mouse 220, and the display 230 because the user does not directly perform input / output operations. can do.

(Parallel processing model)
FIG. 3 is a diagram illustrating a model example of parallel processing according to the present embodiment. Hereinafter, the parallel distributed processing system 100 according to the present embodiment may be appropriately referred to as “the present system 100” or “the system 100”.
In the system 100, entries 22 (22a to 22c) holding data in a key-data value format exchange messages with each other. For example, the entry 22a holds a key “k1” and a data value “v11”. Similarly, the entry 22b holds a key “k2” and a data value “v21”, and the entry 22c holds a key “k3” and a data value “v31”.

The key is an identifier of the entry 22 and is created when the entry 22 is generated, and is not changed during execution of the job. The data value is the data itself held by the entry 22 and may be changed in the course of executing parallel processing.
When each entry 22 receives a message from another entry 22, it performs a calculation process based on the data value held by itself and the received message, updates the data value as necessary, and adds a new value to the other entry 22. Send a message. This process is implemented as a user program. Message transmission is performed by specifying the key of the destination entry. Each entry 22 is distributed and arranged in the slave nodes 2 in the system 100. Since the job manager 11 manages which slave node 2 each entry 22 is arranged in, the user program sends the message destination. There is no need to designate the slave node 2.

  Here, the data value does not have to be a single value, and may be a set of a plurality of values or a nested structure. Therefore, the entire data value may be updated in the data value update process, or a part of the data value may be updated.

As shown in FIG. 4, in this system 100, based on the BSP model, a series of parallel calculations and message transmission are executed in synchronization. In FIG. 4, reference numerals 22a to 22c are the same as those in FIG.
For example, in “Phase 1”, each entry 22 performs processing (calculation processing, data value update, message transmission, etc.) described in the user program. The processing of each entry 22 is executed in parallel and waits until the processing of all the entries 22 is completed (synchronization). When the processing is completed for all the entries 22, the system 100 shifts to “Phase 2”, and each entry 22 performs the processing described in the user program again. At this time, the data values of “v11”, “v21”, and “v31” are updated to “v12”, “v22”, and “v32”. As described above, the message transmitted by the entry 22 in “Phase 1” can be used for the first time in “Phase 2”. In the BSP model, each “Phase” is referred to as “superstep”. Hereinafter, in the present embodiment, this “superstep” is referred to as a “parallel processing period”. Processing performed in a distributed manner in each entry 22 during this parallel processing period is referred to as parallel processing (calculation processing, data value update, message transmission, etc.).

  The system 100 does not assume an edge between the entries 22 as in Non-Patent Document 1, but there is no essential difference regarding the presence or absence of an edge. That is, an edge may be set in the system 100. When the destination entry of the message is not known in the system 100, it is necessary for the source entry 22 to hold the key of the entry 22 as a data value. This is an edge from one entry 22 to another entry 22. This is because it is equivalent to creating

(Partition table)
FIG. 5 is a diagram illustrating an example of a partition table according to the present embodiment.
The partition table is a table for managing which task 21 or in which node the entry 22 having which key is executed.
As described with reference to FIG. 1, in this system 100, each task 21 holds a plurality of entries 22, and each task 21 processes a plurality of entries 22. A list of entries 22 held by each task 21 is managed in the partition table shown in FIG. Each record in the partition table corresponds to one task 21. The range indicated by “start key” and “end key” represents the key range of the entry 22 held by the corresponding task 21. The task ID (Identification) is an identifier of the task 21, and the node ID is an identifier of a node where the task 21 exists. The destination is the reception address of the corresponding node. The partition table is generated for each job, but the job manager 11 and the task manager 23 that manages all the tasks 21 related to the job hold the same partition table.

The job manager 11 creates a partition table (FIG. 5) at the start of the job, and instructs the task manager 23 to generate the task 21. The destination of the task 21 is composed of a set of the slave node 2 address and port number. When a plurality of jobs are executed at the same time, a plurality of task processes are executed on one slave node 2, and therefore it is necessary to assign a different reception port to each task process. Upon receiving the task 21 generation instruction, the task manager 23 assigns a reception port to the task process and generates a task process. When the generation of the task process is completed, the task manager 23 returns the destination (address and port number) of the task 21 to the job manager 11, and the job manager 11 has the destination of the corresponding task 21 in its own partition table (FIG. 5). Update.
When the generation for all the tasks 21 is completed, the job manager 11 transmits the partition table to the task manager 23 that manages all the tasks 21 related to the job being executed. In this way, the same partition table is distributed to all the task managers 23 when the task 21 is generated.

(Data table)
FIG. 6 is a diagram illustrating an example of a data table according to the present embodiment.
The data table is a table for managing which entry 22 holds what data value.
Each task manager 23 refers to the data table shown in FIG. 6 and manages the data value of the entry 22 held by itself. The data table stores the key (k) of the entry 22, “Dn” which is the data value of the result of the calculation process, “Br” which is the message received during the parallel processing, and the entry 22 holds before the start of the parallel processing period. “D”, which is a data value, and “Bu”, which is a message received in the previous parallel processing. “Bu” is a buffer for receiving a message transmitted from another entry in the entry. Here, “Dn” = “D” + “Bu” (as described above, the message “Br” received during the parallel processing cannot be used unless the next parallel processing period is reached). In practice, “Dn” often stores a difference from “D”. Further, the data table has data values “Db” which is a backup of data values and “Bb” which is a backup of messages. Among these, “Db” and “Bb” are used when the entry 22 is duplicated (described later in the third embodiment).
In FIG. 6, each column is blank, but actually data values are stored.

  The job manager 11 has a data table for all the entries 22 as shown in FIG. 6, but the task manager 23 may have a data table for all the entries 22 in the system 100, May have a data table related to the entry 22 corresponding to.

(Overview of parallel processing)
Here, an outline of parallel processing is described. Specific processing will be described later with reference to FIG.
In the parallel processing period, each entry 22 refers to the data table and executes a calculation process defined by the user using “D” and “Bu” of the entry 22 corresponding to itself. As described above, “D” is a data value held in the entry 22 at the start of the parallel processing period, and “Bu” is a message received in the previous parallel processing period. When updating the data value, the entry 22 stores the difference from the pre-update as the result “Dn” of the calculation process. When the parallel processing of all the entries 22 held by each task 21 is completed, “Dn” and “D” are merged (such as data combination), and the merged data value is replaced with a new “D”. To do. Each task 21 updates this new “D” as a data value after the end of the parallel processing period.

  Each task 21 transmits a message to the entry 22 held by the other task 21. At this time, the task 21 determines the destination slave node 2 with reference to the destination (FIG. 5) in the partition table held by the task manager 23. That is, the task 21 detects the entry 22 corresponding to the key of the destination entry 22 from the partition table at the time of message transmission. Then, the task 21 transmits a message to the detected destination 22 of the entry 22 (FIG. 5). The message to be transmitted includes the message body and the destination entry 22 key.

  The task 21 that has received the message refers to the data table (FIG. 6) held by itself and refers to the message received in “Br” (message received during parallel processing) of the entry 22 corresponding to the key included in the message. to add. There are two methods of adding messages to “Br”: (a) a method of simply adding a message, and (b) a method of performing an aggregation calculation. In the case of (a), “Br” is a list of messages finally received from the plurality of entries 22. Further, as an example of (b), there is a process in which the sum of messages received from all entries 22 is the result of “Br”. Since the sum satisfies the combining law and the exchange law, in the case of (b), each time a message is received, the task 21 calculates a data value obtained by adding the received message to the data value of “Br”, and the result is Update as a new "Br" value. Such an operation is an aggregation process, and any operation may be used as long as it satisfies the combining law and the exchange law. As examples of other aggregation processes, maximum, minimum, sum of squares, and the like may be used. Further, when performing the aggregation process, the transmission task 21 may transmit the aggregation result after previously collecting the messages to be transmitted to the same entry 22. As a result, the amount of network transfer can be reduced.

  In the message transmission process, a message may be received from another entry 22 during the parallel processing period, and the data value of “Br” cannot be determined. Therefore, when a certain parallel processing period ends, that is, when all the entries 22 finish parallel processing in a certain parallel processing period, the value of “Br” updated in the parallel processing period is determined. When a certain parallel processing period ends and “Br” is determined, the task 21 moves “Br” to “Bu” (data value held in the entry 22 before the start of the parallel processing period) during the parallel processing. As a result, the task 21 can use the received message in the next parallel processing period.

(Specific example of load balancing)
Next, a load equalization method in the system 100 will be described with reference to FIGS. 7 to 18 with reference to FIG. This is a method of equalizing the calculation load of the slave node 2 by adjusting the amount of data held by each slave node 2 according to the node performance. In order to do this, it is necessary to move the data value between the slave nodes 2. However, as described above, if the data movement is performed between the parallel processing period and the next parallel processing period in the BSP model, the data movement process is performed. Just need time for. Therefore, the system 100 according to this embodiment performs data movement in parallel during execution of the parallel processing period, thereby concealing the time of data movement processing in the processing time of the parallel processing period and reducing the overall processing time. Like that.

  For example, in the initial arrangement state, it is assumed that an entry 22 is assigned to each node (hereinafter, the slave node 2 is simply referred to as “node”) as shown in the upper part of FIG. The horizontal axis in FIG. 7 represents the key of the entry 22 held by each node. As shown in the upper example of FIG. 7, “Node 1” is “k1” to “k150”, “Node 2” is “k151” (not shown) to “k225”, and “Node 3” is “k226”. (Not shown) to the entry 22 corresponding to the keys “k300”.

  If the performance of each node is the same, in the diagram shown in the upper part of FIG. 7, the load related to “Node 1” is large. Therefore, the entry 22 is moved as shown in the lower part of FIG. Turn into. That is, the job manager 11 changes the key range assigned to each node as shown in the lower part of FIG. At this time, the job manager 11 moves the entry 22 in the key range “k101” to “k150” from “node 1” to “node 2”. Then, the job manager 11 moves the entry 22 in the key range “k201” to “k225” from “node 2” to “node 3”. In this way, “Node 1” has entries 22 from “k1” to “k100”, and “Node 2” has entries 22 from “k101” (not shown) to “k200”. “Node 3” has entries 22 from “k201” (not shown) to “k300”. That is, each node has 100 entries 22, and load balancing can be achieved if the performance of each node is the same. Here, to simplify the explanation, it is assumed that the performance of each node is the same. However, when the node performance is different, the distribution ratio of the entry 22 may be adjusted according to the ratio of the node performance. .

<< First Embodiment >>
Next, a first embodiment of a specific procedure for load equalization as shown in FIG. 7 will be described with reference to FIGS. Note that the configuration and hardware configuration of the system 100 in the first to third embodiments are the same as the configurations shown in FIGS.
8 to 25, the vertical axis indicates the key of the entry 22 held by each node (task 21). Each entry 22 has various data values corresponding to the horizontal direction. 8 to 25, a rectangle indicated by a broken line indicates that a data value does not exist, and a solid line rectangle indicates that a data value exists.

  In each of FIGS. 8 to 25, the subscript of the symbol shown in the rectangle indicates the version of the data value. For example, in FIG. 8, “Node 1” has “D”, which is the data value held in the entry 22 before the start of synchronization, and all the ranges for which it is responsible for the message “Bu” received in the previous parallel processing period. Hold against the key. “D” is described as “D1”, and “Bu” is described as “B1”. The subscript “1” is a version. Although details will be described later, in “Node 1” in FIG. 9, the parallel processing of the entry 22 is completed for the keys in the range of “k100” to “k85”, and “d2” is added to the portion of “Dn” that is the difference between the data values. Is described. Here, although the subscript is “2”, this data value is generated based on the data values of the version “1” (“D1” and “B1”), so the version has increased to “2”. It depends.

First, each task 21 executes the parallel processing period at least once before redistributing data values, and measures the execution time of the parallel processing period in each node. When the parallel processing period ends, the measurement results are collected from each task 21 to the job manager 11, and the job manager 11 calculates the ratio of keys to be assigned to each node based on the measurement results.
Specifically, when the distribution ratio of the existing entry 22 is x1: x2: x3 and the processing time ratio is t1: t2: t3, the job manager 11 determines that the distribution ratio of the entry 22 is x1 / t1. By adjusting the ratio to x2 / t2: x3 / t3, the ratio of the next processing time can theoretically be 1: 1: 1. The job manager 11 calculates the distribution ratio of such entries 22 with reference to the partition table (FIG. 5).

As shown in the lower part of FIG. 7, the assignment of the entry 22 to each node calculated in this way is “k1” to “k100” for “node 1” and “k101” to “node 2”. It is assumed that “k201” to “k300” are obtained for “k200” and “node 3”. The job manager 11 distributes the range of keys after rearrangement (that is, the partition table in FIG. 5) to each node, and requests each task 21 to start the next parallel processing period.
Hereinafter, a specific procedure for rearranging the entries 22 from the upper arrangement state of FIG. 7 to the lower arrangement state of FIG. 7 will be described.

(Fig. 8)
The entry 22 held by each node at the start of the parallel processing period is as shown in FIG. Here, the arrangement state of the entry 22 in FIG. 8 is the same arrangement state as the upper stage of FIG. The task 21 of each node determines the key range to be transmitted to other nodes based on the rearranged key range distributed from the job manager 11. Specifically, “Node 1” transmits “k101” to “k150” to “Node 2”, and “Node 2” determines to transmit “k201” to “k225” to “Node 3”. . As a result, after the end of this parallel processing period, “node 1” has entries 22 of “k1” to “k100”, “node 2” has entries 22 of “k101” to “k200”, “Node 3” has entries 22 “k201” to “k300”.

(Fig. 9)
FIG. 9 is a diagram illustrating an entry arrangement state in each node when a certain time has elapsed since the start of the parallel processing period.
The task 21 of each node starts executing parallel processing of the entry 22 in the key range that it is in charge of after the relocation, and the data value (“D” and “Bu”) of the relocation target entry 22 is set to the destination node. Set to task 21.
For example, “Node 2” is responsible for keys in the range of “k101” to “k200” after rearrangement, but does not hold keys in the range of “k101” to “k150” (currently “ Node 1 ”). Accordingly, “node 2” starts execution of the parallel processing of the entry 22 from “k200”, which is the largest key held by itself after relocation, and is still held at the present time. The parallel processing of the entry 22 is advanced in the direction of decreasing (“k200” → “k151”). In FIGS. 9 to 13, the direction in which the parallel processing proceeds is indicated by black arrows.

  At the time of FIG. 9, “Node 2” is proceeding with parallel processing from “k200” to “k185”, and the data value of “Dn” in the entry 22 in the range of “k200” to “k185” as a result of the calculation processing ("D2") is stored.

  In parallel with such calculation processing, each node copies the data values of “D” and “Bu” to the rearranged nodes. At this time, the data values are processed according to the procedure described below. Make a copy of For example, in anticipation of the parallel processing operation of “node 2” as described above, “node 1” preferentially copies to “node 2” from the data value close to the key boundary before relocation with “node 2”. . That is, since “node 2” performs parallel processing of the entry 22 in the direction of decreasing key value from “k200” in order, “node 1” sets the data value in the range of “k101” to “k150” to “node 2”. When copying to “”, copying starts from “k150” which is the key boundary before rearrangement. Then, “node 1” sequentially copies data values from “k150” to “k101” in a direction in which the key value decreases (“k150” → “k101”). 9 to 13, the copying direction is indicated by hatching arrows, and the copying direction is indicated by white arrows. As a result, the processing of the entry 22 in “node 2” can be executed with as little delay as possible.

In FIG. 9, “node 1” copies data values from “k150” to “k130”, but “node 2” receives data values only from “k150” to “k135”. Absent. This is due to a transmission / reception time lag.
In all the nodes, the execution order of the entry 22 and the inter-node transmission order of the entry 22 are executed based on such rules.
Since the calculation process of the entry 22 and the copy of the data value of the entry 22 are performed asynchronously, the copying of all “Dn” and “D” corresponding to the key for which the calculation process of the entry 22 has been completed is completed. Do not mean. Note that the calculation processing of the entry 22 and the copying of the data value of the entry 22 may be performed synchronously.

The arrangement state of entries in “Dn”, “D”, and “Bu” in FIG. 9 is organized as follows.
Node 1: Computation processing has been completed for “k100” to “k85” (“d2”). Data values (“D1”, “B1”) from “k150” to “k130” have been copied to “node 2”.
Node 2: Calculation processing from “k200” to “k185” has been completed (“d2”). Data values (“D1”, “B1”) from “k1225” to “k205” have been copied to “Node 3”. Data values (“D1”, “B1”) from “k150” to “k135” have been received from “node 1”.
Node 3: Calculation processing from “k300” to “k285” has been completed (“d2”). Data values (“D1”, “B1”) from “k225” to “k210” have been received from “node 2”.

A message transmitted from each entry 22 to another entry 22 during the parallel processing period is transmitted based on the entry arrangement after the rearrangement. Therefore, each node receives a message corresponding to a key in a range that it is responsible for after the rearrangement.
For example, since “node 1” is responsible for the entry 22 in the key range “k1” to “k100” after the rearrangement, “Br” receives and holds only the data values in this range. In FIG. 9, in “Node 1”, “Br” corresponding to the key range of “k1” to “k100” is shown as a solid-line rectangle. However, since the parallel processing period is being executed and the value of “Br” has not been determined, in FIG. 9, “Br” does not describe a data value with a version number (for the same reason in the following figures) The version number is not written in “Br” before confirmation. Further, after the rearrangement, since “node 1” is not responsible for the key range entry 22 of “k101” to “k150” in the first place, the broken line display of the corresponding “Br” portion is not performed in FIG.
For the same reason, “Br” corresponding to the key range of “k101” to “k200” in “Node 2”, and “Br” corresponding to the key range of “k201” to “k300” in “Node 3”. Shown as a solid rectangle. In the subsequent drawings, “Br” is indicated in the same manner, and therefore description of “Br” is omitted in FIGS. 10 and 11.

(Fig. 10)
FIG. 10 is a diagram illustrating an entry arrangement state in each node when a further time elapses from FIG. 9.
The arrangement state of entries in each node is as follows.
Node 1: Calculation processing from “k100” to “k55” has been completed (“d2”). Data values (“D1”, “B1”) from “k150” to “k105” have been copied to “node 2”.
Node 2: Calculation processing from “k200” to “k155” has been completed (“d2”). Data values (“D1”, “B1”) from “k1225” to “k201” have been copied to “Node 3”. Data values (“D1”, “B1”) from “k150” to “k110” have been received from “node 1”.
Node 3: Calculation processing from “k300” to “k255” has been completed (“d2”). Data values (“D1”, “B1”) from “k225” to “k201” have been received from “node 2”.

  At the time shown in FIG. 10, the copying of the data values (“D” and “Bu”) from “Node 2” to “Node 3” is completed, but the data from “Node 1” to “Node 2” is complete. The value copy is incomplete.

(Fig. 11)
FIG. 11 is a diagram showing an entry arrangement state in each node when a further time elapses from FIG.
The arrangement state of entries in each node is as follows.
Node 1: Computation processing has been completed for “k100” to “k15” (“d2”). Data values (“D1”, “B1”) from “k150” to “k101” have been copied to “node 2”.
Node 2: The calculation processing from “k200” to “k115” has been completed (“d2”). Data values (“D1”, “B1”) from “k1225” to “k201” have been copied to “Node 3”. Data values (“D1”, “B1”) from “k150” to “k101” have been received from “node 1”.
Node 3: Calculation processing from “k300” to “k2215” has been completed (“d2”). Data values (“D1”, “B1”) from “k225” to “k201” have been received from “node 2”.

  At the time shown in FIG. 11, in addition to copying data values (“D” and “Bu”) from “Node 2” to “Node 3”, copying data values from “Node 1” to “Node 2” is also performed. Completed. However, the calculation process (“d2”) at each node is incomplete.

(Fig. 12)
FIG. 12 is a diagram illustrating an entry arrangement state in each node when a further time elapses from FIG. 11.
In FIG. 12, the copying of the data value and the calculation process at each node are completed. At the time of FIG. 12, the parallel processing in the parallel processing period ends, and the state shifts to the synchronous arrangement state in the BSP model.
Since the state of “Br” has been determined because the parallel processing period has ended, “B2” and a value with a subscript of the version number are written inside the rectangle of “Br” in each node.
In addition, “k150” to “k101” in “node 1” and “k225” to “k201” in “node 2” are not subjected to calculation processing in the corresponding node, and thus are broken-line rectangles having no data value. ing.

(Fig. 13)
FIG. 13 is a diagram showing an entry arrangement state in each node when a further time elapses from FIG.
FIG. 13 shows the arrangement state of entries during synchronization. During this synchronization, the task 21 of each node merges the value of “Dn” with “D” (“d2” + “D1”). Dn ”is cleared. Also, the task 21 of each node moves “Br” to “Bu” and clears “Br”.
In FIG. 13, the value of “D” is “D2”, but “D1”, which is the data value of the previous version, is merged with the difference “d2” to the new version data value to “D2”. (“D2” + “D1” = “D2”). Finally, the task 21 of each node updates the start key and end key of the partition table (FIG. 5) held by itself to values after the rearrangement.

(Effect of 1st Embodiment)
As described above, the system 100 conceals the relocation processing time of the entry 22 by copying (transmitting) the data value while each node performs processing on the relocated entry 22 during the parallel processing period. And the overall processing time can be shortened.
That is, in the parallel processing period, each node performs the calculation process of the entry 22 and copies the data value according to the entry 22 after the rearrangement. It can be hidden, and the overall processing time can be shortened. Note that the copy of data values has a significantly lower usage rate of the CPU 201 (FIG. 2) and the memory 204 (FIG. 2) than the calculation process, so the influence of the copy of the data values on the parallel processing time is small.

<< Second Embodiment >>
Next, a second embodiment of a specific procedure for load equalization as shown in FIG. 7 will be described with reference to FIGS.
In the technique of the first embodiment, “D” and “Bu” are transmitted between the nodes in the relocation processing of the entry 22, but when the amount of data to be copied is large, a long time is required for the data value copy processing. There is a problem that the effect of reducing the overall processing time is reduced. Therefore, the system 100 according to the second embodiment will explain a method of transmitting “Dn” and “D” instead of “D” and “Bu” in the rearrangement process.
This method is effective when the data amount of “Dn” is sufficiently smaller than “Bu”.

  Since the difference from the first embodiment is the relocation processing of the entry 22, only the relocation processing of the entry 22 will be described below. Similar to the first embodiment, a case where the rearrangement of the entries 22 is performed as shown in FIG. 7 will be described as an example. The process up to the process of determining the key range of the entry 22 to be copied to the other node by the task 21 of each node is the same as that in the first embodiment, and the description thereof is omitted.

(Fig. 14)
FIG. 14 is a diagram illustrating an entry arrangement state in each node at the start of the parallel processing period.
FIG. 14 is the same as FIG.

(Fig. 15)
When the parallel processing period starts, each node performs processing for the entry 22 before the rearrangement, and copies “Dn” and “D” as a result of the calculation processing to the rearrangement destination for the entry 22 after the rearrangement. .
For example, “Node 1” holds an entry 22 in the key range “k1” to “k150” at the start of synchronization, and among these, data corresponding to the entry 22 in the key range “k101” to “k150” The value needs to be sent to “Node 2”. Depending on the state of the network 4 (FIG. 1), it may take time to transmit the data value, so the system 100 performs parallel processing of the transmission target entries 22 (“k101” to “k150” in “node 1”). Is executed with priority, and the copy processing of the data value in this key range is started. Therefore, “Node 1” executes parallel processing in the entry 22 in the key range “k101” to “k150” before the entry 22 (“k1” to “k100”) remaining in “Node 1” after the rearrangement. Then, “Dn” (“d2”), which is the calculation processing result, is copied to “node 2” in order from the one that was created.

In order to perform such control, “node 1” starts parallel processing of entry 22 from “k150” and performs parallel processing in a direction in which the key value decreases (“k150” → “k1”). That is, “Node 1” starts parallel processing of the entry 22 from a data value close to the key boundary (here, “k150”) between the nodes before rearrangement.
In FIG. 15, “Node 1” has completed the calculation processing of the entry 22 from “k150” to “k125”, and the creation of the corresponding “Dn” (“d2”) has been completed. Since the calculation processing of the entry 22 and the copy of the data value of the entry 22 are performed asynchronously, the copying of all “Dn” and “D” corresponding to the key for which the calculation processing of the entry 22 has been completed is not complete. Absent. Note that the calculation processing of the entry 22 and the copying of the data value of the entry 22 may be performed synchronously.
In FIG. 15, copying of “Dn” and “D” in the range of “k150” to “k130” among the data values of “k150” to “k125” for which calculation processing has been completed in “node 1” is completed. (See “Node 2”).

In the second embodiment, “Dn” and “D” corresponding to the same entry 22 are paired and copied to the transmission destination node. However, it is not always necessary to copy “Dn” and “D” as a pair. If there is a margin in the bandwidth of the network 4, all “D” must be copied before copying of all “Dn” is completed. You may complete the copy.
In FIGS. 15 and 16, the direction of calculation and copying is indicated by a black arrow, and the direction of copying is indicated by a white arrow.

  Similarly, in FIG. 15, since it is necessary to rearrange the entries 22 in the range of “k225” to “k201” to “node 3”, “node 2” is changed from “k225” to “k225” before the rearrangement. The parallel processing of the entry 22 is started in the direction of “→ K201”, and the calculation processing is finished up to “k205”. “Node 3” has received “Dn” (“d2”) and “D” (“D1”) in the range of “k225” to “k210”.

The arrangement state of entries in “Dn” and “D” in FIG. 15 is organized as follows.
Node 1: Calculation processing from “k150” to “k125” has been completed (“d2”). Data values (“d2”, “D1”) from “k150” to “k130” have been copied to “node 2” (see “Dn”, “D” of “node 2”).
Node 2: Calculation processing has been completed for “k225” to “k205” (“d2”). Data values from “k225” to “k210” (“d2”, “D1”) have been copied to “node 3” (see “Dn”, “D” of “node 3”). Data values (“d2”, “D1”) from “k150” to “k130” have been received from “node 1”.
Node 3: Calculation processing from “k300” to “k280” has been completed (“d2”). Data values (“d2”, “D1”) from “k225” to “k210” have been received from “node 2”.

In the second embodiment, since the result of calculation processing (“Dn”) is already sent by the destination node, the destination node performs calculation processing using “D” and “Bu” again. There is no need. Therefore, “Bu” is not copied in each node (the same applies to FIGS. 16 and 17).
Further, “Br” is the same as that in the first embodiment, and thus the description thereof is omitted.

(Fig. 16)
FIG. 16 is a diagram showing an entry arrangement state in each node when a further time elapses from FIG. As described above, the direction of calculation and copying is indicated by a black arrow, and the direction of copying is indicated by a white arrow.
The arrangement state of entries in each node is as follows.
Node 1: Computation processing has been completed for “k150” to “k95” (“d2”). Data values (“d2”, “D1”) from “k150” to “k101” have been copied to “node 2” (see “Dn”, “D” of “node 2”).
Node 2: Computation processing has been completed for “k225” to “k170” (“d2”). Data values (“d2”, “D1”) from “k225” to “k201” have already been copied to “node 3” (see “Dn”, “D” of “node 3”). Data values (“d2”, “D1”) from “k150” to “k101” have been received from “node 1”.
Node 3: Calculation processing from “k300” to “k226” has been completed (“d2”). Data values (“d2”, “D1”) from “k225” to “k201” have been received from “node 2”.

  In FIG. 16, the copying of the data value at each node is completed, but the calculation processing is incomplete at “node 1” and “node 2” (the calculation processing is also completed at “node 3”).

(Fig. 17)
FIG. 17 is a diagram illustrating an entry arrangement state in each node when a further time elapses from FIG.
The arrangement state of entries in each node is as follows.
Node 1: Calculation processing from “k150” to “k1” has been completed (“d2”). Data values (“d2”, “D1”) from “k150” to “k101” have been copied to “node 2” (see “Dn”, “D” of “node 2”).
Node 2: Calculation processing has been completed for “k225” to “k151” (“d2”). Data values (“d2”, “D1”) from “k225” to “k201” have already been copied to “node 3” (see “Dn”, “D” of “node 3”). Data values (“d2”, “D1”) from “k150” to “k101” have been received from “node 1”.
Node 3: Calculation processing from “k300” to “k226” has been completed (“d2”). Data values (“d2”, “D1”) from “k225” to “k201” have been received from “node 2”.

  At the time of FIG. 17, the calculation process and data value copying at each node have been completed. Then, at the time of FIG. 17, the parallel processing in the parallel processing period is completed, and the process goes to the synchronized state in the BSP model. Here, as in the first embodiment, since the state of “Br” has been determined by the end of the parallel processing period, “B2” and the version number are appended to the inside of the rectangle of “Br”. The value of is described.

(Fig. 18)
FIG. 18 is a diagram illustrating an entry arrangement state in each node when a further time elapses from FIG.
FIG. 18 shows the arrangement state of entries during synchronization. During this synchronization, the task 21 of each node merges the value of “Dn” into “D” for the entry 22 after relocation (“ d2 "+" D1 "=" D2 ") and Dn is cleared. Then, the task 21 of each node moves “Br” to “Bu” and clears “Br”. Note that the version number in “D” is the same as in the first embodiment, and a description thereof will be omitted.
Finally, the task 21 of each node updates the start key and end key of the partition table held by itself to values after rearrangement.

(Effect of 2nd Embodiment)
According to the second embodiment, instead of “Bu”, “Dn” (this value is often a difference and thus the amount of data is small) is copied (transmitted) to the destination node, thereby transmitting data. The amount can be reduced.

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described with reference to FIGS.
The system 100 according to the third embodiment aims to conceal the data movement time while maintaining fault tolerance.
The technique described in Patent Document 2 can be cited as a technique for achieving both fault tolerance and node load equalization, but as described above, extra time is required for data value duplication processing. There is a problem that processing time increases. In response to such a problem, the system 100 according to the third embodiment reduces the overall processing time by performing data value replication while performing parallel processing in the entry 22.

  Since the difference between the third embodiment, the first embodiment, and the second embodiment is the processing of the entry 22 and the data replication processing in the parallel processing period, this portion will be mainly described. In the third embodiment, the data values of all entries 22 are held by two nodes. The node that mainly manages the data value and processes the entry 22 is called a primary node (simply referred to as “primary” as appropriate: a first node), holds a copy of the data value, and stores the entry 22 when the load is small. A node that performs processing on behalf (that is, a backup node) is referred to as a replica node (simply referred to as “replica” as appropriate: a second node). In the third embodiment, since the replica also serves as a primary for another key range, it is assumed that the calculation processing capability (proxy execution capability) is lower than that of the primary. However, the replica may be a dedicated node and may have a calculation processing capability equivalent to that of the primary.

(Fig. 19)
FIG. 19 is a diagram showing the arrangement state of entries in the primary and replica at the start of the parallel processing period.
As shown in FIG. 19, the data value (“D”, “Bu”) of entry 22 for key ranges “k1” to “k100” is duplicated in the primary and replica.

(Fig. 20)
FIG. 20 is a diagram illustrating an entry arrangement state in each node when a certain time has elapsed since the start of the parallel processing period.
When the parallel processing period is started and the processing of the entry 22 is started, the primary starts the parallel processing of the entry 22 from “k1” having the smallest key value, and the direction of “k100” (“k1” → “k100”). Continue parallel processing. The primary copies “Dn” (“d1”), which is the calculation processing result, to the replica in order from the entry 22 where the calculation processing is completed. Since the calculation process of entry 22 and the copy of “Dn” are performed asynchronously, the key of entry 22 being calculated does not match the key corresponding to Dn being copied. The calculation process of the entry 22 and the copy of “Dn” may be performed synchronously. In FIG. 19, the calculation process in the entry 22 from “k1” to “k40” is completed, and the copy of “Dn” from “k1” to “k20” is completed (refer to replica).
In FIG. 20 to FIG. 23, the calculation and copying directions are indicated by solid arrows, and the copying directions are indicated by white arrows.

If the replica has a sufficient load and can be executed by proxy, it takes over the processing of the entry 22 performed by the primary. In the example of FIG. 20, the replica performs the processing of the entry 22 in the reverse order to the primary so that the execution and processing of the primary do not overlap. That is, the processing of the entry 22 is started from “k100” having the maximum key value, and proceeds in the direction of “k1” (“k100” → “k1”). The replica copies “Dn” (“d1”), which is the result of the calculation process of the entry 22 executed by proxy, to the primary. In the example of FIG. 19, the replica completes the calculation processing in the entries 22 from “k100” to “k85”, and among these, copying of the data value (“Dn”) is completed for the entries 22 from “k100” to “k90” (Primary reference).
As long as the parallel processing of the primary and the replica does not overlap, the direction of the parallel processing is not limited to the direction shown in FIG. For example, the respective directions may be reversed, or parallel processing may proceed in the opposite directions from any entry 22.

  In this way, the primary and the replica share the processing of the entry 22, but the other entry 22 performing the parallel processing transmits whether the node that executed the parallel processing of the entry 22 is the primary or the replica. Messages are sent to the primary. Accordingly, in FIGS. 19 to 25, “Br” does not exist in the replica. However, messages to other entries 22 are transmitted from the primary and replica. That is, the replica transmits a message for the entry 22 calculated by itself.

The arrangement of entries in the primary and replica at the time of FIG. 20 is organized as follows.
Primary: The calculation processing from “k1” to “k40” has been completed (“d1”). Data values from “k1” to “k20” have been copied to the replica (see “Dn” of the replica). Data values “k100” to “k90” have been received from the replica (“d1”).
Replica: Calculation processing from “k100” to “k85” has been completed (“d1”). Data values from “k100” to “k90” have been copied to the primary (see “Dn” of the primary). Data values “k1” to “k20” have been received from the primary (“d1”).

(Fig. 21)
FIG. 21 is a diagram showing an entry arrangement state in each node when a further time elapses from FIG.
The layout of entries in the primary and replica is as follows.
Primary: The calculation processing from “k1” to “k80” has been completed (“d1”). Data values from “k1” to “k50” have been copied to the replica (see “Dn” of the replica). Data values “k100” to “k80” have been received from the replica (“d1”).
Replica: Calculation processing from “k100” to “k70” has been completed (“d1”). Data values from “k100” to “k80” have been copied to the primary (see “Dn” of the primary). Data values “k1” to “k50” have been received from the primary (“d1”).

It is assumed that the parallel processing period has ended at the time of FIG.
Then, as shown in FIG. 21, the primary holds the calculation processing results (“d1”) for all the entries 22. The point where the entry 22 where the primary itself performed the calculation process and the calculation process result received from the replica match is “k80”.
On the other hand, at the time of FIG. 21, the proxy execution started from “k100” has progressed to “k70”, and data values “k100” to “k80” are copied to the primary.

That is, for the entry 22 from “k70” to “k80”, calculation processing is performed in both the primary and replica. As described above, since the message is transmitted for the entry 22 calculated by both the primary and the replica, the message is transmitted from both the primary and the replica regarding the entry 22 of “k70” to “k80”. . As a result, the primary in the other node receives a message transmitted from both the primary and the replica. If the same message is applied twice to “Br”, the value of “Br” becomes invalid, and it is necessary for the primary of each node to detect duplicate reception of messages. For example, the primary of each node can detect duplicate reception by holding a set of keys of transmission source entries of received messages. In other words, when the primary of each node receives a message, it holds the key of the sender of the message. Each time a primary message is received, the primary compares the message transmission source key with the held key to determine whether or not duplicate messages have been received.
When it is detected that a message has been received twice, the primary performs processing such as discarding a message received later.

(Fig. 21 → Fig. 22)
Here, as described above, the parallel processing period ends at the time of FIG. 21, and the BSP model is in a synchronized state.
During this time, the replica merges “Dn” held by itself into “D”, and “D” and “Bu” of the key range (“k51” to “k69”) not holding “Dn” Back up to “Db” and “Bb” of the replica itself.

  Further, the primary merges “Dn” into “D” in all the entries 22 and moves “Br” to “Bu”. Further, the primary backs up Dn (“d1”) of “k51” to “k69” that the replica does not hold to “Db” of the primary. Further, the primary copies “Br” in all the entries 22 together to the replica. When the replica receives “Br” from the primary, the replica stores the received “Br” in “Bu”.

In this way, for the data values (“D”, “Bu”) corresponding to the key ranges “k1” to “k50” and “k70” to “k100”, the data values after execution of the parallel processing period are It will be duplicated.
For the other key ranges, ie, “k51” to “k69”, the primary holds the data values (“D” (“D1”), “Bu” (“B1”)) after execution of the parallel processing period). Yes. However, the replica does not hold “D” for this key range (“k51 to“ k69 ”). However, since the replica holds data values (“D0”, “B0”) before execution of the parallel processing period in “Db” and “Bb” in this key range (“k51” to “k69”), By recalculating using this, it is possible to create “D” (“D1”) after execution of the parallel processing period (“D0” + “B0” = “d1”, “d1” + “D0”) = "D1").

  That is, the replica is equivalent to having the data values ("D1", "B1") after execution of the parallel processing period, and it can be said that the data values are duplicated without any problem.

  The result of completing the above processing is the state shown in FIG.

Even if the parallel processing between the primary and the replica is completed, the calculation processing in the other entry 22 may not be completed. In such a case, since it is necessary to wait for the calculation processing of the other entry 22 to be completed, the primary replica may perform the following processing, but in the present embodiment, the following processing is performed. The explanation of the processing will proceed as if it did not exist.
That is, using this waiting time, the primary may copy “Db” (“d1”) of “k51” to “k69” to the replica at appropriate times.
When the replica receives “d1”, it merges with “Db” (“D0”) and stores the merge result in “D” (“D1”). Then, the replica may sequentially clear “Db” and “Bb” corresponding to the keys that have been updated in parallel with the merge and storage of the merge result. In the primary, “Db” corresponding to the keys for which “Db” (“d1”) has been copied may be sequentially cleared.

(Fig. 22 → Fig. 23)
In the next parallel processing period, as a direction to advance the key when the calculation processing of the entry 22 is executed by proxy in the replica, (a) “k1” → “k100”, and (b) “k100” → “k1” 2 There are two options. In this case, it is necessary to set in which direction the proxy execution of the calculation process is performed as shown below.

  Since the replica does not hold the data value (“D”) of the entry 22 in the intermediate key range “k51” to “k69” at the time of FIG. 22, the calculation process in the entry 22 in this range is not executed. Can not. Therefore, in order to enable the replica to execute proxy execution for as many entries 22 as possible, the proxy execution is started from the side where the number of entries 22 is large in the key range where “D” exists. If the arrangement state is as shown in FIG. 22 at the start of the next parallel processing period, the replica of the key range of “k1” to “k50” is more than the key range of “k70” to “k100”. The data value “D” (“D1”) is held. Therefore, the replica sets the proxy execution direction so that the proxy execution proceeds in the direction of “k1” → “k100”.

  Since the replica holds the entry 22 that it manages as a primary, the proxy execution is not started immediately after the parallel processing period starts. The replica determines the direction of proxy execution at the start of the parallel processing period, and actual proxy execution is performed when its own load is low.

When the replica determines the proxy execution direction, the primary performs parallel processing of the entry 22 in the reverse order (ie, “k100” → “k1”). As a result, it is possible to prevent the entry 22 performing parallel processing from overlapping between the primary and the replica.
In addition, the primary sends the remainder of “Db” (“d1”) to the replica, but copies “Db” to be in the same direction as the proxy execution of the replica so that the proxy execution can be continued as much as possible. To do.
That is, in this embodiment, since the replica performs proxy execution in the direction of “k1” → “k100”, the primary performs “Db” (“d1”) in the direction of “k51” → “k69” that is the same direction. Send.

(Fig. 23)
FIG. 23 is a diagram illustrating an entry arrangement state when a predetermined time has elapsed since the start of the parallel processing period.
The entry arrangement state at the time of FIG. 23 is as follows.
Primary: Computation processing has been completed from “k100” to “k40” (“d2”). Data values (“d2”) from “k100” to “k55” have been copied to the replica (see “Dn” of the replica). Data values (“d2”) from “k1” to “k10” have been received from the replica (“d2”). “Db” (“d1”) from “k51” to “k65” has already been transmitted to the replica.
Replica: Calculation processing from “k1” to “k10” has been completed. Data values (“d2”) from “k1” to “k10” have been copied to the primary (see “Dn” of the primary). Data values (“d2”) from “k100” to “k55” have been received from the primary. “D” (“D1”) has been generated by receiving data values (“d1”) from “k51” to “k65” from the primary and merging them with their own “Db” (“D0”) .
Here, the key range copied from the replica to the primary is the same as “k1” to “k10” because the calculation / copying is performed when the replica has sufficient load.

(Fig. 24)
FIG. 24 is a diagram showing an entry arrangement state immediately after the time elapses from the time of FIG. 23 and the parallel processing period ends.
The entry arrangement state at the time of FIG. 24 is as follows.
Primary: Calculation processing has been completed from “k100” to “k30”. Data values (“d2”) from “k100” to “k50” have been copied to the replica (see “Dn” of the replica). Data values (“d2”) from “k1” to “k30” have been received from the replica (thus, all data values of “Dn” are retained). “Db” (“d1”) from “k51” to “k69” has been transmitted to the replica.
Replica: Calculation processing from “k1” to “k30” has been completed. Data values (“d2”) from “k1” to “k30” have been copied to the primary (see “Dn” of the primary). Data values (“d2”) from “k100” to “k50” have been received from the primary. “D” (“D1”) is generated by receiving data values (“d1”) from “k51” to “k69” from the primary and merging it with its own “Db” (“D0”) (Thus, all data values of “D” (“D1”) are retained).

  Here, although the calculation processing from “k100” to “k30” is completed at the primary, the replica receives only the data values from “k100” to “k50”. This is because it is lower than the primary.

  At the time of FIG. 24, the replica can hold all the data values “D” (“D1”) in the key range “k51” to “k69” that have not been processed in the previous parallel processing period. However, the calculation process is not in time for the data value “Dn” (“d2”) in the key range “k31” to “k49”, and the data value in the range is not held.

(FIG. 24 → FIG. 25)
If the parallel processing period ends at the time of FIG. 24, the primary and replica perform the same processing as in FIG. 21 to FIG. 22 from the entry arrangement state in FIG.
That is, the replica merges “Dn” (“d2”) of the key range (“k1” to “k30”, “k50” to “k100”) held by itself with “D” (“D1”). The merged result (“D2”) is stored in “D”.
The replica also has its own “Db” for “D” (“D1”) and “Bu” (“B1”) in the key range (“k31” to “k49”) that does not hold “Dn”. ”And“ Bb ”.

The primary merges “Dn” (“d2”) into D (“D1”) and moves “Br” (B2 ”) to“ Bu ”. Then, the primary backs up “Dn” (“d2”) in the key range “k31” to “k49” not held by the replica to its own “Db”. In addition, the primary transmits “Br” (“B2”) of all key ranges together to the replica. When the replica receives “Br” (“B2”) from the primary, the replica substitutes the received data value (“B2”) for “Bu”.
By performing the above processing, the entry arrangement state in each entry 22 is as shown in FIG.
Then, the job manager 11 updates the data table by reflecting information on each entry 22 in the data table.

(Effect of the third embodiment)
According to the third embodiment, the system 100 performs the proxy execution of the key range that did not end in the next parallel processing period even if the parallel processing period ends without completing all the proxy executions in the replica. Since new proxy execution can also be performed, replication can be performed efficiently.

<< 4th Embodiment >>
FIG. 26 is a diagram illustrating an outline of parallel calculation processing according to the fourth embodiment. In the fourth embodiment, the system configuration and the hardware configuration are the same as those in FIGS. 1 and 2, and thus illustration and description of the system configuration and the hardware configuration are omitted.
In the first embodiment to the third embodiment, as shown in FIG. 4, the same entry group is based on a model that executes parallel computation in each parallel processing period.
However, in practice, as shown in FIG. 26, different entry groups may execute parallel computation for each parallel processing period. In the fourth embodiment, a case where such a parallel distributed processing system 100 is applied to the first to third embodiments will be described.

In FIG. 26, each entry 22 has an attribute of type (type) in addition to (key, data value) as shown in FIGS. For example, the entry 22 described in the upper left of FIG. 11 describes “(T1, k1, v11)”, which has the type “T1”, the key “k1”, and the data value “v11”. ".
In this parallel distributed processing system 100, the same type of entry group performs parallel calculation processing in the same parallel processing period, and in the next parallel processing period after synchronization, an entry group of a different type from the previous parallel processing period is parallel. Perform the calculation process. Then, as the parallel processing period is repeated, the entry group that has executed the previous processing in a certain parallel processing period performs parallel processing again.
For example, in “Phase 1” in FIG. 26, an entry group of type “T1” performs parallel calculation processing, and in the next “Phase 2”, an entry group of type “T2” performs parallel calculation processing. In “Phase 3”, the processing of the entry group whose type is “T1” is performed again.
Note that “w11” in the entry group of the type “T2” indicates a data value.

  In order to realize such a parallel distributed processing system 100, the partition table shown in FIG. 5 and the data table shown in FIG. 6 are created for each type. As described above, since the partition table and data table were originally created for each job, in the fourth embodiment, the partition table and data table are created for each combination of job and type. Since processing of entry groups belonging to one type is executed in parallel, it is desirable that load equalization can be realized in units of types. For this reason, a partition table and a data table are created for each type.

  Next, processing when the parallel distributed processing system 100 shown in FIG. 26 is applied to each of the first to third embodiments will be described.

(When applied to the first embodiment)
In the data rearrangement processing described in FIGS. 8 to 13 of the first embodiment, the message transmitted by the entry 22 is received by the group of entries that are executing parallel processing in the parallel processing period. Thus, an entry group of a type that executes parallel computation processing in the parallel processing period (that is, an entry group different from the entry group currently performing parallel computation processing) is received. Accordingly, in FIG. 8 to FIG. 13, the message transmitted by each entry 22 is not transmitted to “Br” shown in the figure, but the message is transmitted to “Br” of the entry group that performs parallel calculation processing in the next parallel processing period. Sent.

  In the rearrangement, the system 100 does not rearrange the data values of the message transmission destination entry group, but rearranges the data values of the message transmission source entry group. That is, the system 100 rearranges the entry group that is currently performing the parallel calculation process, but does not perform the rearrangement for the entry group that performs the parallel calculation process in the next parallel processing period. Therefore, the node that actually transmits the message of the entry 22 is determined using a partition table of a type that performs parallel processing next. Other processes are the same as the processes described in the first embodiment.

  However, when applied to the first embodiment, the execution time of each task is measured in a certain parallel processing period, and after the data allocation to each task is determined, the system 100 is the type of the first task next. While parallel processing of different task groups is being performed, data relating to the first task can be rearranged. In addition, if the task group of the same type as the task whose execution time was first measured is executed before the data rearrangement is completed, the task group should be processed in parallel while continuing the data rearrangement. Is possible. As described above, the system 100 can improve the efficiency of the data rearrangement process. For example, in the example of FIG. 26, the execution time of a task group that performs parallel calculation processing of an entry group of “Phase1” and type “T1” is calculated, and the optimal data arrangement for each task is determined. Then, while the parallel calculation processing of the “T2” entry group is performed in “Phase2”, the rearrangement of the “T1” entry group is performed.

(When applied to the second embodiment)
The case where the parallel distributed processing system 100 according to the fourth embodiment is applied to the second embodiment is the same as that applied to the first embodiment.
That is, the node that actually transmits the message of the entry 22 is determined using a partition table of a type that performs parallel processing next. 15 to 17, the received messages are accumulated in “Br” of the entry 22 that is executing the parallel calculation process in the parallel processing period. However, when the fourth embodiment is applied, the process is executed in the parallel processing period. The middle entry group does not receive the message, and the entry group that performs parallel calculation processing in the next parallel processing period receives the message. Other processes are the same as those in the second embodiment.

  Also in the second embodiment, it is possible to implement the data rearrangement efficiency technique when the fourth embodiment is applied to the first embodiment. However, since the second embodiment calculates the entry and transmits the calculation result to the relocation destination, if the data relocation processing is not completed before the next parallel calculation processing is started, each task The parallel computation processing is performed on the data held at the start of the parallel computation processing, and the result is transmitted to the relocation destination.

(When applied to the third embodiment)
When the parallel distributed processing system 100 according to the fourth embodiment is applied to the third embodiment, the message transmission of the entry 22 is the same as that applied to the first embodiment and the second embodiment. The message transmitted by each entry 22 is transmitted to the primary of the entry group that executes parallel calculation processing in the next parallel processing period. Accordingly, in the process of shifting to FIG. 21 and FIG. 22, the primary transmits the contents of “Br” to the replica, but when the fourth embodiment is applied, an entry group that executes the next parallel processing period. The contents of “Br” are transmitted from the primary to the replica that executes the next parallel calculation process.
That is, assuming that the execution order of parallel processing is as shown in FIG. 26, at the end of “Phase 1”, the primary of “T2” transmits the content of “Br” to the replica of “T2”, and the next “ At the end of Phase 2, the primary of “T 1” transmits the content of “Br” to the replica of “T 1”.

  In FIG. 22, the unsent data value (“d2”) backed up to the primary “Db” remains, but in the next parallel processing period (FIGS. 23 to 25), these primary and replica Since the parallel calculation processing is not performed, the data value (“d2”) of the primary “Db” is transmitted to the replica during the processing of the next parallel processing period, and the replica performs “D” based on the received data value. Can be generated. Therefore, more efficient duplication than in the third embodiment is possible. Other processes are the same as those in the third embodiment.

(Effect of 4th Embodiment)
According to the fourth embodiment, it is possible to reduce the load on the entire parallel distributed processing system 100 by performing parallel calculation processing with different entry groups for each parallel processing period.

1 Master node 2 Slave node (including nodes: first node and second node)
3 Client node 11 Job manager 12 Distributed file system (master node)
21 task 22, 22a-22c entry 23 task manager 24 distributed file system (slave node)

Claims (11)

A parallel distributed processing method that has a plurality of nodes and is a virtual calculation unit that operates in each node, and performs parallel processing in a plurality of entries that can move between the nodes,
The node is
A parallel distributed processing method characterized in that, in the entry, calculation processing is performed, and data of an entry whose node is changed is moved to a node to which the entry is changed. The node is
In the entry, calculation processing is performed, and the data of the entry whose node is changed is copied to the change destination node of the entry,
The parallel distributed processing method according to claim 1, wherein the migration is performed by deleting data relating to the copied entry after completion of the calculation processing and copying. The node is
The parallel distributed processing according to claim 1 or 2, further comprising a partition table in which at least the entry, a node having the entry, and identification information of the node are stored as set information. Method. The node is
The calculation process is started from the entry to be left in the own node, and the calculation process using the data moved from the change source node is performed after the end of the calculation process in the entry to be left in the own node. The parallel distributed processing method according to any one of claims 1 to 3. In the calculation process of each entry, the node can transmit a message from each entry to a different entry, and each node receives a message transmitted from another entry by an entry held by the node. With a buffer,
The parallel distributed processing method according to claim 4, wherein the data in the buffer is moved to the change destination node together with the data. The node is
While starting the calculation process from the move entry, copy the result of the calculation process and the data used for the calculation process to the change destination node,
While performing the calculation process, move the data used for the calculation process,
The calculation process for the entry to be left is performed when the calculation process for the moving entry is completed, and the result of the calculation process is stored in its own storage unit. The parallel distributed processing method described in 1. It is a virtual distributed processing method that has a plurality of nodes and is a virtual calculation unit that operates in each node, and performs parallel processing in a plurality of entries that can move between the nodes.
A first node and a second node that are in a relationship of backup of the entry with each other and a backup destination of the entry;
The first node performs a calculation process, copies a calculation result that is a result of the calculation process to the second node, and
The second node performs a calculation process, and copies a calculation result that is a result of the calculation process to the first node;
When all the calculation results are obtained at the first node, but no calculation results are obtained at the second node,
The first node holds a calculation result in an entry for which no calculation result is obtained in the second node;
When the next calculation process starts, the first node and the second node mutually perform the calculation process and copy the calculation result, and the first node stores the held calculation result. Copying to the second node. A parallel distributed processing method. The calculation processing performed after holding the calculation result in the entry for which the calculation result is not obtained is performed in the first node and the second node,
The parallel distributed processing method according to claim 7, wherein the calculation of the entry holding the calculation result is performed later. The entries are classified into a plurality of types,
The parallel distributed processing method according to any one of claims 1 to 8, wherein the parallel processing is performed with the different types of entries for each parallel processing. A parallel distributed processing system in which a plurality of entries are arranged, which is a virtual calculation unit that performs parallel processing in a plurality of nodes,
The node is
A parallel distributed processing system characterized in that, in the entry, calculation processing is performed, and data of an entry whose node is changed is moved to a node to which the entry is changed. A parallel distributed processing system in which a plurality of entries are arranged, which is a virtual calculation unit that performs parallel processing in a plurality of nodes,
A first node and a second node that are in a relationship of backup of the entry with each other and a backup destination of the entry;
The first node performs a calculation process, copies a calculation result that is a result of the calculation process to the second node, and
The second node performs a calculation process, and copies a calculation result that is a result of the calculation process to the first node;
When all the calculation results are obtained at the first node, but no calculation results are obtained at the second node,
The first node holds a calculation result in an entry for which no calculation result is obtained in the second node;
When the next calculation process starts, the first node and the second node mutually perform the calculation process and copy the calculation result, and the first node stores the held calculation result. Copying to said 2nd node. The parallel distributed processing system characterized by the above-mentioned.

Images