JP5331549B2 - Distributed processing system and distributed processing method - Google Patents

Description

  The present invention relates to a distributed processing system including a plurality of computers and a plurality of relay devices, and a distributed processing method executed by the system.

  Conventionally, various techniques have been developed to increase the processing efficiency of a distributed processing system. For example, Patent Document 1 below discloses a technique for determining a node to which a calculation process is assigned in consideration of the time required to change the activation state of software. Specifically, this technology compares the environmental information indicating the active software acquired from each node with the environmental condition information indicating the software required for processing, and processes a node with fewer new software startups. Is preferentially selected as the allocation destination.

JP 2008-140120 A

  Depending on the type of distributed processing, the speed at which data is read from a hard disk or the like, and the communication speed when data is moved between computers become factors that reduce the overall processing speed. For example, when the amount of intermediate data generated is small, such as a map reduction (grep) process, the data read speed from the hard disk becomes a bottleneck in each computer. Further, when a large amount of intermediate data is generated as in the MapReduce sort process, a large amount of traffic is concentrated on a relay device such as an L2 switch, which becomes a bottleneck.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a distributed processing system and a distributed processing method capable of processing data at high speed.

  The distributed processing system of the present invention is a distributed processing system including a plurality of groups including a relay device that relays data and at least one computer that is directly connected to the relay device, and each computer that has executed the distributed processing Acquisition means for acquiring the amount of input data of the process and the amount of output data as a result of the process, and before and after distributed processing based on the input data amount and output data amount acquired by the acquisition means Setting means for determining a variance of a plurality of divided data to be distributed when a change in the amount of data is determined and the amount of data increases due to the processing, compared to a case where the amount of data decreases due to the processing When the degree of distribution set by the setting means is equal to or less than a predetermined threshold, a plurality of divided data to be distributed is assigned to a plurality of groups. Placed one group Chi, otherwise, characterized in that it comprises an arrangement means for arranging the data segments of the plurality of the two or more groups among the groups, the.

  The distributed processing method of the present invention is a distributed processing method executed by a distributed processing system including a plurality of groups including a relay device that relays data and at least one computer directly connected to the relay device. The acquisition step of acquiring the data amount of the input data of the processing and the data amount of the output data as a result of the processing in each computer that has executed the distributed processing, the input data amount acquired in the acquisition step, and When a change in the data amount before and after the distributed processing is determined based on the output data amount and the data amount is increased by the processing, a plurality of pieces of divided data to be distributed are processed rather than when the data amount is decreased by the processing. A setting step for setting the degree of dispersion of a low value, and when the degree of dispersion set in the setting step is equal to or less than a predetermined threshold Arranging a plurality of divided data to be distributed in one group among a plurality of groups, if not, arranging the plurality of divided data in two or more groups among a plurality of groups; It is characterized by including.

  According to such a distributed processing system and distributed processing method, it is determined whether the data amount has increased due to the distributed processing based on the input data amount and the output data amount in each computer. The degree of data distribution is determined. The degree of dispersion is set relatively low when the amount of data increases, and is set relatively high when the amount of data decreases. If the set degree of dispersion is equal to or less than a predetermined threshold, the divided data is aggregated into one group, and otherwise, the divided data is distributed into a plurality of groups.

  When the amount of data increases, suppressing network traffic in subsequent processing leads to faster processing. On the other hand, when the amount of data decreases, distributing the data to be processed thereafter to as many computers as possible and maximizing the processing capacity of the system leads to faster processing. Therefore, by setting the degree of dispersion based on the change in the amount of data before and after processing and arranging the divided data based on the degree of dispersion, it is possible to realize the flexible data arrangement as described above, and as a result, the data It becomes possible to process at high speed.

  In the distributed processing system of the present invention, it is preferable that the distributed processing is executed by a map reduction programming model.

  In this case, data can be processed at high speed in the map reduce process.

  According to such a distributed processing system and distributed processing method, the degree of distribution is set based on the change in the amount of data before and after the processing, and the divided data is arranged based on the degree of dispersion, so that the data is processed at high speed. can do.

1 is a diagram illustrating an overall configuration of a distributed processing system according to an embodiment. It is a figure which shows the function structure of the master node shown in FIG. It is a figure which shows the hardware constitutions of the master node shown in FIG. It is a figure which shows the example of the information memorize | stored in the dispersion degree table shown in FIG. It is a figure which shows the function structure of the slave node shown in FIG. It is a figure which shows the example of the group information memorize | stored in the rack structure table shown in FIG. It is a sequence diagram which shows operation | movement of the distributed processing system shown in FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  First, the function and configuration of the distributed processing system 1 according to the embodiment will be described with reference to FIGS.

  The distributed processing system 1 is a computer system that executes distributed processing using a programming model called MapReduce. The data to be processed is assumed to be distributed in advance in the distributed processing system 1 by a predetermined method. Hereinafter, the distributed data is referred to as divided data. As shown in FIG. 1, the distributed processing system 1 includes one master node 10, a plurality of slave nodes 20, and a plurality of network switches (hereinafter simply referred to as “switches”) 30 that connect the nodes. I have. The master node 10 is a computer that supervises distributed processing, and the slave node 20 is a computer that actually processes data. The switch 30 is a device that relays data transmitted between nodes, and is, for example, an L2 switch, an L3 switch, or a router.

  The master node 10, each slave node 20, and each switch 30 are stored in any one of a plurality of racks R. In the example of FIG. 1, the distributed processing system 1 is configured to include one master node 10, 11 slave nodes 20, 4 switches 30, and 4 racks R. Each rack R stores three nodes (master node 10 or slave node 20) and one switch 30, and each node is directly connected to the switch 30 in each rack R. The plurality of slave nodes 20 stored in one rack R form one group (hereinafter referred to as “node group”).

  Next, the master node 10 will be described. As shown in FIG. 2, the master node 10 includes a reception unit 11, a distribution degree table 12, an instruction unit 13, and a distribution degree setting unit (setting unit) 14 as functional components.

  As shown in FIG. 3, the master node 10 includes a CPU 101 that executes an operating system, application programs, and the like, a main storage unit 102 that includes a ROM and a RAM, an auxiliary storage unit 103 that includes a hard disk, and a network. The communication control unit 104 includes a card, an input unit 105 such as a keyboard and a mouse, and an output unit 106 such as a monitor. Each function of the master node 10 reads predetermined software on the CPU 101 and the main storage unit 102, operates the communication control unit 104 under the control of the CPU 101, and stores data in the main storage unit 102 and the auxiliary storage unit 103. This is realized by reading and writing.

  Returning to FIG. 2, the accepting unit 11 is a part that accepts a distributed processing execution instruction input by a user operation or input from another computer system (not shown). This execution instruction includes application information for identifying the distributed processing program and, in some cases, includes a parameter indicating the degree of distribution. Here, the degree of distribution is a numerical value that indicates how much a plurality of intermediate data to be distributed (intermediate results of distributed processing) are distributed and arranged. If the degree of distribution is small, the intermediate data is arranged in a narrow range (for example, in one node group (rack R)), and if the degree of dispersion is large, the intermediate data is arranged in a wide range (for example, all slave nodes 20 in the system 1). Is done. The accepting unit 11 outputs the accepted execution instruction to the instructing unit 13.

  The degree of dispersion table 12 is a part that stores the degree of dispersion set for each application. For example, the degree of distribution table 12 stores the degree of application dispersion as shown in FIG. According to FIG. 4, when the application A is executed, the intermediate data is collected into a group consisting of only a specific slave node, and when the application B is executed, the intermediate data is collected into a wider range of slave nodes. Be placed. Further, when executing the application C, it is arranged in a wider range of slave nodes than in the case of the application B.

  The instruction unit 13 is a part that instructs each slave node 20 to execute distributed processing. Specifically, the instruction unit 13 transmits an instruction signal including the input application information and the degree of distribution corresponding to the application to each slave node 20.

  When the degree of dispersion is input, the instruction unit 13 transmits the degree of dispersion, that is, the degree of dispersion designated by the user or another system in the instruction signal. On the other hand, when the degree of distribution is not input, the instruction unit 13 reads out the degree of dispersion corresponding to the application information from the degree-of-dispersion table 12, and when it can be obtained, transmits the degree of dispersion included in the instruction signal. If it cannot be obtained, the instruction unit 13 transmits a default value of the degree of dispersion that is held in advance in the instruction signal. A phenomenon in which the degree of distribution cannot be acquired may occur, for example, when an application is executed for the first time in the distributed processing system 1.

  The degree-of-distribution setting unit 14 determines a change in the data amount before and after the distributed processing based on the input data amount and the output data amount received from each slave node 20, and if the data amount increases due to the processing, the data amount This is a part in which the degree of distribution of a plurality of divided data to be distributed is set to be lower than that in the case where the decrease is. The input data amount is the data amount of the divided data, and the output data amount is the data amount of the operation result data. These will be described later again.

The degree-of-dispersion setting unit 14 checks the magnitude relation between the amount of input data and the amount of output data. If the amount of output data is larger, the degree of dispersion is set to a predetermined value a or less. A value larger than the predetermined value a is set. For example, the dispersion degree setting unit 14, for each slave node 20, calculates the ratio of the amount of output data to the input data amount r = (output data amount) / (input data amount), the average value r _a of the ratio r calculate. And set to 1 degree of dispersion if r _a> 1, if r _a ≦ 1, as the degree of dispersion higher r _a smaller increase, sets the degree of dispersion in two or more. Further, the dispersity setting unit 14, an input data amount received within a predetermined time and the amount of output data only, i.e. only on the basis of the amount of data received from some of the slave node 20, similarly to the above average value r _a May be calculated to set the degree of dispersion.

  Subsequently, the dispersity setting unit 14 stores the calculated dispersity and the application information received by the receiving unit in the dispersity table 12 in association with each other. As a result, the degree of distribution of the application being executed is registered or updated, and is referred to the next time an execution instruction regarding the same application is input.

  Next, the slave node 20 will be described. As shown in FIG. 5, the slave node 20 includes a rack configuration table 21, a processing unit (arranging unit) 22, and an acquiring unit (acquiring unit) 23 as functional components. The hardware configuration of the slave node 20 is the same as that shown in FIG. 3, and how each function of the slave node 20 is realized on the hardware is the same as that of the master node 10.

  The rack configuration table 21 is a part that stores information regarding the switch 30 and the slave node 20 (node group) directly connected to the switch 30. Specifically, the rack configuration table 21 includes a switch ID for identifying the switch 30, a node ID for identifying the slave node 20 directly connected to the switch 30, and a node group including the slave node 20. Group information in which rack IDs to be identified are associated with each other is stored. If the distributed processing system 1 has the configuration shown in FIG. 1, the rack configuration table 21 stores the group information shown in FIG. From this information, for example, three slave nodes 20 with node IDs “a”, “b”, and “c” form one node group, and these nodes 20 are identified by the switch ID of “SW1”. It is possible to know that the switch 30 is directly connected.

  The information in the rack configuration table 21 is transmitted at a predetermined timing from the master node 10 that manages the group information. By storing this information in the rack configuration table 21, the rack configuration table is synchronized between the master node 10 and the slave node 20.

  The processing unit 22 is a part that performs an operation on the divided data based on the information sent from the master node 10. The processing unit 22 receives the instruction signal sent from the master node 10 and extracts the application information and the degree of distribution. Subsequently, the processing unit 22 determines whether or not processing in the own node is necessary based on the application information and the divided data stored in advance. At this time, if it is determined that the processing is not necessary, the processing unit 22 waits until there is a request from another node. On the other hand, if it is determined that processing is necessary, the processing unit 22 executes an application program indicated by the application information and executes a predetermined calculation (map processing) on the divided data.

  The processing unit 22 performs map processing on the divided data and generates calculation result data. Then, Reduce processing is performed for each part of the calculation result data.

  At this time, the processing unit 22 selects a node group to execute the next distributed processing based on the input degree of distribution and the group information read from the rack configuration table 21. The node group selection method is not limited to one. For example, the processing unit 22 selects only one node group when the degree of distribution is less than or equal to the threshold value c, and selects two or more node groups when the degree of dispersion exceeds the threshold value c. In addition, although it is arbitrary which node group is specifically designated, it is preferable that the node group with the largest number of stored divided data is included.

Here, the threshold value c is whether the average value r _a is greater than 1, i.e. may be set in accordance with the distributed processing whether the reference data amount is increased in the front and rear. In this case, when the amount of data increases, only one node group is selected. Otherwise, a plurality of node groups are selected. Of course, the threshold c, the average value r _a may be set regardless of whether more than one. In this case, only one node group is selected only when the amount of data increases by a certain percentage or more.

  Subsequently, the processing unit 22 selects one of the slave nodes 20 included in the selected node group as a reducer. Then, the processing unit 22 transmits a part of the operation result data to the selected slave node 20. It can be said that the operation result data to be transmitted is intermediate data that requires further processing. The processing unit 22 repeatedly performs such reduction selection and transmission processing until all the processing on the operation result data is completed.

  When the processing unit 22 of each slave node 20 executes such processing, the plurality of intermediate data subjected to the map processing is arranged in any of the slave nodes 20 in any of the racks R, and thereafter A Reduce process is executed according to the MapReduce programming model. That is, the processing unit 22 functions as an arrangement unit.

  The acquisition unit 23 is a part that acquires the data amount of the divided data (input data) processed by the processing unit 22 and the data amount of calculation result data (output data). The acquisition unit 23 acquires these data amounts by monitoring the arithmetic processing of the processing unit 22 and transmits the data amounts to the master node 10.

  Next, the operation of the distributed processing system shown in FIG. 1 will be described with reference to FIG. 7, and the distributed processing method according to the present embodiment will be described.

  In the master node 10, when the accepting unit 11 accepts an execution instruction (step S11), the instructing unit 13 acquires the degree of dispersion corresponding to the designated application (step S12). Specifically, the instruction unit 13 reads the degree of dispersion from the degree-of-dispersion table 12, extracts the degree of dispersion from the execution instruction, or uses the default value of the degree of dispersion that is held in advance, thereby determining the degree of dispersion. get. Subsequently, the instruction unit 13 transmits the application information and the degree of distribution to all the slave nodes 20 (Step S13).

  In each slave node 20, the processing unit 22 determines whether or not processing is necessary based on the application information and the pre-stored divided data. Map processing is executed (step S14). At this time, the acquisition unit 23 acquires the data amount of the divided data that is input data and the data amount of calculation result data that is output data (step S14, acquisition step).

  Subsequently, the processing unit 22 selects one or a plurality of node groups based on the degree of distribution and the group information read from the rack configuration table 21 in order to transmit the calculation result data part by part (step S15, arrangement). Step), one slave node is selected from the selected node group (Step S16, placement step). And the process part 22 transmits a part of calculation result data to the slave node 20 (step S17, arrangement | positioning step). The processes of steps S15 to S17 are repeatedly executed until all the processes for the calculation result data are completed.

  When the processing unit 22 of each slave node 20 executes the above-described processing, the plurality of intermediate data subjected to the map processing are collected in one or a plurality of groups, and thereafter, the reduction processing is performed in the slave nodes 20 in the group. Is executed (step S18). FIG. 7 shows only one “selected slave node”, but actually there are a plurality of such slave nodes 20.

  In the slave node 20 that has executed the Map process, in parallel with the processes of Steps S15 to S17, the acquisition unit 23 transmits the data amounts of the divided data and the operation result data to the master node 10 (Step S19).

  In response to this, in the master node 10, the degree-of-dispersion setting unit 14 calculates a ratio between the data amounts, and sets the degree of dispersion based on the ratio (step S20, setting step). For example, when the amount of data increases due to distributed processing, the degree-of-distribution setting unit 14 sets the degree of distribution so that intermediate data is collected in one node group (rack R) the next time the same application is executed. To do. On the other hand, when the amount of data decreases due to distributed processing, the degree-of-distribution setting unit 14 increases the amount of data so that intermediate data is distributed to two or more node groups the next time the same application is executed. Set a higher degree of dispersion than the case. The dispersity setting unit 14 stores the dispersity set in this way in the dispersity table 12 (step S21, setting step). This degree of distribution is referred to when an execution instruction for the same application is input.

  As described above, according to the present embodiment, it is determined whether or not the data amount has increased due to the distributed processing based on the data amount of the divided data and the operation result data in each slave node 20. Based on this, the degree of distribution of the intermediate data is determined. The degree of dispersion is set relatively low when the data amount increases (for example, set to 1), and relatively high when the data amount decreases (for example, set to 3). If the set degree of dispersion is equal to or less than a predetermined threshold, the intermediate data is aggregated into one group, and if not, it is distributed into a plurality of groups.

  When the amount of data increases, distributed processing within one group (rack R) so that the data does not pass through the plurality of switches 30 leads to an increase in speed. On the other hand, when the amount of data decreases, it is possible to maximize the processing capacity of the system 1 by distributing the data to as many slave nodes 20 as possible without considering which group (rack R). It leads to high speed. Therefore, by setting the degree of dispersion based on the change in the amount of data before and after processing and placing intermediate data based on the degree of dispersion, flexible data placement as described above can be realized, and as a result, data can be transferred at high speed. Can be processed.

  For example, when the sort processing is executed in the distributed processing system 1, the amount of data after the calculation is larger, so the intermediate data (calculation result data) is gathered in one group (rack R). It is possible to avoid the occurrence of a large amount of data moving across and the concentration of traffic on the switch 30. On the other hand, in the case of the grep process, since the data after the operation is smaller, the intermediate data is distributed to a plurality of groups (rack R), and as a result, the processing efficiency is improved. On the other hand, when the sort process is performed using the conventional method, the intermediate data is distributed to a plurality of groups (rack R), so that a large amount of traffic is concentrated on the switch, and the processing speed is lowered as a whole.

  In addition, according to the present embodiment, since the degree of dispersion is automatically set for each application, a more optimal degree of dispersion can be flexibly set as compared with the case where the degree of dispersion is manually set as in the past. Is possible.

  Further, according to the present embodiment, when the operation result data is arranged in one group, the operation result data is collected in the group that stores the most divided data, so that the data is stored in one group. The amount of communication and the amount of processing when moving can be reduced.

  The present invention has been described in detail based on the embodiments. However, the present invention is not limited to the above embodiment. The present invention can be modified in various ways as described below without departing from the scope of the invention.

  The present invention can also be applied to a case where distributed processing is executed by a method other than MapReduce.

  DESCRIPTION OF SYMBOLS 1 ... Distributed processing system, 10 ... Master node, 11 ... Reception part, 12 ... Dispersion degree table, 13 ... Instruction part, 14 ... Dispersion degree setting part (setting means), 20 ... Slave node (computer), 21 ... Rack structure Table, 22... Processing unit (arrangement unit), 23... Acquisition unit (acquisition unit), 30.

Claims (3)

A distributed processing system comprising a plurality of groups including a relay device for relaying data and at least one computer directly connected to the relay device,
An acquisition means for acquiring the amount of input data of the processing and the amount of output data as a result of the processing in each computer that has performed distributed processing;
A change in the data amount before and after the distributed processing is determined based on the input data amount and the output data amount acquired by the acquisition means, and when the data amount increases by the processing, the data amount decreases by the processing. Setting means for setting the degree of dispersion of a plurality of divided data to be distributed lower than the case,
If the degree of distribution set by the setting means is less than or equal to a predetermined threshold, a plurality of divided data to be distributed is arranged in one group among the plurality of groups, and if not, the plurality of divided data Arrangement means for arranging the divided data in two or more groups among the plurality of groups;
A distributed processing system comprising: The distributed processing is executed by a map reduction (MapReduce) programming model.
The distributed processing system according to claim 1. A distributed processing method executed by a distributed processing system including a plurality of groups including a relay device that relays data and at least one computer directly connected to the relay device,
In each of the computers that have executed distributed processing, an acquisition step of acquiring the amount of input data of the processing and the amount of output data that is the result of the processing;
A change in the data amount before and after the distributed processing is determined based on the input data amount and the output data amount acquired in the acquisition step, and when the data amount increases by the processing, the data amount decreases by the processing. A setting step for setting the degree of dispersion of the plurality of divided data to be distributed lower than the case,
When the degree of distribution set in the setting step is equal to or less than a predetermined threshold, a plurality of divided data to be distributed are arranged in one group of the plurality of groups, and if not, the plurality of divided data Placing the divided data in two or more of the plurality of groups;
A distributed processing method comprising:

Images