JP5577745B2 - Cluster system, process allocation method, and program - Google Patents

Description

  The present invention relates to a cluster system, a process arrangement method, and a program.

In the field of high performance computing that performs processing with a large amount of calculation, a calculation system having high calculation performance is required.
Conventionally, a calculation cluster in which a small number of calculation nodes with high unit performance are connected has been used. In this case, the connection form is a single-stage crossbar system in which direct communication can be performed between all the nodes, thereby achieving uniform interconnect communication performance.

  Nowadays, computers have become more commoditized, and computation clusters are often used in which many computation nodes with relatively low unit performance are connected. In this case, if the single-stage crossbar method is adopted, the number of connections becomes about the square of the number of nodes, and therefore the number of nodes increases the number of network parts explosively.

  For this reason, in a calculation cluster in which a large number of calculation nodes are connected, it is common to use a multistage connection such as a fat tree or a mesh. Multi-stage connection can suppress an increase in the number of network components to some extent, but there is a problem that the interconnect communication performance is not uniform because the number of stages through which the nodes are combined varies.

  In the case of a calculation cluster in which a large number of calculation nodes are connected, generally, the calculation node is centrally managed by a batch server. The batch server accepts a job execution request from the front end, and determines job assignment according to the availability of resources such as CPUs and memories of each computation node. The batch server equalizes the load on the computation nodes and realizes efficient operation as a whole computation cluster.

  However, the current batch server considers only the availability of resources of the calculation node itself such as CPU and memory in determining job allocation, and hardly considers the interconnect communication performance between the calculation nodes. For this reason, the interconnect communication performance of the computation node group assigned to the distributed parallel job may be biased.

  An example of a job is a distributed parallel job. A distributed parallel job divides the calculation into individual processes, and advances the calculation in parallel while appropriately exchanging data between the processes. Distributed parallel jobs can exhibit high computing performance due to parallelization by assigning individual processes to different computing nodes.

  When executing a distributed parallel job, there may be a characteristic in communication between processes depending on the implementation of a calculation cluster. For example, when the plane direction is calculated in more detail using an X-Y-Z axis lattice model, communication between processes arranged on the XY plane is more than communication between processes arranged in the Z direction.

  As described above, when there is a characteristic in communication between processes, the performance of the distributed parallel job may not be sufficiently exhibited. This is because the batch server performs allocation without considering the interconnect communication performance between compute nodes and the communication characteristics of distributed parallel jobs, so the bias of interconnect communication performance does not match the communication characteristics between processes of distributed parallel jobs, This is because optimum interconnect communication performance cannot be obtained.

  Patent Document 1 discloses a logical node created according to a connection form of a plurality of nodes and inter-node connection devices in a lattice type computer system in a lattice type computer system in which a plurality of nodes each having a processor are connected in a lattice shape. The grid model consisting of is divided into square areas containing one or more logical nodes associated with one or more service requests made from the outside, and executed at any logical node in this square area The assigned scheduler assigns a program for processing a task to other logical nodes in the rectangular area based on the parallelism and seriality of the tasks constituting the service request job corresponding to the rectangular area. Has been.

JP 2007-206987 A

  Patent Document 1 describes a method for optimally arranging tasks in calculation nodes, but it can be applied only to a form in which nodes are connected in a grid pattern.

  Accordingly, an object of the present invention is to optimize the communication time and improve the job execution performance by optimally arranging the processes in the calculation nodes regardless of the connection form of the calculation nodes.

  The cluster system according to the present invention is a cluster system including a plurality of computing nodes and a batch server that allocates batch processing requested via a front-end device to the plurality of computing nodes, and the batch server includes: An interconnect communication performance table creation unit that creates a table including interconnect communication performance information between each of the calculation nodes, and at the start of operation, transmits the interconnect communication performance table to each of the calculation nodes, and the batch processing A job included in the batch process and an information distribution unit that transmits the communication characteristics of the job to each of the calculation nodes at the time of the request, and the calculation node includes the communication characteristics of the job and the interconnect Place each computing node by matching the communication performance Process arrangement calculation unit for determining a process comprises a.

  According to the present invention, it is possible to optimize the communication time and improve the job execution performance by optimally arranging the processes in the calculation nodes based on the job communication characteristics regardless of the connection form of the calculation nodes. Can do.

It is a figure which shows the structure of the cluster system by embodiment of this invention. It is a figure which shows the detail of a structure of the front end by embodiment of this invention, a batch server, and a calculation node. It is a flowchart of operation | movement of the cluster system by embodiment of this invention. It is a figure which shows the example of an interconnect communication performance table. It is a flowchart of operation | movement of the cluster system by embodiment of this invention. It is a figure which shows the example of the communication characteristic contained in a batch request. It is a figure which shows typically the process arrangement | positioning of a job. It is a flowchart of operation | movement of the cluster system by embodiment of this invention. It is a figure which shows the example of an allocation node list. It is a flowchart of operation | movement of the cluster system by embodiment of this invention. It is a figure explaining process arrangement | positioning determination processing. It is a figure which shows the example of the division | segmentation of a node group. It is a flowchart of operation | movement of the cluster system by embodiment of this invention. It is a figure which shows the example which performed the optimization process with respect to a node group. It is a figure explaining the matching of a process arrangement | positioning and a node group.

Next, the best mode for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing a configuration of a cluster system 10 according to the present embodiment. As shown in the figure, the cluster system 10 includes a front end 100, a batch server 200, and a computation node 300. Each front end 100, batch server 200, and each calculation node 300 are connected via a communication network. Further, the interconnect communication performance between the computation nodes 300 is not uniform, and it is necessary to communicate with the computation nodes 300 that are not adjacent via the other computation nodes 300.

FIG. 2 is a diagram showing details of the configuration of the front end 100, the batch server 200, and the calculation node 300.
As shown in the figure, the front end 100 includes a batch request unit 101. The batch request unit 101 transmits a batch request to the batch server 200. The batch request includes communication jobs and job communication characteristics. The batch request unit 101 is a functional block realized by being executed in a computer processor.

  The batch server 200 includes a job reception unit 201, a job management unit 202, an interconnect communication performance table creation unit 203, an information distribution unit 204, and a batch request storage unit 205.

The job reception unit 201 receives a batch request transmitted from the front end 100.
The job management unit 202 refers to the batch request storage unit 205 and determines assignment of the calculation nodes 300.
The interconnect communication performance table creation unit 203 acquires the interconnect communication performance of all the computation nodes 300 and creates the interconnect communication performance table.

The information distribution unit 204 transmits an interconnect communication performance table to each computation node 300. In addition, a batch request and an allocation node list are transmitted to each calculation node 300.
The batch request storage unit 205 stores the batch request received by the job reception unit 201.

  The job reception unit 201, job management unit 202, interconnect communication performance table creation unit 203, and information distribution unit 204 are functional blocks that are realized by being executed by a processor of a computer. The batch request storage unit 205 is realized by a storage device such as a memory or a hard disk.

  The calculation node 300 includes a job execution unit 301. The job execution unit 301 includes an information acquisition unit 302, a process arrangement calculation unit 303, a process activation unit 304, and an interconnect communication performance table storage unit 305.

  The information acquisition unit 302 receives the interconnect communication performance table and stores it in the interconnect communication performance table storage unit 305. The batch request and the allocation node list are received and provided to the process arrangement calculation unit 303.

  The process arrangement calculation unit 303 calculates a process arrangement with reference to the allocation node list, job communication characteristics, and interconnect communication characteristics table.

  The process activation unit 304 generates a process arranged in itself based on the process arrangement calculated by the process arrangement calculation unit 303 and starts execution.

  The job execution unit 301, the information acquisition unit 302, the process arrangement calculation unit 303, and the process activation unit 304 are functional blocks that are realized by being executed by a processor of a computer. The interconnect communication performance table storage unit 305 is realized by a storage device such as a memory or a hard disk.

Next, the operation of the cluster system 10 will be described.
The operation of the cluster system 10 can be divided into three in the order of execution: when the system starts operating, when batch processing is requested, and when job execution starts.

First, processing at the start of operation of the cluster system 10 will be described with reference to the flowchart of FIG.
When the operation is started, the interconnect communication performance table creation unit 203 of the batch server 200 acquires the interconnect communication performance of all the connected computing nodes 300, and creates the interconnect communication performance table (step S101).

  FIG. 4 is a diagram illustrating an example of an interconnect communication performance table. As shown in the figure, the interconnect communication performance between the computation nodes 300 is indicated by numerical values. The smaller the number, the higher the interconnect communication performance.

  Next, the information distribution unit 204 transmits an interconnect communication performance table to each computation node 300 (step S102).

  Next, each computing node 300 receives the interconnect communication performance table by the information acquisition unit 302 of the job execution unit 301 and stores it in the interconnect communication performance table storage unit 305 (step S103).

Next, processing when a batch processing request is made will be described with reference to the flowchart of FIG.
First, a batch request is transmitted from the batch request unit 101 of the front end 100 to the batch server 200 (step S201). The batch request includes a job execution request and job communication characteristics.

FIG. 6 is a diagram illustrating an example of communication characteristics of a job included in a batch request transmitted from the batch request unit 101. FIG. 7 is a diagram schematically showing the process arrangement of a job.
As shown in FIG. 7, the job communication characteristics are such that the job process arrangement is regarded as a grid, and the priority of each dimension of the grid and the number of processes arranged on the dimension axis are set. The process number in FIG. 7 is a number for identifying each process included in the job. Numbers are assigned from 0 in ascending order of dimension.

  The batch server 200 receives the batch request at the job reception unit 201 and stores it in the batch request storage unit 205 (step S202).

  Next, the job management unit 202 refers to the batch request storage unit 205 and determines assignment of the calculation nodes 300 (step S203).

Next, processing at the start of job execution will be described using the flowchart of FIG.
When the execution time of the job stored in the batch request storage unit 205 is reached, the batch server 200 transmits a batch request and an allocation node list to each computation node 300 by the information distribution unit 204 (step S301).

  FIG. 9 is a diagram illustrating an example of an allocation node list. As shown in FIG. 9, the assignment node list lists the calculation nodes 300 assigned to the job.

  When the information acquisition unit 302 receives the batch request and the allocation node list, the calculation node 300 calculates a process allocation with reference to the allocation node list, job communication characteristics, and interconnect communication characteristics table in the process allocation calculation unit 303 ( Step S302).

  Next, each calculation node 300 generates a process arranged in itself based on the process arrangement calculated in step S302 in the process activation unit 304, and starts execution (step S303).

The process arrangement calculation process in step S302 will be described in detail with reference to the flowchart in FIG.
Here, a case where the node shown in FIG. 9 is assigned to the job having the communication characteristics shown in FIG. 6 will be described as an example. The interconnect communication performance between the nodes is as shown in FIG.

  First, the process placement calculation unit 303 refers to the communication characteristics of the job and the list of assigned nodes, and determines the process placement order (step S401). As shown in FIG. 7, the process numbers are assigned in ascending order of dimension.

  As shown in FIG. 11, the process arrangement calculation unit 303 determines the process arrangement by collecting numbers in units of processes in the dimension direction in which the priority of communication characteristics of jobs is large.

  Next, the process placement calculation unit 303 collects the calculation nodes 300 included in the allocation node list and creates one node group (step S402).

  Next, the process arrangement calculation unit 303 refers to the job communication characteristics and divides the node group by the number of processes with the lowest priority (step S404). Here, as shown in FIG. 12, the node group of the allocation node list is divided by the number of processes “3” in the first dimension, which is the dimension with the lowest priority of communication characteristics shown in FIG.

  Next, the process placement calculation unit 303 performs optimization processing between the divided node groups (step S405).

  The inter-node group optimization process in step S405 will be described with reference to the flowchart of FIG.

  First, the process placement calculation unit 303 selects two node groups (step S501). Hereinafter, the selected node group is referred to as Ga and Gb.

  Next, the process arrangement calculation unit 303 selects one node at a time from Ga and Gb (step S502).

  Next, the process placement calculation unit 303 refers to the interconnect communication performance table and calculates the total interconnect communication performance within the node group for each of Ga and Gb (step S503). Hereinafter, the calculated sum is assumed to be Ta and Tb.

  Next, the process arrangement calculation unit 303 creates new node groups Ga ′ and Gb ′ in which the nodes selected from Ga and Gb are swapped (step S504).

  Next, the process placement calculation unit 303 calculates the total values Ta ′ and Tb ′ of the interconnect communication performance within the node group for the node groups Ga ′ and Gb ′ (step S505).

  Next, when (Ta ′ + Tb ′) <(Ta + Tb) (step S506: Y), the process arrangement calculation unit 303 updates Ga and Gb with Ga ′ and Gb ′ (step S507).

  The process placement calculation unit 303 performs the processing of steps S501 to S507 for all combinations of node groups (step S508), and ends the optimization processing when there is no update in all the node groups (step S509).

  FIG. 14 is a diagram illustrating an example in which optimization processing is performed on the three divided node groups illustrated in FIG.

  When the inter-node group optimization process in step S405 of FIG. 10 is completed, the process placement calculation unit 303 divides each divided node group with the number of processes of the dimension with the next lowest priority.

  When the number of unprocessed dimensions becomes one, the process placement calculation unit 303 ends the division process and merges the divided node groups (step S403).

  Further, as shown in FIG. 15, the process placement calculation unit 303 matches the process placement with the node group and determines a process to be assigned to each computation node 300 (step S406).

As described above, according to the present embodiment, the interconnect communication performance table between the computation nodes 300 is transmitted from the batch server 200 to each computation node 300 at the start of operation of the cluster system 10, and at the time of batch request, the batch server The communication characteristics of the job are transmitted together with the job from 200 to the calculation node 300, and each calculation node 300 matches the communication characteristics of the job with the interconnect communication performance to determine a process to be arranged in each calculation node 300.
As a result, regardless of the connection form of the calculation nodes, processes can be optimally arranged in the calculation nodes, communication time can be optimized, and job execution performance can be improved.

A part or all of the above embodiment can be described as in the following supplementary notes, but is not limited thereto.
(Supplementary note 1) a plurality of calculation nodes;
A cluster system comprising a batch server that assigns batch processing requested via a front-end device to the plurality of computing nodes,
The batch server
An interconnect communication performance table creating unit for creating a table including interconnect communication performance information between each of the computing nodes;
At the start of operation, the interconnect communication performance table is transmitted to each of the calculation nodes, and at the time of the batch processing request, the job included in the batch processing and the communication characteristics of the job are transmitted to each of the calculation nodes. And an information distribution department,
The compute node is
A cluster system comprising: a process placement calculation unit that determines a process to be placed in each computation node by matching the communication characteristics of the job with the interconnect communication performance.

(Appendix 2) In the cluster system described in Appendix 1,
The communication characteristics of the job include the priority of each dimension constituting the grid and the arrangement on each dimension axis when the processes included in the job are arranged in a grid based on the communication characteristics between the processes. Information on the number of processes
The process arrangement calculation unit
Based on the priority and the number of processes arranged on each dimension axis, a process to be arranged in each calculation node is determined so that communication characteristics between the processes and interconnect communication performance between the calculation nodes match. , Cluster system.

(Additional remark 3) The process of creating the table containing the interconnect communication performance information between each calculation node,
Transmitting the interconnect communication performance table to each of the computing nodes;
Receiving a request for batch processing, sending a job included in the batch processing to each of the calculation nodes, and communication characteristics of the job;
And a step of determining a process to be arranged in each calculation node by matching the communication characteristics of the job with the interconnect communication performance.

(Appendix 4)
An interconnect communication performance table creation unit for creating a table including interconnect communication performance information between a plurality of computing nodes;
At the start of operation, the interconnect communication performance table is transmitted to each of the calculation nodes, and at the time of requesting batch processing, the job included in the batch processing and the communication characteristics of the job are transmitted to each of the calculation nodes. A program that functions as an information distribution department.

  10 cluster system, 100 front end, 101 batch request unit, 200 batch server, 201 job reception unit, 202 job management unit, 203 interconnect communication performance table creation unit, 204 information distribution unit, 205 batch request storage unit, 300 calculation node, 301 Job execution unit 302 Information acquisition unit 303 Process allocation calculation unit 304 Process activation unit 305 Interconnect communication performance table storage unit

Claims (4)

Multiple compute nodes;
A cluster system comprising a batch server that assigns batch processing requested via a front-end device to the plurality of computing nodes,
The batch server
An interconnect communication performance table creating unit for creating a table including interconnect communication performance information between each of the computing nodes;
At the start of operation, the interconnect communication performance table is transmitted to each of the calculation nodes, and at the time of the batch processing request, the job included in the batch processing and the communication characteristics of the job are transmitted to each of the calculation nodes. And an information distribution department,
The compute node is
And communication characteristics of the job, by matching the interconnect communication performance, process arrangement calculation unit for determining a process of placing each compute node, Bei give a,
The communication characteristics of the job include the priority of each dimension constituting the grid and the arrangement on each dimension axis when the processes included in the job are arranged in a grid based on the communication characteristics between the processes. Information on the number of processes
The process arrangement calculation unit
Based on the priority and the number of processes arranged on each dimension axis, a process to be arranged in each calculation node is determined so that communication characteristics between the processes and interconnect communication performance between the calculation nodes match. , Cluster system. Creating a table containing interconnect communication performance information between each compute node;
Transmitting the interconnect communication performance table to each of the computing nodes;
Receiving a request for batch processing, sending a job included in the batch processing to each of the calculation nodes, and communication characteristics of the job;
The computing nodes, the communication characteristics of the job, by matching the interconnect communication performance, Bei example a step, the determining the process of placing each compute node,
The communication characteristics of the job include the priority of each dimension constituting the grid and the arrangement on each dimension axis when the processes included in the job are arranged in a grid based on the communication characteristics between the processes. Information on the number of processes
In the step of determining a process to be arranged in each calculation node,
Based on the priority and the number of processes arranged on each dimension axis, a process to be arranged in each calculation node is determined so that communication characteristics between the processes and interconnect communication performance between the calculation nodes match. , process alignment. Computer
An interconnect communication performance table creation unit for creating a table including interconnect communication performance information between a plurality of computing nodes;
At the start of operation, the interconnect communication performance table is transmitted to each of the calculation nodes, and at the time of requesting batch processing, the job included in the batch processing and the communication characteristics of the job are transmitted to each of the calculation nodes. , Function as the information distribution department ,
The communication characteristics of the job include the priority of each dimension constituting the grid and the arrangement on each dimension axis when the processes included in the job are arranged in a grid based on the communication characteristics between the processes. A program that contains information on the number of processes that will be executed . Computer
A program that functions as a computation node that executes batch processing assigned to a plurality of computation nodes by a batch server,
The program causes the computer to
Functions as a process placement calculation unit that determines the process to be placed in each computation node by matching the communication characteristics of the job included in the batch processing received from the batch server and the interconnect communication performance information between the plurality of computation nodes Let
The communication characteristics of the job include the priority of each dimension constituting the grid and the arrangement on each dimension axis when the processes included in the job are arranged in a grid based on the communication characteristics between the processes. Information on the number of processes
The process arrangement calculation unit
Based on the priority and the number of processes arranged on each dimension axis, a process to be arranged in each calculation node is determined so that communication characteristics between the processes and interconnect communication performance between the calculation nodes match. ,program.