JP4063256B2 - Computer cluster system, management method therefor, and program - Google Patents

Description

  The present invention relates to a computer cluster system, a management method therefor, and a program, and more particularly, to a cluster system for a NUMA computer, a management method therefor, and a program therefor.

  Most conventional cluster systems are constructed by connecting a large number of SMP (Symmetric-Multi-Processor) computers equipped with a small number (typically about 2 to 8) processors. The management method and various management software are premised on such a configuration.

  Therefore, the granularity of many management items is in units of machines.

  In other words, the cluster management software manages various resources (CPU, memory, etc.) of a cluster in which a plurality of computers are bundled, and manages the allocation of resources to jobs to be submitted. In order to realize exclusive resource allocation, it has a function of dividing the entire cluster into a plurality of partitions.

  In conventional cluster management software, this partitioning can be done only at the granularity of each computer.

  On the other hand, many NUMA (Non-Uniform-Memory-Access) type multi-processor shared memory systems are used.

  A computer designed based on the NUMA architecture internally connects a plurality of SMP computers using a high-speed connection mechanism. Therefore, the number of processors mounted on one normal computer is larger than that of a computer having an SMP configuration.

  In addition, as an operating system for a NUMA multiprocessor system, the following operating system (OS) that distributes a work load is shown.

  The OS includes a plurality of leaf nodes corresponding to each job processor, a root node representing at least one system resource shared by all processors, and a plurality of intermediates representing resources shared by different combinations of job processors. Maintain a hierarchical tree with nodes.

  This OS monitors the progress of active thread groups distributed throughout the system, and is associated with a medium-term scheduler that assists the declining thread groups, and one of each job processor, monitors its status, and executes it. A plurality of dispatchers for obtaining a thread group, and further, virtual and physical memories are allocated using a plurality of memory pools and frame treasuries (see Patent Document 1).

JP-A-9-237215 (first page)

  When a cluster configured by connecting a plurality of NUMA type computers as described above is managed by conventional cluster management software, there is a problem that the partitioning has a large granularity and practical partitioning cannot be performed.

  Next, the NUMA multiprocessor system OS is an OS that operates on one NUMA computer, and is not intended for a cluster in which a plurality of NUMA computers are connected.

  In addition, this dispatcher is a part of a task scheduler existing in the OS kernel, and is not suitable for the purpose of minimizing the modification of the OS.

  In addition, the entire resource is regarded as one, and it is aimed at how many jobs can be shared and used effectively.

  An object of the present invention is to secure resources that are not affected by each other by first dividing the entire resource into a plurality of resources, and realize independent job scheduling in each resource.

  In addition, the minimum size of the division can be set to one subnode of each computer constituting the cluster.

  The first computer cluster system of the present invention is a computer cluster system including a plurality of computers composed of a plurality of interconnected sub-nodes, and these sub-nodes are connected to one or more processors and local computers. The local memory is accessed locally by the processor and includes a storage area that is distributed and shared among the sub-nodes in the computer, and obtains resource information available or resource usage status of each sub-node, Means for collecting these into available resource information for each partition that is a computer partition, and means for determining the execution destination partition and start timing of the input job using the resource information, and as the configuration of the partition, It is also possible to configure one subnode. That.

  The second computer cluster system of the present invention is the first computer cluster system, and further, based on the determination, in the process of the job to be started, the processor and local memory of the sub-node constituting the execution destination partition Means for allocating the storage area and starting the process.

  A third computer cluster system of the present invention is the first computer cluster system, and the computer cluster system also includes a control computer that is connected to the plurality of computers for communication, and includes a computer for each partition. It is characterized by comprising means for collecting the available resource information and means for determining the execution destination partition and start timing of the input job.

  The fourth computer cluster system of the present invention is a computer cluster system including a plurality of computers composed of a plurality of interconnected sub-nodes, and these sub-nodes are connected to one or more processors and a local computer. The local memory is accessed locally by the processor and includes a storage area that is distributed and shared among subnodes in the computer, and operates in cooperation with the job scheduler in each of the job scheduler and the computer. A dispatcher and an OS, wherein the job scheduler obtains available resource information or resource usage status of each sub-node from the OS, and summarizes them into available resource information for each partition which is a computer partition; Accept user jobs, Based on the calculated information, the priority of the job, and the available resource information of each partition, the means for selecting the next job to be started in each partition, and the dispatcher on the computer, Means for instructing the start of the selected job after designating information on subnodes constituting the partition, and the dispatcher sends a process on the designated subnode to the process constituting the job through the OS. It has a means for allocating a storage area of a processor and a local memory, and a means for starting a process, and the configuration of the partition can also be a configuration of one subnode.

  The fifth computer cluster system of the present invention is a computer cluster system including a plurality of computers composed of a plurality of interconnected sub-nodes, and these sub-nodes are connected to one or more processors and local The local memory is accessed locally by the processor and includes a storage area that is distributed and shared among subnodes in the computer, and operates in cooperation with the job scheduler in each of the job scheduler and the computer. A dispatcher and an OS, and the OS allocates resource information or a resource usage status of each subnode of the local computer, and assigns a processor and a processor set of the specified subnode to the specified process; The processor or processor set belongs Means for allocating a storage area from a local memory, and the job scheduler obtains available resource information or resource usage status of each sub-node from the OS, and uses them as available resource information for each partition which is a computer partition. Collecting means, accepting user jobs, calculating resources required for them, and selecting the next job to start in each partition according to the calculated information, job priority, and available resource information for each partition And means for instructing the dispatcher on the computer to start the selected job after designating information on the subnodes constituting the partition, and the dispatcher sends the designation to the process constituting the job. The processor and memory allocation on the assigned subnode It means for instructing the S, and means for starting the process, as a constituent of the partition, and wherein the sub-node one configuration that can also be.

  A sixth computer cluster system according to the present invention is the fourth or fifth computer cluster system, and the computer cluster system also includes a control computer connected to the plurality of computers. The job scheduler is provided.

  The first program of the present invention obtains resource information or resource usage status that can be used for each of a plurality of subnodes constituting each computer connected to the network, and uses these resources for each partition that is a computer partition. The computer causes the computer to execute a procedure for collecting information and a procedure for determining a partition and a start timing of an input and received job using the resource information.

  The second program of the present invention obtains available resource information or resource usage status from the OS of the computer for each of a plurality of subnodes constituting each computer connected to the network, and these are computer partitions. The procedure for collecting the available resource information for each partition, accepting the user's job, calculating the resources required for this, and depending on the calculated resource information, job priority, and available resource information for each partition, The computer is caused to execute a procedure for selecting a job to be started in each partition and a procedure for instructing the program on the computer to start the selected job after specifying information on subnodes constituting the partition.

  The third program of the present invention receives a partition information of a job and an execution destination, assigns a processor of a subnode constituting the execution destination partition and a storage area of a local memory to a process constituting the job, and the process The computer is caused to execute the procedure for starting up.

  The fourth program of the present invention manages available resource information or resource usage status for each of a plurality of subnodes constituting a computer, and presents them in response to a request and a designated process. And assigning a processor or processor set of the designated sub-node and allocating a storage area in a local memory to which the processor or processor set belongs.

  The management method in the first computer cluster system of the present invention includes a plurality of computers composed of a plurality of interconnected subnodes, which are network-connected, and each subnode includes one or more processors and a local memory. A management method in a computer cluster system, which is a procedure for acquiring available resource information or resource usage status of each sub-node and collecting them into available resource information for each partition which is a computer partition, and the resource information , And a procedure for determining the execution destination partition and start timing of the input job, and the configuration of the partition may be a configuration of one subnode.

  The management method in the second computer cluster system of the present invention is the management method in the first computer cluster system, and further, the execution destination partition is configured in the process of the job to be started based on the determination. It also has a procedure for allocating a sub-node processor and a local memory storage area and starting a process.

  The management method in the third computer cluster system of the present invention includes a plurality of computers composed of a plurality of interconnected subnodes, which are connected to a network, and each subnode includes one or more processors and a local memory. A management method in a computer cluster system, which obtains available resource information or resource usage status of each sub-node, and summarizes them into available resource information for each partition which is a computer partition, and a user job. Accept, calculate the resources required for this, configure the partition and the procedure to select the job to start next in each partition according to the calculated resource information, job priority, and available resource information of each partition Specify the subnode information to be selected. A procedure for instructing start of a job, a procedure for allocating a storage area of a processor and local memory on the designated subnode to a process constituting the job, and a step for starting the process. As a configuration, a configuration with one subnode is also possible.

  The first effect is that a fine-grained resource can be divided in a cluster composed of NUMA computers.

  This is because the cluster management software uses information provided by the OS and a mechanism for control.

  The second effect is that the processes constituting the job can operate more efficiently in the NUMA type computer.

  The reason is that it is highly possible that the memory used by the process can be obtained from the local memory of the NUMA node.

  Next, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  In the present invention, the granularity of partitioning is realized by the cooperation of the OS function for the NUMA computer and the cluster management software.

  FIG. 1 is a block diagram including a computer cluster system of the present invention. Referring to FIG. 1, the computer cluster system of the present invention is a cluster system in which two or more NUMA type computers 1 including a plurality of NUMA subnodes 11 to 14 and connecting means 15 are connected by a high-speed network 3 or the like.

  The connection means 15 is a high-speed connection means such as a crossbar switch.

  The control computer 4 is connected to the NUMA type computers 1 and 2 through a high-speed LAN 100 or the like, and accepts jobs from the user terminals 6-1 to 6-n through the network 200 and the communication terminal 5.

  As shown in FIG. 2, each NUMA subnode has a plurality of CPUs 111 to 113 connected to a system control unit 116 through a bus 115, and a local memory 117 that can be accessed at high speed by the CPUs 111 to 113 is connected to the system control unit 116. Has been.

  Each of the CPUs 111 to 113 includes cache memories 1111,.

  The system control unit 116 accesses the local memory 117 if the upper part of the address of the memory access request from the CPUs 111 to 113 indicates the local memory 117.

  If the upper part points to any local memory of the NUMA subnodes 12 to 14, a memory access request is sent to the corresponding system control part through the connection means 15.

  In the NUMA type computers 1 and 2, the NUMA subnode 11 and the NUMA subnode 21 are in charge of communication with the control computer 4, and a LAN communication unit 118 is also connected to these system control units.

  Similarly, a high-speed communication unit is connected to the system control unit of the NUMA subnode that is in charge of high-speed communication between the NUMA computers 1 and 2.

The OS (operating system) operating on each NUMA computer has the following functions.
(1) A function of binding an arbitrary process to an arbitrary processor or an arbitrary processor set.
(2) A function of allocating a memory to a process from a NUMA subnode allocated to the process or a process in which the process is operating or a set of memories.
(3) A function that makes it possible to acquire the total resource amount and the current resource (local memory, processor) usage amount from each user program or daemon in each NUMA subnode.
(4) A memory resource management mechanism in units of NUMA subnodes.

  The cluster management software has a job scheduler and a job dispatcher in order to realize partitioning.

  The job scheduler has an instance for each partition of the cluster, and then executes on each partition according to restrictions on job execution of the cluster (necessary resources, job priority, free resources on the NUMA computer, etc.) Select a job to start.

  Here, the job scheduler acquires the amount of available resources or the resource usage status in each NUMA subnode by the function (3) of the NUMA type computer OS, and sums them according to the configuration information of each partition. You know the amount of resources available for the partition.

  The partition is a computer section for dividing the NUMA type computers 1 and 2 into one or more sections and handling each section as one computer.

  Define 1 to 8 NUMA subnodes in one partition.

  For example, the NUMA sub-node 11 is the partition # 1, the NUMA sub-node 12 is the partition # 2, the NUMA sub-nodes 13 and 14 are the partition # 3, and so on.

  In some cases, the NUMA subnode 11 is the partition # 1, the NUMA subnode 12 is the partition # 2, the NUMA subnodes 13 and 14, and the NUMA subnodes 21 to 24 are the partition # 3.

  The job dispatcher is executed on each NUMA type computer, and starts executing the job selected by the job scheduler after performing a necessary initialization operation on each NUMA type computer.

  At this time, the job dispatcher binds the operation process to a processor on a specific NUMA sub-node using the function (1) of the OS according to the instruction of the job scheduler.

  In addition, the process memory is set to be acquired from the scheduled or operating NUMA subnode using the functions (2) and (4) of the OS, thereby affecting the memory outside the partition. Does not reach.

  In addition, when binding each process that constitutes a job to a processor, if the user gives a hint about the communication pattern or communication topology between processes, the information can be automatically determined from the operation of the process. In this case, it is possible to optimize the communication speed by allocating a process set having a large amount of communication to a certain NUMA sub-node.

  FIG. 3 shows the cooperation between the above-mentioned OS 17 and 27 for the NUMA computer, the job scheduler 41 and the job dispatchers 16 and 26 in the cluster management software.

  The OS 27 and the job dispatcher 26 are not shown, but are installed in the NUMA computer 2.

  With these linkages, it is possible to achieve the granularity of cluster partitioning, which has conventionally been in units of NUMA computers, in units of NUMA subnodes in the computer.

  The NUMA type computer 1 has a plurality of NUMA subnodes 11 to 14, and the OS 17 operating on the NUMA type computer 1 discloses the resource information 171 of each NUMA subnode so that it can be seen by the user.

  The resource information 171 is resource information of each NUMA subnode, and is available free resource information or resource usage status (including usage amount and used process identification).

  The same applies to the NUMA computer 2, but the number of NUMA subnodes is not necessarily the same as that of the NUMA computer 1.

  In this embodiment, the OS 17 and the job dispatcher 16 are mounted on a part of the NUMA sub-node 11.

  This cluster is divided into a plurality of partitions by the administrator, and the user is permitted to execute jobs in partitions each composed of 1 NUMA subnodes from the NUMA computers 1 and 2.

  On the control computer 4 that manages the cluster, a job scheduler 41 that accepts job input from the user and performs job scheduling is operating.

  On the NUMA type computers 1 and 2 constituting the cluster, job dispatchers 16 and 17 for starting execution of programs input by the job scheduler 41 operate as processes on the NUMA type computers 1 and 2.

  The job scheduler 41 collects and combines the resource information 171 and 271 of all the NUMA subnodes constituting the partition through the OSs 17 and 27, and determines a job to be started next.

  Next, the job scheduler 41 instructs the job dispatchers 16 and 26 on the NUMA type computers 1 and 2 to start the job after specifying the information (identification) of the NUMA subnodes constituting the partition.

  The job dispatcher 16 instructs the OS 17 to bind the process constituting the job onto the CPU on the NUMA subnode belonging to the partition.

  The OS 17 is instructed to allocate the memory to the process, and the OS 17 allocates the memory on the NUMA subnode (the NUMA subnode to which the CPU belongs) constituting the partition.

  The job dispatcher 16 starts the process.

  As a result, the effect of partitioning in units of NUMA nodes can be obtained.

  Next, the operation of the best mode for carrying out the present invention will be described with reference to the drawings. FIG. 4 is a flowchart showing the operation of the embodiment of the present invention.

  When the job scheduler 41 receives a job input from the user terminal 6-1 through the communication terminal 5 or the like (step A1), the job scheduler 41 calculates a resource required by the job (step A2), and according to the priority. Then, scheduling is performed (step A3).

  When there are enough free resources in the partition obtained from the OS 17 to execute the job and the job becomes executable (step A4), the job scheduler 41 configures the job in the job dispatcher 16. The process is passed, information on the allocated resource is notified, and activation is instructed (step A5).

  The job dispatcher 16 receives the activation instruction (step B2), and passes the accompanying resource designation information to the OS 17 to secure the resources (step B3).

  After securing the resources, the job dispatcher 16 starts executing the job (step B4).

  When the execution of the job is completed (step B5), the output of the job is collected and transmitted to the job scheduler 41 (step B6), the reserved resources are released (step B7), and the job dispatcher issues an instruction to start the next job. wait.

  The job scheduler 41 outputs the result of the received job. For example, it returns to the user terminal 6-1 (step A7).

  In this embodiment, the computer cluster system includes a control computer and is equipped with a job scheduler 41. However, the job scheduler 41 and the job dispatcher 16 are provided in one processor of the NUMA subnode 11 of the computer cluster system. There is also an example to install.

  In the example of FIG. 1, an example with two NUMA computers has been described. However, any number of NUMA computers can be used as long as the number of NUMA computers is two or more.

  Conversely, the present invention can also be applied to the case where it is desired to partition and operate the inside of one NUMA type computer.

  The present invention can be applied to a NUMA computer cluster system, but it can also be applied to applications that realize finer granular resource control by implementing functions equivalent to those for NUMA computer OS in the OS for SMP computers. It is.

  For example, partitioning can be performed in units of one CPU.

1 is an overall block diagram including a computer cluster system of the present invention. The block diagram which shows the structure of the NUMA subnode 11 of FIG. The figure which shows cooperation with OS for NUMA type computers, and cluster management software (a job scheduler and a job dispatcher). The flowchart which shows operation | movement of embodiment of this invention.

Explanation of symbols

1, 2 NUMA type computers 11-14, 21-24 NUMA subnode 111-113 CPU
1111,..., 1311, Cache memory 115 Bus 116 System control unit 117 Local memory 118 LAN communication unit / High speed communication unit 15 Connection unit 16, 26 Job dispatcher 17, 27 OS
171 Resource information of each NUMA subnode 3 High-speed network 4 Control computer 41 Job scheduler 5 Communication terminal 6-1 to 6-n User terminal 100 High-speed LAN
200 network

Claims (11)

A computer cluster system including a plurality of computers composed of a plurality of interconnected sub-nodes, these being network-connected,
Each sub-node includes one or more processors and local memory, and the local memory is locally accessed by the processor and includes a storage area that is distributed and shared among the sub-nodes in the computer.
The computer cluster system, comprising the construction of said subnodes one, until structure comprising said subnodes of a plurality of pre-Symbol in the computer definable, the cluster management means for handling separated to the computer compartment i.e. partitions,
The cluster management means obtains available resource information or resource usage status of each sub-node, and summarizes them into available resource information for each partition;
A computer cluster system comprising means for determining the execution destination partition and start timing of an input job using the resource information. 2. The apparatus according to claim 1, further comprising means for activating a process by allocating a processor and a local memory storage area of a subnode constituting the execution destination partition to a process of a job to be started based on the determination. The computer cluster system described. The computer cluster system also includes a control computer that is communicatively connected to the plurality of computers, and includes means for collecting the available resource information for each partition, a partition of the execution destination of the input job, and a start timing The computer cluster system according to claim 1, further comprising means for determining. A computer cluster system including a plurality of computers composed of a plurality of interconnected sub-nodes, these being network-connected,
Each sub-node includes one or more processors and local memory, and the local memory is locally accessed by the processor and includes a storage area that is distributed and shared among the sub-nodes in the computer.
Said computer cluster system, from the sub-nodes one configuration, until structure comprising said subnodes of a plurality of pre-Symbol in the computer definable, and cluster management software that handles separated to the computer compartment i.e. partitions, each of said computer OS With
The cluster management software has a job scheduler and a dispatcher that operates in cooperation with the job scheduler in each of the computers,
The job scheduler obtains resource information or resource usage status of each sub-node from the OS, collects these into the available resource information for each partition, and accepts a user job and is necessary for this A partition is created in the dispatcher on the computer and means for selecting the next job to start in each partition according to the calculated information, job priority, and available resource information for each partition. Means for instructing the start of the selected job after designating information of subnodes to be performed,
The dispatcher includes means for allocating a processor and a local memory storage area on the designated subnode to a process constituting a job through the OS, and means for starting the process.
Examples partition configuration, computer cluster system from the service nodes send one configuration, characterized in that possible to structure comprising a plurality said computing machine of the subnode. The OS allocates the resource information or resource usage status of each subnode of the own computer, and assigns the processor or processor set of the specified subnode to the specified process, and the local to which the processor or processor set belongs. 5. The computer cluster system according to claim 4, further comprising means for allocating a storage area from a memory. 6. The computer cluster system according to claim 4, wherein the computer cluster system includes a control computer connected to the plurality of computers and includes the job scheduler. 7. Separated computer cluster system the computer comprises a plurality comprising a plurality of interconnected sub-node from the subnode one configuration, the plurality of pre-Symbol definable up structure comprising said subnodes in the computer, the computer compartment i.e. partition And obtaining resource information that can be used for each of the plurality of subnodes, or resource usage status, and collecting them into available resource information for each partition;
A program for causing a computer to execute an execution destination partition of a received and received job and a procedure for determining a start timing using the resource information. Separated computer cluster system the computer comprises a plurality comprising a plurality of interconnected sub-node from the subnode one configuration, the plurality of pre-Symbol definable up structure comprising said subnodes in the computer, the computer compartment i.e. partition For each of the plurality of sub-nodes, the available resource information or resource usage status is obtained from the OS of the computer, and the procedure for collecting them into the available resource information for each partition, and the user job, Accepting, calculating the resources required for this, and selecting the job to start next in each partition according to the calculated resource information, job priority and available resource information of each partition,
A program for causing a computer program to execute a procedure for instructing activation of the selected job after designating information on subnodes constituting a partition. A management method in a computer cluster system comprising a plurality of computers comprising a plurality of interconnected subnodes, these being network-connected, and each subnode including one or more processors and a local memory,
The computer cluster system, comprising from the subnode one configuration, until structure comprising said subnodes of a plurality of pre-Symbol in the computer definable, the cluster management procedures for handling separated to the computer compartment i.e. partitions,
The cluster management procedure acquires available resource information or resource usage status of each sub-node, and summarizes them into available resource information for each partition;
A management method in a computer cluster system comprising a procedure for determining a partition to execute an input job and a start timing using the resource information. 10. The method according to claim 9, further comprising: allocating a sub-node processor and a local memory storage area constituting the execution destination partition to the process of the job to be started based on the determination, and starting the process. A management method in the computer cluster system described. A management method in a computer cluster system comprising a plurality of computers comprising a plurality of interconnected subnodes, these being network-connected, and each subnode including one or more processors and a local memory,
Said computer cluster system, from the sub-nodes one configuration, until structure comprising said subnodes of a plurality of pre-Symbol in the computer definable, and cluster management procedures for handling separated to the computer compartment i.e. partitions, each of said computer OS And
The cluster management procedure obtains available resource information or resource usage status of each sub-node from the OS, and summarizes them into available resource information for each partition;
A procedure for accepting a user's job, calculating the resources required for this, selecting the next job to be started in each partition according to the calculated resource information, job priority, and available resource information for each partition; ,
A procedure for instructing the start of the selected job after designating information of subnodes constituting the partition;
Allocating storage areas of a processor and local memory on the designated subnode to a process constituting the job through the OS;
A management method in a computer cluster system, comprising: a procedure for starting a process.

Images