JP2009086741A - Distributed processing control method in heterogeneous node existing distributed environment and its system and its program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To optimize load distribution, and to improve performance according to the state change (load fluctuation, failure occurrence, and new node addition) of each node, and to continue the distributed processing when any failure occurs, and to easily add computer resources in a distributed environment where nodes whose performances or platforms are different coexist. <P>SOLUTION: This distributed processing control method and a distributed processing control program is provided with a distributed object communication base part for achieving platform independency and position permeability by the CORBA and SDO of OMG standard; a state monitoring part for collecting the load state of each node; a resource management part for executing distributed processing by determining the distribution of distributed processing units according to the load state; a dynamic fail over control part for taking over distributed processing by dynamically changing configurations when detecting node failure; and a distributed automatic reference control part for, when detecting a new mode, making it automatically participate in the distributed processing. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to distributed processing control. In particular, in a distributed environment where nodes (computers or computer systems) with different performance and platforms coexist, autonomous distributed processing control is realized in response to changes in the state of each node (load fluctuation, failure occurrence, new node addition) The present invention relates to a distributed processing control method and a distributed processing control program.

  Conventionally, a method of allocating processing units to be distributed according to the load of each node is widely known for load distribution in a distributed environment. Patent Document 1 is a method of allocating to a server with a low load on a transaction basis. Patent Document 2 is a method of converting the urgency level and priority level of processing in units of programs. Patent Document 3 is a method of dynamically changing the execution location of an object itself according to the amount of communication between objects. These methods distribute the load at a granularity for each transaction, program, or object of interest.

  As a dynamic fail over technique required when a failure occurs in a distributed environment, a method of switching to a standby server is widely known. Patent Document 4 is a method of switching to a standby server in the event of a failure and changing the server location. Further, Patent Document 5 is a method of changing the setting of the communication path for faulty work from the active system to the standby system.

  Patent Document 6 is a general form of distributed processing, and is a method of distributing a load between a dedicated server that performs transaction distribution and execution control and a plurality of servers that distribute the load. In this method, when a new node is added, it is necessary to change a system configuration setting or a program in advance.

JP2003-296289 JP 7-282013 A JP 2000-242609 JP 2000-99483 JP 9-293059 A JP 2006-113827

  In the techniques described in Patent Documents 1 to 3, when a large number of computer resources are required for one of a transaction, a program, or an object that is a distributed processing unit, there is a possibility that the load distribution is biased. In the techniques described in Patent Documents 4 and 5, a standby server is indispensable in the case of a dynamic failover due to the occurrence of a failure, and a process for changing a server location or a communication path is necessary.

  Therefore, in the distributed processing system in which nodes having different performance and platforms are mixed, the present invention appropriately sets the distributed processing unit to each node according to the state change of each node (load fluctuation, occurrence of failure, addition of new node). The purpose is to distribute, load balance and improve performance.

  Another object of the present invention is to continue distributed processing in the event of a failure and to make it easy to add computer resources.

  One aspect of the present invention is distributed processing control that distributes a distributed processing unit to each node in a distributed processing system in which n nodes having different CPU performances coexist. The management node collects the CPU usage rate of each node, averages the collected CPU usage rate over a predetermined period, and uses the average CPU usage rate, CPU performance, and the sum of the CPU performance of each node, and the margin rate of each node Is distributed processing control in which distributed processing units are distributed to each node in accordance with the margin rate.

  Another aspect of the present invention is that even if a failure occurs in one of the n nodes to become an n−1 distributed processing system, or a new node is added to become an n + 1 distributed processing system, This is distributed processing control that distributes distributed processing units to each node in accordance with the margin ratio of each node.

  By distributing the distributed processing units according to the load state of each node based on the margin rate, fine load distribution can be achieved and the performance of distributed processing is improved.

  Embodiments will be described with reference to the drawings. FIG. 1 is a diagram for explaining the optimal distribution of distributed processing units according to the load of each node in a distributed processing system in which nodes having different performances and platforms are mixed, when load variation is considered as a state change of each node. is there.

  The management node 100 in FIG. 1 is a node that manages the entire distributed processing system. The computation node 110 is a node that executes a distributed processing unit (generally there are a plurality of computation nodes, but they are omitted in this figure).

  Although FIG. 1 shows that the management node 100 and the calculation node 110 are different nodes, when the load as the management node is low, the margin is used as the calculation node.

  SDO middleware 300 uses SDO (Super Distributed Objects: OMG standard super distributed object specification) and CORBA (Common Object Request Broker Architecture: OMG standard object specification) by OMG (Object Management Group), a standard organization for distributed object technology. The distributed processing according to the node state is controlled. The SDO middleware 300 is common regardless of the distributed processing application. The distributed processing application includes a user program A200 and a user program B210. The user program A200 corresponds to a distributed processing request source, generates a distributed processing parameter, and makes a distributed processing request. On the other hand, the user program B210 corresponds to a distributed processing request destination, and executes the distributed processing unit distributed by the distributed processing request.

  With reference to FIG. 1, the optimum load distribution mechanism will be described in the order of load state monitoring, margin ratio calculation, distributed processing unit number allocation determination, and distributed processing unit distribution execution.

  First, load state monitoring will be described. The state monitoring unit 310 of each node periodically transmits the load state of the own node including the CPU usage rate to the resource management unit 320 of the management node via a DDS (Data Distributed Service) that is a publish / subscribe distribution service. Notify Specific processing of the state monitoring unit 310 will be described with reference to FIG. The processing shown in FIG. 5 is periodically started in response to the occurrence of an event such as a failure as necessary, and acquires the failure status information of the own node (S500). If the state is normal, the failure state information indicates a normal state. Next, CPU load information and usable memory capacity information are acquired (S505, S510), and the acquired information is sent to the resource management unit 320 of the management node 100 (S115).

  In the resource management unit 320, the average CPU usage rate U (t) of each node is calculated by equation (1).

  U (t) is the average CPU usage rate at time t, c is the number of cycles to be averaged, p is the length of the cycle (collection interval), k is an integer of 1 or more, and ΣU {tp (k-1)} is a fixed time Shows the sum of the CPU usage rates collected. Here, the reason for calculating the average CPU usage rate will be described. When determining the distribution of distributed processing units, if the CPU usage rate (instantaneous value) collected from each node is used as it is, there is a possibility that the distribution of distributed processing units is biased due to the influence of a large deviation of the instantaneous value. In order to prevent this, an average CPU usage rate is calculated.

  Next, calculation of a margin ratio that is an index indicating how much computer resources each node can provide for distributed processing will be described. Here, the margin rate is defined as the remaining capacity of the computer resource (the ratio of the CPU usage rate that can be used for the distribution processing distribution), and is represented by (100−average CPU usage rate). In the resource management unit 320, together with the average CPU usage rate and the absolute evaluation value (benchmark value or the like) of the performance of each calculation node, the margin rate F (i) of each node is calculated by Equation (2).

  F (i) indicates a margin rate at node i. U (t) is the average CPU usage rate at time t, P (i) is the absolute evaluation value of the performance of node i (benchmark value, etc.), n is the total number of nodes in this distributed control system, j is an integer of 1 or more , ΣP (j) represents the sum of absolute evaluation values of all nodes.

  Here, the reason why the absolute value of performance is used in the calculation formula will be described. If (100−average CPU usage rate) is simply a margin rate, appropriate load distribution cannot be performed in an environment where nodes with different performances coexist. For example, when there are two nodes having the same margin ratio, and the difference in performance between the two nodes is double, when the allocation is performed only by the margin ratio, the ratio is 1: 1. Actually, the ratio of the margin ratio of the node having the superior performance to the other node to the other node is 2: 1, and the optimum distribution ratio is 2: 1 corresponding to this. Therefore, it is necessary to calculate the margin ratio of each node based on the absolute evaluation value of performance, and the calculation formula for this is formula (2).

  By calculating the margin ratio using the absolute evaluation value, the margin ratio is calculated based on the same standard in nodes having different performances. By calculating the margin ratio, in a distributed control system in which nodes having different performances are mixed, distributed processing units can be appropriately distributed according to the load of each node.

  FIG. 6 shows the processing in the resource management unit 320 of the management node 100 described above. Status monitoring information such as failure status information, CPU load information, and usable memory capacity information from each node is received (S600). Based on the received state monitoring information, an average CPU usage rate of each node is obtained (S605). From the obtained average CPU usage rate, a margin rate of each node is obtained.

  Here, the margin ratio is obtained based on the CPU usage rate that is CPU load information, but the available memory capacity obtained as one of the monitoring status information is the allocation target when the distributed processing unit is allocated. It will be apparent to those skilled in the art that there may be constraints that become a node.

  Next, determination of distribution of the number of distributed processing units and execution of the distributed processing units distributed will be described. In response to a processing request from the distributed processing request source, the resource management unit 320 determines the distribution of the number of distributed processing units according to the ratio of the calculated margin ratio according to Equation (3), and distributes processing to each calculation node 110 via CORBA. Allocate units. The processing of the distributed processing unit distributed by each calculation node 110 is executed.

Formula (3) for obtaining the distribution number D (i) of the distributed processing units in each node will be described. D (i) is the number of distributed processing units distributed to node i, D _all is the total number of distributed processing units to be distributed, F (i) is the margin ratio at node i, and n is all nodes of this distributed control system The number, j is an integer of 1 or more, and ΣF (j) indicates the sum of the margin rates of all nodes.

As described above, with the mechanism of FIG. 1, fine load distribution can be achieved by distributing distributed processing units according to the load state of each node, and the performance of distributed processing is improved.
2 and 3 are diagrams for explaining dynamic fail over when occurrence of a failure is considered as a state change of each node in a distributed processing system in which nodes having different performances and platforms are mixed.

  FIG. 2 is a diagram for explaining dynamic fail over when the management node 100 fails. With reference to FIG. 2, the mechanism of dynamic fail over in the event of a management node failure will be described in the order of failure detection, selection of a new management node, registration of reference information, and takeover of distributed processing.

  First, failure detection will be described. In order to realize this dynamic fail over, a resource management unit 320 including a dynamic fail over control unit is arranged and operated in each computation node 110. Each computing node 110 monitors an operation notification periodically generated by the management node 100 and detects a failure of the management node 100.

  The selection of a new management node to replace the management node 100 in which a failure has been detected will be described. Since the new management node is automatically determined, the resource management unit 320 of the calculation node 101 that detects the failure (hereinafter, the new management node, but at this time is the calculation node) determines the priority order of the own node. A representative selection message to which is added is sent to another node. Each calculation node (for example, calculation node 110) compares the priority of the received representative selection message with the priority of its own node, and continues its function as a calculation node when its own priority is low. When the priority order of the own node 110 is higher than the priority order of the received representative selection message, this node transmits a representative selection message to which the priority order of the own node 110 is newly added to other nodes. If a new representative selection message is not received by a predetermined time after sending the representative selection message, this node that has sent the representative selection message becomes the new management node. When the priority of the calculation node 101 that has detected the failure is the highest due to the selection of the new management node, the calculation node 101 becomes the new management node. In this manner, a new management node can be selected without depending on the position of the alternative node or the prior arrangement.

  The process for selecting a new management node will be described with reference to FIG. The process of FIG. 7 is executed as a process of the resource management unit 320 of each computation node 101. The management node 100 monitors operation notifications periodically generated and detects a failure of the management node 100 (S700). The priority order of the own node is sent to other calculation nodes (S705). For a predetermined time (S710), priority order information from other calculation nodes is received (S715). The priority received from the other calculation node is compared with the priority of the own node sent in step 705 (S720). When the priority from the other calculation node received is higher than the priority of the own node, The process ends. When the priority from other calculation nodes exceeding the priority of the own node is not received within the predetermined time, the own node is determined as a new management node (S725).

  Next, registration of new reference information will be described. After the new management node 101 is determined, the resource management unit (new representative) 320 of the new management node 101 registers new reference information in the naming service. By registering the new reference information, the object reference of the new management node can be performed thereafter.

  A table reference using a distributed shared memory (not shown) is used as a mechanism for taking over the intermediate result of the distributed processing at the time of dynamic fail over during the distributed processing.

  As described above, the dynamic fail-over mechanism of FIG. 2 can take over the distributed processing in the event of a failure of the management node and continue the distributed processing without depending on the position of the alternative node. . Thereafter, the distributed processing is performed by the new management node 101 and the plurality of computing nodes 110 in a state where distributed processing according to the load is possible.

  The load on the new management node 101 due to the failure of the management node 100 increases as the function of the management node is executed. Therefore, by applying the distributed control mechanism shown in FIG. 1 to the n−1 nodes constituting the distributed control system excluding the management node 100, the distributed operation according to the load can be performed. . In this case, it is desirable that the distributed processing unit to be newly allocated is targeted for the unprocessed distributed processing unit by the new management node 101. This is because processing overhead for redistribution becomes large when unprocessed distributed processing units of all nodes are targeted.

  FIG. 3 is a diagram for explaining the dynamic fail over when the computing node 110 fails. With reference to FIG. 3, the mechanism of dynamic fail over in the event of a computation node failure will be described in the order of failure detection, SDO restart, and registry registration.

  A failure of the calculation node 110 is detected by the resource management unit 320 of the management node 100. For example, when no state information is reported for a predetermined time from the state monitoring unit 310 of the calculation node 110, it is detected as a failure of the calculation node 110. In addition, if the status monitoring unit 310 on each computation node makes periodic access to the SDO registry (registry that stores and manages information related to SDO, node addresses, etc.) and cannot be accessed, “failure” is set as status information. Is reported to the resource management unit 320 of the management node 100. This is also a failure detection of the computation node.

  Next, the restart of SDO will be described. When a failure of a calculation node is detected, the resource management unit 320 (including the dynamic fail-over control unit) instructs the fail-over calculation node 111 to restart the SDO (fail over). .

  Next, registry registration will be described with reference to FIG. In the computing node 111 that has restarted the SDO, SDO management (a module for registering, updating, and deleting SDO) that is a part of the dynamic fail-over control unit is registered in the SDO registry (S800). At the time of SDO activation on the own node, the SDO activation notification including the SDO ID (unique identifier) and the CORBA object reference is transmitted simultaneously in a one-to-many manner through the event channel (S805).

  Each node updates the node address information of the SDO registry when receiving an SDO activation notification corresponding to the SDO ID managed by the node. By this registry registration, the object reference of the new calculation node can be made thereafter.

As in the case of a management node failure, a table reference using the distributed shared memory is used as a mechanism for taking over the intermediate result of the distributed processing at the time of dynamic fail over during the distributed processing.
As described above, with the dynamic fail-over mechanism of FIG. 3, when a failure occurs in a computation node, the distributed processing can be taken over and continued without depending on the position of the alternative node. .

  With the failure of the computation node 110, the distributed processing according to the load can be performed by applying the distributed control mechanism shown in FIG. Act as possible. In this case, it is desirable that the distributed processing unit to be newly allocated is a distributed processing unit that has not been processed by the computation node 110. This is because processing overhead for redistribution increases when the unprocessed distributed processing units of all nodes are targeted.

  FIG. 4 is a diagram for explaining the automatic participation in the distributed processing of a new node, which is for the addition of a new node among the state changes of each node. With reference to FIG. 4, a mechanism of distributed automatic participation by adding a new node will be described in the order of new node detection, new node participation preparation, new node addition, and distributed participation icon.

  A method for detecting a new node will be described. The resource management unit 320 of the management node 100 monitors the status information periodically transmitted from each calculation node, and detects the new node 120 by receiving the status information from the new node 120 that is not under management. By detecting the new node 120, the management node 100 can automatically detect the new node 120.

  Next, preparation for participation of a new node will be described. The resource management unit 320 of the management node 100 instructs distributed participation to the new node 120 at the timing when the new node 120 is detected. The processing of the distributed automatic participation control unit 330 of the new node 120 instructed to perform distributed participation will be described with reference to FIG. The distributed automatic participation control unit 330 of the new node 120 instructed to participate in the distributed participation notifies the management node 100 of the preset absolute performance evaluation value regarding the own node 120 (S900), and then participates in the distributed processing. The new node 120 receives the necessary definition file and DLL (Dynamic Link Library) from the management node 100 using FTP (File Transfer Protocol) (S905). By preparing this new node, preparations necessary for participating in distributed processing can be automatically performed.

  For the absolute evaluation value of performance, a preset value is used. Alternatively, the benchmark is executed on the new node 120 that is instructed to be distributed from the management node 100, and the absolute evaluation value of the performance is acquired.

  The distributed automatic participation control unit 330 of the new node 120 notifies the resource management unit 320 of the management node 100 of preparation completion (including the absolute evaluation value of the new node) after completion of the FTP transfer (S910).

   Next, addition of a new node will be described. The resource management unit 320 of the management node 100 adds new node information to the distributed environment management table at the reception timing of completion of preparation. With this addition, the new node 120 has joined the distributed environment. This is because the resource management unit 320 of the management node 100 refers to the distributed environment management table when determining the distribution of distributed processing units.

  Next, the distributed participation icon will be described. The resource management unit 320 of the management node 100 notifies the new node 120 of completion of participation (addition completion to the distributed environment management table) that the new node 120 has completed participation in the distributed environment. The distributed automatic participation control unit of the new node 120 displays an icon indicating that distributed participation is in progress at the time when the notification of completion of participation is received.

  In addition, this icon indicates that the node is participating in the distributed environment, and the node can be removed from the distributed environment by icon operation. When an icon operation is performed to cause the node to leave the distributed environment, the node information is deleted from the distributed environment management table of the management node 100. This deletion leaves the distributed environment. This is because the resource management unit 320 of the management node 100 refers to the distributed environment management table when determining the distribution of distributed processing units. After leaving, delete the icon.

  As described above, with the automatic participation mechanism of the new node in FIG. 4 as described above, a series of automatic participation in the distributed processing including detection of a new node, setting for participating in the distributed processing, and start of distributed processing Can do.

  With the addition of the new node 120, distributed processing according to the load is possible by applying the distributed control mechanism shown in FIG. 1 to the n + 1 nodes constituting the distributed control system including the new node 120. Act as a state. In this case, the timing for distributing the distributed processing unit to the new node 120 should be considered. Next, it is desirable to distribute according to the timing when the distributed processing unit is generated. This is because processing overhead for redistribution becomes large when unprocessed distributed processing units of all nodes are targeted.

  As described above, in a distributed environment where nodes with different performance and platforms coexist, the distributed processing units are optimally allocated to improve performance according to the state change of each node (load fluctuation, failure occurrence, addition of new node). be able to. When a failure occurs, the distributed processing can be continued without a dedicated standby server, and the new node can be automatically participated in the distributed processing only by connecting to the LAN.

  For example, according to the load status of each node, the processing unit (distributed processing unit) divided into the minimum units of calculation parameters that can be processed in parallel and parallel is optimally distributed to each node, thereby achieving extremely fine load distribution and distribution. The processing performance can be improved.

  Here, the reason why the performance is improved by the finely distributed processing unit distribution according to the load will be described. If the granularity of distribution is coarse as in a transaction unit, the transaction is executed on the node with the lowest load. In the case where the transaction requires a large amount of computer resources, there is no problem until now. On the contrary, there is a possibility that a node having a low load will be overloaded. On the other hand, if distributed processing units with fine granularity are distributed according to the load of each node, leveling can be achieved without load unevenness, and as a result, the performance of distributed processing is improved.

  In addition, a dedicated standby server is not required in the event of a node failure, and it is possible to continue the distributed processing by automatically taking over the intermediate result of the distributed processing by dynamic fail over. In the user program, there is no need to change the server location setting or the communication path setting. Further, even if the new node is connected to the LAN without any special setting and distributed processing is in progress, computer resources can be added and the distributed processing can be continued.

It is a figure explaining the optimal allocation according to the load of the distributed processing in the distributed environment where nodes with different performances and platforms are mixed. It is a figure explaining the dynamic fail over at the time of a management node failure. It is a figure explaining the dynamic fail over at the time of a calculation node failure. It is a figure explaining the automatic participation to the distributed process of a new node. It is a process flowchart of a state monitoring part. It is a processing flowchart of a resource management part. It is a processing flowchart of new management node selection. It is a processing flowchart of the registry registration in the calculation node of a fail over destination. It is a process flowchart of the distributed automatic participation control part of a new node.

Explanation of symbols

  100 ... Management node, 110 ... Compute node, 300 ... SDO middleware, 310 ... Status monitoring unit, 320 ... Resource management unit, 330 ... Distributed automatic participation control unit,

Claims (6)

  A distributed processing control method for allocating a distributed processing unit to each of the n nodes in a distributed processing system in which n nodes having different CPU performances coexist, wherein a certain node determines the CPU usage rate of each node. Collecting, averaging the collected CPU usage of each node over a predetermined period, and using the averaged average CPU usage of each node, the CPU performance of each node, and the sum of the CPU performance of each node A distributed processing control method for obtaining a margin rate of each node and allocating the distributed processing unit to each node in accordance with the margin rate of each node.   In response to detection of a failure of the certain node, another node included in each of the n nodes collects the CPU usage rate of each of the n−1 nodes excluding the certain node, and the collected The CPU usage rate of each of n−1 nodes is averaged over a predetermined period, and the averaged CPU usage rate of each of the n−1 nodes averaged, the CPU performance of each of the n−1 nodes, and the n Using the sum of the CPU performance of each of the -1 nodes, the margin ratio of each of the n-1 nodes is obtained, and the distributed processing unit is determined according to the margin ratio of each of the n-1 nodes. The distributed processing control method according to claim 1, wherein the distributed processing control method is allocated to each of the n−1 nodes.   In response to detection of a failure of a node other than the certain node, the certain node collects the CPU usage rate of each of the n−1 nodes excluding the node where the failure is detected, and the collected n -Average the CPU usage rate of each node over a predetermined period, and average the average CPU usage rate of each of the n-1 nodes, the CPU performance of the n-1 nodes, and the n- Using the sum of the CPU performance of each node, the margin ratio of each of the n−1 nodes is obtained, and the distributed processing unit is set to correspond to the margin ratio of each of the n−1 nodes. The distributed processing control method according to claim 1, wherein the distributed processing control method is allocated to each of n−1 nodes. In response to the joining of a new node to each node constituting the distributed processing system, the certain node collects the CPU usage rate of each of n + 1 nodes including the joined node, and the collected The CPU usage rate of each of n + 1 nodes is averaged over a predetermined period, and the averaged CPU usage rate of each of the n + 1 nodes averaged, the CPU performance of each of the n + 1 nodes, and the CPU of each of the n + 1 nodes 2. The margin ratio of each of the (n + 1) nodes is obtained using a sum of performance, and the distributed processing unit is allocated to each of the (n + 1) nodes in accordance with the margin ratio of each of the (n + 1) nodes. The distributed processing control method as described.   A distributed processing system that distributes a distributed processing unit to each node having different CPU performance, collecting the CPU usage rate of each node, averaging the collected CPU usage rates of each node over a predetermined period, and calculating the average Using the average CPU usage rate of each node, the CPU performance of each node and the sum of the CPU performance of each node, obtain the margin rate of each node, corresponding to the margin rate of each node, A distributed node having a node that distributes the distributed processing unit to the nodes, and another node that responds to the CPU usage rate in response to the CPU usage rate collection request and executes the distributed processing unit Processing system.   In a distributed processing system in which nodes having different CPU performances are mixed, a distributed processing control program for a certain node that distributes a distributed processing unit to each node, the step of collecting the CPU usage rate of each node, The step of averaging the CPU usage rate of each node over a predetermined period, the averaged CPU usage rate of each node, the CPU performance of each node, and the sum of the CPU performance of each node, A distributed processing control program for causing a computer to execute a step of obtaining a margin rate of each node and a step of allocating the distributed processing units to the nodes in correspondence with the margin rates of the nodes.

Images