JP6325995B2 - Distributed system, load balancing method and program - Google Patents

Description

  The present invention relates to a distributed system, a load distribution method, and a program capable of correcting a load imbalance among a plurality of nodes.

  In recent years, with the rise of cloud computing, it has been required to efficiently process and hold a large amount of data. Thus, distributed processing technology has been developed that realizes efficient processing by operating a plurality of servers in a coordinated manner. When performing distributed processing, it is necessary to determine data for each server (hereinafter also referred to as a node) that configures a cluster and constructs a distributed system. At this time, in order to increase the data processing capacity of the entire distributed system, the number of data handled by each node is preferably averaged.

  By the way, as a representative data management method, a remainder obtained by dividing a value obtained by multiplying the key (key) of each data by a hash function (hereinafter also referred to as hash (key): hash key) by the number of nodes N, that is, hash. There is a method in which a node having (key) mod N as a number manages data. At this time, however, numbers from 0 to N-1 are assigned to the nodes in advance. When such a management method is used, when a node is added or removed, the value of N changes, and the node in charge is changed for a lot of data, so that the node in charge needs to be rearranged.

  In the data management method using the consistent hash method, the loads indicated by the circles and ● are different from those of the nodes A, B, C, D, and E in the circular ID space indicated by reference numeral 5 in FIG. ID (identification) is assigned to both load data. The node in charge of load data follows the ID space 5 from the node in charge (for example, B) in the clockwise direction, and is responsible for load data arranged between the node C that first hits. As an example of how IDs are given to the nodes A to E, a value obtained by applying an IP (Internet Protocol) address to a hash function {this is also referred to as a hash (IP address)} can be given.

  In a distributed system having a cluster configuration, for example, when the performance of each node is equal, the amount of data handled by each node A to E is equal, that is, the distance between nodes in the ID space 5 of the consistent hash method (hereinafter, It is desirable that the nodes are in charge of each other.

  In order to solve this problem, a method of virtually giving a plurality of IDs to the nodes A to E is used. Since each node A to E has a plurality of virtual IDs, even if the assigned area for each virtual ID is different, the assigned areas of the nodes A to E are averaged according to the law of large numbers. Conventional techniques such as the consistent hash method and virtual ID make it possible to equalize the number of data handled between nodes and distribute the load.

  However, since data with a high access frequency and data with a long processing time (high load data) are unevenly generated at specific nodes among the nodes A to E, the number of data handled by the nodes A to E itself Even if they are equal, load imbalance occurs between nodes.

  As a countermeasure against such a load increase in the distributed system of the consistent hash method, a new node F shown in FIG. 14 is added to the distributed system, and the distributed system is scaled out. Node), for example, a method of reducing the load by reducing the number of data handled by the high-load node C.

  In addition, if the load is distributed appropriately by changing the spatial arrangement of the nodes (for example, B) on the consistent hash (this is called rebalancing), it can be handled with the current number of nodes without adding additional nodes. There are also cases. Non-Patent Document 1 identifies situations to be dealt with by scale-out / rebalance, and in situations to be dealt with by rebalance, between adjacent nodes on the consistent hash space (ID space 5) (for example, In the case of E, a method has been proposed in which rebalancing is performed in the adjacent A or D) indicated by two arrows to correct the load imbalance between nodes.

Yasu Tsuruta et al., “Node Load Balancing Optimization Method in Distributed Server System”, IEICE General Conference, Mar. 2014, B-7-84

  By the way, even if the method of Non-Patent Document 1 is used, the adjacent node is not necessarily a node suitable for rebalancing, such as a node having a high load next to the high load node. In this case, even if there is another node with a low load, the load cannot be immediately rebalanced there. However, if rebalancing is repeated, it is possible to finally correct the load imbalance. However, in this case, since rebalancing has to be repeated, it may take a lot of time and is not practical. In other words, there is a problem that rebalancing cannot be performed efficiently.

  The present invention has been made in view of such a background, and an object of the present invention is to provide a distributed system, a load distribution method, and a program capable of efficiently performing rebalancing.

In order to solve the above-mentioned problem, the invention according to claim 1 is a distributed system having a plurality of nodes to which information from a plurality of client machines using a communication service is distributed via a network, The following formula (1) is stored in the part,
Number of virtual nodes = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of nodes having current virtual nodes) (1)
In accordance with the load status of each node, the current number of virtual nodes, the average value of the loads of all the nodes, and the actually measured value of the load of the node having the current virtual node are obtained, and these obtained numerical values are represented by the formula (1). ) To calculate the number of virtual nodes for each node, change the size of the current area per virtual node of the virtual node, and the virtual node having the size of the current area after the change A distributed system comprising a rebalancing unit that performs rebalancing to allocate virtual nodes to the corresponding nodes so that the calculated number of virtual nodes exists.

  According to this configuration, the rebalancing unit determines the number of virtual nodes that each node has without specifying the target node to which the assigned area is transferred and the appropriate transfer size of the corresponding node. Depending on the number of virtual nodes, the number of virtual nodes for obtaining a necessary load amount can be obtained. That is, rebalancing adapted to the required load amount of the own node can be performed.

The invention according to claim 2 is a load distribution method having a plurality of nodes to which information from a plurality of client machines using a communication service is distributed via a network, and the node stores the following formula ( 1)
Number of virtual nodes = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of nodes having current virtual nodes) (1)
In accordance with the load status of each node, the current number of virtual nodes, the average value of the loads of all the nodes, and the actually measured value of the load of the node having the current virtual node are obtained, and these obtained numerical values are represented by the formula (1). ) To calculate the number of virtual nodes for each node, change the size of the current area per virtual node of the virtual node, and change the virtual area having the changed size of the current area. And performing a rebalancing step of allocating virtual nodes to the corresponding nodes so that there are as many virtual nodes as the calculated number of virtual nodes.

  According to this method, rebalancing suitable for the required load amount of the node can be performed.

The invention according to claim 3 stores the following formula (1) in the storage unit as the node of the distributed system having a plurality of nodes to which information from a plurality of client machines using the communication service is distributed via the network. Computer
Number of virtual nodes = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of nodes having current virtual nodes) (1)
In accordance with the load status of each node, the current number of virtual nodes, the average value of the loads of all the nodes, and the actually measured value of the load of the node having the current virtual node are obtained, and these obtained numerical values are represented by the formula (1). ), The virtual node having the size of the assigned area after the change, the means for calculating the number of virtual nodes for each node, the size of the assigned area per virtual node of the current virtual node is changed Is a program for functioning as means for performing rebalancing to allocate virtual nodes to the corresponding nodes so that the calculated number of virtual nodes exists.

According to this program, the same effect as that of claim 2 can be obtained.

  According to the present invention, it is possible to provide a distributed system, a load distribution method, and a program capable of efficiently performing rebalancing.

1 is a block diagram illustrating a configuration of a distributed system according to a first embodiment of the present invention. The configuration of the node in the distributed system of this embodiment is shown, (a) is a block diagram showing the configuration of the control unit, (b) is a block diagram showing the information in the storage unit. It is a figure which shows the hash space divided | segmented by several nodes AE. (A) A figure showing an example of a node identifier management table, (b) A figure showing an example of a distribution ID table. (A) A diagram showing an example of node load measurement data (node unit), (b) a diagram showing an example of node load measurement data (virtual node unit), and (c) an example of node load measurement data (data unit). FIG. It is a figure which shows an example of distributed system load total data. (A) The figure which shows an example of the distribution ID table before rebalance, (b) The figure which shows an example of the distribution ID table after rebalance, (c) The other example of the distribution ID table after rebalance is shown FIG. It is a figure which shows the some virtual node of a node, and its charge area (charge hash space). It is a figure which shows the list of the load amount by the various load data in the charge area | region of each virtual node shown in FIG. It is a bar graph showing the deviation of the load amount which each node AC holds. It is a 1st flowchart for demonstrating the operation | movement at the time of performing the rebalance of each node of the distributed system of this embodiment. It is a 2nd flowchart for demonstrating the operation | movement at the time of performing the rebalance of each node of the distributed system of this embodiment. (A) A figure showing an example of a distribution ID table before rebalancing, (b) A figure showing an example of a distribution ID table after rebalancing. It is a figure which shows the hash space for demonstrating a prior art.

Embodiments of the present invention will be described below with reference to the drawings.
<Configuration of First Embodiment>
FIG. 1 is a block diagram showing a configuration of a distributed system according to the first embodiment of the present invention.
A distributed system 10 shown in FIG. 1 is a system that performs data management using a plurality of nodes 15 using a consistent hash method. A feature of the present invention is that, when a load imbalance occurs between the nodes 15 constituting the distributed system 10, when it is possible to cope with the number of current nodes 15, the efficiency is determined based on the load imbalance of the current node 15. Rebalance to correct the load imbalance.

  The distributed system 10 includes a load balancer 13 connected to a plurality of client machines (also simply referred to as clients) 11 via a network 12 such as the Internet, and a plurality of nodes 15 constituting a cluster 14. Yes.

  Each node 15 is a physical device such as a computer or a logical device such as a virtual machine, in other words, a physical or virtual server. A message from the client 11 is distributed to each node 15 by the load balancer 13. This distribution is performed by a simple round robin method or the like.

The node 15 includes a control unit 18 and a storage unit 19. However, it is assumed that the control unit 18 and the storage unit 19 are configured by software. It may be configured by hardware.
As shown in FIG. 2A, the control unit 18 includes a node identifier management unit 18a, a distribution unit 18b, a signal processing unit 18c, a node load measurement unit 18d, a distributed system load rebalancing unit (simply, 18e).

  As shown in FIG. 2B, the storage unit 19 includes a node identifier management table 19a, a distribution ID table 19b, data 19c, node load measurement data 19d, distributed system load summary data 19e, and call control. The status flag 19f is stored. The node identifier management table 19a is also called a management table 19a, the distributed system load summary data 19e is also called summary data 19e, and the call control status flag 19f is also called flag 19f.

The distribution unit 18b distributes the message (information) from the client 11 to the node 15 in charge of the message based on, for example, a consistent hash method.
The signal processing unit 18 c performs predetermined signal processing in response to a message from the client 11 and provides a service to the client 11. That is, the node 15 in charge of the message provides a service to the client 11 by performing predetermined signal processing in the signal processing unit 18c. The processing operations of the distribution unit 18b and the signal processing unit 18c will be described in detail later.

  However, in the distributed system 10, the load balancer 13 does not exist, and a message can be transmitted from the client 11 to an arbitrary node 15 (distribution unit 18b). Further, the allocating unit 18b and the signal processing unit 18c may be simultaneously present on the same node 15 as illustrated in FIG.

  In the control unit 18, the node identifier management unit 18a manages the ID space handled by the node 15 by accumulating node information on the distributed system 10 in the node identifier management table 19a. This ID space is a space (hash space) on the consistent hash in the consistent hash method.

  For example, as shown in FIG. 3, the hash space is divided by a plurality of nodes A to E, and assigned areas of the nodes A to E are determined and managed. At this time, the hash space handled by the node A is an area from the node A to the node B in the clockwise direction, and the responsible node A holds data existing in the hash space. The same applies to the other nodes B to E. In addition, as the size of the hash space (area in charge) is larger, more data can be held.

  Returning to FIG. 2A, the distribution unit 18b performs processing related to determination of the distribution destination of data such as a message in the distributed system 10, and the distribution destination information of the data obtained by this processing is assigned to the distribution ID table. It manages as 19b.

The signal processing unit 18 c performs signal processing at the node 15. Data 19c to be accessed at the time of this signal processing is stored in the storage unit 19.
The node load measuring unit 18d measures the load of the node 15, records the measurement result as the node load measurement data 19d in the storage unit 19, and determines the privileged node 15 (for example, the node B shown in FIG. ).

The distributed system load rebalancing unit 18e collects the node loads of the entire distributed system 10, calculates the average value and standard deviation of the collected loads, and stores the distributed system load summary data 19e as the calculation results. 19 Further, the rebalancing unit 18e performs rebalancing by performing rebalancing execution determination and rebalancing design based on the stored total data 19e.
The call control state flag 19f stored in the storage unit 19 is information for determining whether or not the state is a state for controlling a new call.

  Here, message distribution processing and signal processing by the distribution unit 18b and the signal processing unit 18c of the node 15 shown in FIG. 1 will be described in more detail.

The distribution unit 18b identifies the node 15 in charge of signal processing based on the information in the message called from the client 11, and distributes the message to the other node 15 concerned. The message is divided into a new call (for example, Initial-INVITE or the like in SIP (Session Initiation Protocol)) and a subsequent call (for example, BYE in SIP).
Whether the call is a new call or a subsequent call can be determined based on whether or not a distribution key described later is embedded in the call. For example, in SIP, it can be determined by Tag of To / From header or the like.

  The distribution key includes a virtual node ID based on a node identifier (call-id in SIP) or a hash value. Since the hash value is a fixed-length value without regularity obtained from the original data by a certain calculation procedure, it is preferable that the virtual node ID and the node identifier are composed of hash values.

  On the other hand, in the case of a subsequent call as a result of the above-described determination of whether the call is a new call or a subsequent call, the distribution unit 18b allocates a distribution ID space that is an assigned region for each node 15 on the distribution ID table 19b { 4 (b) and will be described later} is compared with the hash value in the distribution key to identify the node 15 in charge. Further, the address of the node 15 in charge is specified from the node identifier management table 19a shown in FIG. 4A described later, and transferred to the specified node 15.

  On the other hand, as a result of the determination described above, since a distribution key does not exist in the case of a new call, a call-id (node identifier) is extracted from the message and introduced into a hash function to derive a hash value. Further, the distribution unit 18b compares the distribution ID space {shown in FIG. 4B and described later}, which is the assigned area for each node 15 on the distribution ID table 19b, with the derived hash value. The node 15 in charge is specified. Further, the address of the node 15 in charge is specified from the node identifier management table 19a shown in FIG. 4A described later, and transferred to the specified node 15.

  Even when a new call is received by the signal processing unit 18c, a Call-id (node identifier) is extracted from the message, is introduced into a hash function, a hash value is derived, and a distribution key is generated. Further, after the signal processing by the signal processing unit 18c, the distribution key is embedded in the message to be sent to the client 11 (To / From header tag in SIP) and sent.

  Thereafter, a message is sent after embedding the real distribution key in the subsequent call from the client 11, and the distribution unit 18b distributes the call based on the hash value of the main distribution key. Subsequent calls can reach the processed node 15.

Next, the node identifier management unit 18a described above will be described in more detail.
The node identifier management unit 18a updates the identifier information (node identifier) of the node 15 constituting the distributed system 10 when the node 15 is added to or removed from the distributed system 10, and this is updated as shown in FIG. The node identifier management table 19a shown in FIG. In the example of FIG. 4A, an address (for example, “10.45.0.1”) is associated with a node identifier (or node ID) (for example, “Node 1”). The node identifier is assigned by the node identifier management unit 18a of the privileged node and distributed to all the nodes 15.

  In the consistent hash method, the virtual node identifier (or virtual node ID) shown in FIG. 4B is subordinate to the node identifier. This virtual node ID is an arbitrary ID (by hash value) in the distribution ID space. For example, the node identifier “Node1” shown in FIG. 4A is subordinate to at least one or more virtual node IDs “Node1-1” and “Node1-2” shown in the distribution ID table 19b of FIG. 4B. doing. In other words, one or more virtual nodes are subordinate to the node 15. However, this is a basic configuration, and the virtual node may not be subordinate to the node 15 in some cases.

  Thus, in conjunction with the update of the node identifier management table 19a described above, the assigned area of the distribution ID space handled by the node 15 is updated and managed as the distribution ID table 19b. In the distribution ID table 19b, for example, the virtual node ID “Node1-1” is associated with the data size of “0 to 199 (D = 200)” as the assigned area of the assigned ID space, and the virtual node The data size of “600 to 999 (D = 400)” is associated with the ID “Node1-2” as the assigned area of the assigned ID space. That is, D = 200 indicates that the data size of the assigned area is 200. The same applies to other D = 400 and the like.

Next, information collection of the load measured by the node load measurement unit 18d described above and sending of the collected load to the privileged node will be described.
The node load measurement unit 18d measures the load on the node 15 at a predetermined cycle, and records and accumulates the load on the storage unit 19 as node load measurement data 19d. Further, the node load measurement unit 18d sends the node load measurement data 19d accumulated in the privileged node (for example, the node B shown in FIG. 3) at a predetermined cycle.

  However, the load of the node 15 measured at a predetermined cycle by the node load measuring unit 18d described above is any parameter that can be acquired by the node 15 such as a CPU (Central Processing Unit) usage rate, a memory usage rate, and an access frequency. used. Also, which numerical value is the bottleneck, and what value is the threshold to be rebalanced depends on the system characteristics of the distributed system, and there are cases where it is determined by a combination of multiple parameters. is there. Therefore, it can be used without being limited to a specific parameter type.

  Further, the unit of load measurement by the node load measuring unit 18d is measured in any one of (a) node unit by node ID, (b) virtual node ID unit by virtual node ID, and (c) data unit in FIG. It doesn't matter. In addition, when the load is measured in the virtual node unit of FIG. 5B, it is possible to calculate the node unit of (a) by summing up the load, and when measuring the load in the data unit of (c), It is possible to calculate the load in units of virtual nodes in (b) and in units of nodes in (a) by summing up. In FIG. 5, the load measured by the node load measuring unit 18 d shows an access frequency (number of accesses) as an example.

Such tables in FIGS. 5A to 5C are for the distributed system 10 including two nodes 15 and have the following configuration.
In one node 15 (for example, node ID = Node1) shown in FIG. 5A, two virtual nodes with two virtual node IDs (Node1_1, Node1_2) shown in FIG. 5B are held. Furthermore, it is assumed that two data (data1, data2) per virtual node with one virtual node ID (for example, Node1_1) shown in FIG. The same applies to the other nodes.

  In this case, the load measurement unit is a virtual node unit as shown in FIG. 5B, and the access frequency (number of times) as a load to be collected is a 10 second period (10: 15: 00 → 10: 15: 10). → 10:15:20) The case of collecting and accumulating is assumed.

  Next, a description will be given of processing for calculating the load deviation of the node 15 and rebalancing execution determination by the distributed system load rebalancing unit 18e.

  The rebalance unit 18e is based on the node load measurement data 19d collected from each node 15 at a predetermined period, and the average value, standard deviation, deviation, and deviation / standard deviation of the load of the nodes 15 of the entire distributed system 10 (standard deviation). The value divided by the deviation is calculated. Further, the rebalance unit 18e records the calculation results in the total data 19e as shown in FIG. 6 as an example, and based on the recorded total data 19e, the following three conditions (1) to (3) If any one of the above is satisfied, rebalancing is performed to correct the load imbalance among the nodes 15. If neither is satisfied, rebalancing is not performed. The condition (1) is the first condition described in the claims, the condition (2) is the second condition described in the claims, and the condition (3) is the third condition described in the claims.

  In the aggregated data 19e shown in FIG. 6, the collection time, node ID (node identifier), average value (access frequency), standard deviation, actual measurement value (access frequency), deviation, and deviation / standard deviation are recorded. . The average value and the actual measurement value are the average value and the actual measurement value of the access frequency.

  Condition (1), the rebalance unit 18e checks whether or not the load on the node 15 exceeds the upper limit value (predetermined upper limit value) permitted by the node 15 based on the total data 19e. When there is a node exceeding the upper limit value, rebalancing is executed.

  Condition (2), the rebalance unit 18e checks whether or not the standard deviation of the load of the entire node 15 is equal to or less than a predetermined threshold (first threshold) based on the total data 19e, and exceeds the threshold And rebalance.

  Condition (3), the rebalance unit 18e checks whether or not the load deviation / standard deviation for each node 15 is equal to or less than a predetermined threshold (second threshold) based on the aggregated data 19e. If so, perform a rebalance.

  However, in the aggregated data 19e shown in FIG. 6, the load measurement unit is the node unit shown in FIG. 5A (there may be a virtual node unit), and the load average value and the actual measurement value are the access frequency. Furthermore, the time interval for calculating the average value and the standard deviation is set to 20 seconds (for example, 10:14:40 to 10:15:00). The example of FIG. 6 is the calculated value and the like at the time of 10:15:00. Further, the upper limit value of the load permitted by the node 15 is the access frequency measured value “90” {condition (1)}, the standard deviation threshold is “15” {condition (2)}, and the deviation / standard deviation threshold ( This is an example when “1.2” {condition (3)} is also set as the deviation threshold.

  In this example, the conditions (1) and (2) are not satisfied. However, in FIG. 6, the deviation / standard deviation of the node identifiers “Node 1”, “Node 3”, and “Node 4” is “1.5”, which exceeds the threshold value “1.2” and satisfies the condition (3) Therefore, rebalancing is executed.

Here, the rebalancing executed by the rebalancing unit 18e will be described.
The rebalance corrects the load bias by transferring a transfer area (described later) in the assigned area (responsible ID space) of the node 15 having a high load to the node 15 having a low load. At this time, in order to correct the load divergence, the size of the necessary transfer area of the assigned area is estimated, and only the transfer area is transferred. However, the transfer area may be all of the assigned areas or an area with a ratio of less than 100% of the assigned areas.

This transfer method is as follows (T1) to (T4).
(T1) It is assumed that the transfer area in the assigned area of the node 15 having the highest load among all the nodes 15 is transferred to the lowest node 15.

  (T2) The transfer of the transfer area ends in the following case. That is, when all of the deviations that cause any of the above conditions (1) to (3) no longer exist (T2-1), or a part of the deviation (previously designated deviation cancellation ratio) The process ends when the transfer of the transfer area that resolves (deviation satisfying) is determined (T2-2), or when there is no transfer destination node 15 that can transfer the transfer area (T2-3). And

  (T3) The transfer unit of the transfer area may be a node unit or a virtual node unit, not a virtual node unit, but a unit for transferring half of the area in charge of the virtual node, or a transfer unit of only data using one hash value I do not care.

  (T4) The rebalancing design executed in advance when the rebalancing unit 18e performs rebalancing determines in which unit the load measurement unit is executed from among the node unit, the virtual node unit, and the data unit described above. The granularity of possible rebalance designs varies as described below.

That is, in the case of load measurement in node units, the method is only used when the rebalance granularity described later is coarse.
In the case of load measurement in units of virtual nodes, it is possible to use a method in which the rebalance granularity described later is coarse and the rebalance granularity is intermediate (intermediate between coarse and fine).
In the case of load measurement in units of data, all three methods can be employed when the rebalance granularity described later is coarse, intermediate, and fine.

First, the case where the rebalance particle size is coarse will be described.
An average load amount “Lv_ave” per virtual node obtained by dividing the total load amount of the entire node 15 by the number of virtual node IDs of all the nodes 15 is calculated. Next, paying attention to the node 15 with the highest load in the node 15, the deviation of the load amount to be eliminated for this node 15 (because the sign of this deviation is +, also referred to as a plus deviation) “Ltarget” is calculated. To do. Next, a value obtained by dividing “Ltarget” by “Lv_ave” is considered as the number of virtual node IDs “Vtarget_num” necessary for eliminating the load to be eliminated.

  A virtual node ID of “Vtarget_num” is extracted from the virtual nodes of the node 15 at random. At this time, when “Vtarget_num” includes a decimal number, as described in the outline in (T3) above, the area in charge of the virtual node having a predetermined virtual node ID may be assigned based on the decimal number, for example. For example, in the case of “1.5”, the assigned area of one virtual node is assigned and the assigned area of the second virtual node is divided in half. Further, an integer number of virtual node IDs may be extracted by rounding down, rounding up, or rounding off the decimal part.

  As described above, when the transfer area in the assigned area of the virtual node with the virtual node ID “Vtarget_num” extracted at random is transferred, the transfer is performed in order from the node 15 having the lowest load among all the nodes 15. I will do it. At this time, it is necessary to determine an allowable transfer size of the assigned area so that the load on the transfer destination node 15 does not increase too much due to the transfer.

  Specifically, the transfer destination node 15 can tolerate a load amount deviation (since the sign of this deviation is-, it is also referred to as a minus deviation). For this reason, the number of virtual node IDs “Vget_num1” required to eliminate the load amount, which is a value obtained by dividing the minus deviation of the load amount by the average load amount “Lv_ave”, is the allowable number of virtual node IDs.

  Here, when the assigned area of the transfer destination node 15 exceeds the allowable amount, an allowable virtual node ID number “Vget_num2” is obtained for the node 15 having the next lowest load in the same procedure, and Vtarget_num <Vget_num1 + Vget_num2 + When the transfer of the transfer area in all the necessary assigned areas is completed, the process ends.

  Thereafter, the same processing is executed for the next highest load node, and the transfer of the transfer area in the assigned area that corrects the load divergence is completed in all the nodes 15 that need to eliminate the load divergence. Or, it is executed until there is no node that can be transferred to the transfer area.

Next, a case where the rebalance granularity is intermediate will be described.
Since it is basically the same as the case where the rebalance granularity described above is coarse, only the difference from the coarse case will be described.
As described above, instead of randomly extracting virtual nodes from the virtual nodes of the node 15, virtual nodes that cause a positive deviation of the load to be eliminated are selectively extracted, and the extracted virtual nodes The transfer area in the node's area is transferred. In this case, since the load amount is measured in units of virtual nodes, the measured load is higher than when the granularity is coarse, but the transfer area can be transferred more accurately to correct the load divergence. It becomes.

Next, the case where the rebalance granularity is fine will be described.
Since it is basically the same as the case where the rebalance granularity described above is coarse, only the difference from the coarse case will be described.

  As described above, instead of randomly extracting the virtual nodes from the virtual nodes of the node 15, the hash value of the data causing the positive deviation of the load amount to be eliminated is selectively extracted, and the hash Only the value is transferred. In this case, the load is measured in units of data, so the measured load is higher than when the rebalance granularity is intermediate, but the transfer of the transfer area to correct the load divergence should be performed more accurately. Is possible. Also, the unit of transfer can be minimized.

  After the rebalancing unit 18e determines to execute the rebalancing, the rebalancing unit 18e considers the rebalancing design that corrects the load imbalance, reflects it in the distribution ID table 19b, and sends it to all the nodes 15.

  For example, as shown in FIG. 7A, the distribution ID table 19b before rebalancing is assigned to the virtual node ID “Node1-1” as “0-199 (D = 200) ”,“ Node 2-1 ”is“ 200 to 399 (D = 200) ”,“ Node 3-1 ”is“ 400 to 599 (D = 200) ”, and“ Node 1-2 ”. Are associated with a data size of “600 to 999 (D = 400)”.

  Here, for example, all the assigned areas “600 to 999 (D = 400)” of the virtual node “Node 1-2” illustrated in FIG. 7A are transferred to the virtual node having the other virtual node ID “Node3_2”. And In this case, as shown in FIG. 7B, the area in charge of the virtual node with the virtual node ID “Node3_2” has a size of “600 to 999 (D = 400)”.

  Also, half of the assigned area “600 to 999 (D = 400)” of the virtual node “Node 1-2” illustrated in FIG. 7A is transferred to the virtual node having the other virtual node ID “Node3_2”. To do. In this case, as shown in FIG. 7C, the virtual node with the virtual node ID “Node1_2” has a size of “600 to 799 (D = 200)”, and the virtual node with the virtual node ID “Node3_2” The area in charge is “800 to 999 (D = 200)”.

<Operation of First Embodiment>
Next, in the distributed system 10 according to the first embodiment, the operation (first rebalancing operation) when executing the rebalancing to correct the load imbalance between the nodes 15 will be described with reference to FIGS. I will explain.

  First, preconditions for executing the rebalance will be described with reference to FIG. As shown in FIG. 8, the hash space is divided into the virtual nodes A1 to A6, B1 to B5, and C1 to C5 indicated by the three nodes A to C, and the virtual nodes A1 to A6, B1 to B5, C1 to C5 are divided. The area of responsibility is determined and managed. However, A to C are node IDs, and A1 to A6, B1 to B5, and C1 to C5 are virtual node IDs.

  For example, the hash space (responsible area) handled by the virtual node B1 is an area from the virtual node B1 to the virtual node A1 in the clockwise direction, and the responsible virtual node B1 holds data existing in this responsible area. The same applies to the other virtual nodes A1 to A6, B2 to B5, and C1 to C5.

  In addition, each of the virtual nodes A1 to A6, B1 to B5, and C1 to C5 holds load data in the assigned area. In this example, low load data with a load amount “1” indicated by a symbol “で”, Assume that an arbitrary number of medium load data with a load amount of “2” and an arbitrary number of high load data with a load amount of “3” indicated by a symbol ● are retained.

  For example, the virtual node B1 holds three pieces of high load data ●, and holds a total of “9” load data. In the case of the virtual node A1, two pieces of high load data ● and one piece of medium load data ◎ are held, and a total of “8” load data is held. As described above, the load data is also held in the other virtual nodes A2 to A6, B2 to B5, and C1 to C5.

  FIG. 9 shows a table of load data held by the virtual nodes A1 to A6, B1 to B5, and C1 to C5. In the load data table of FIG. 9, nodes A, B, and C are shown in the top row column, and virtual node IDs “1” to “6” are shown in order from the top in the leftmost column.

  However, in node A in the top row column, the load amount held in the assigned area is (40), the deviation / standard deviation is “+1.39”, which is larger than the threshold value “1.2”, and is a high load. This is shown as a precondition. Node B indicates that the load amount is (21), the deviation / standard deviation is “−0.93”, which is smaller than the threshold value “1.2”, and is a medium load. The node C indicates that the load amount is (25), the deviation / standard deviation is “−0.44”, which is smaller than the threshold value “1.2”, and is a medium load as a precondition.

  In the load data table, the column “1” of the node A indicates the load data of the virtual node A1, and in this example, there are two “high (3)” indicating the load amount of the high load data ●. One “medium (2)” indicating the load amount of the medium load data ◎ is shown. This aspect is the same in the columns of the virtual nodes A2 to A6, B1 to B5, and C1 to C5 of the other nodes A to C.

  FIG. 10 is a bar graph showing the deviation of the load amount held by each of the nodes A to C. The total load amount of each node A to C shown in this bar graph is “86”, the average load amount per node A to C is “28.6”, the standard deviation is “8.17”, and the deviation threshold is “1.2”. ”, The number of virtual nodes is“ 16 ”, the average load amount per virtual node is“ 5.375 ”, the deviation of node A is“ +11.4 ”, the deviation of node B is“ −7.6 ”, The deviation is “−3.6”.

  Further, the load amount of each of the nodes A to C can be regarded as the sum of the loads in the area in charge of all the virtual nodes of one node (for example, A).

Under such preconditions, the operation when rebalancing the nodes A to C will be described with reference to the flowcharts shown in FIGS.

  First, in step S1 shown in FIG. 11, the distributed system load rebalancing unit 18e of a predetermined node (any one of A to C) is based on the node load measurement data 19d collected from each node A to C at a predetermined cycle. The average value, standard deviation, deviation, and deviation / standard deviation of the loads of the nodes A to C are calculated.

  Next, in step S2, the rebalance unit 18e records the calculation result in step S1 in the storage unit 19 as total data 19e (see FIG. 6).

  In step S3, the rebalance unit 18e determines whether any one of the three conditions (1) to (3) described above is satisfied based on the data 19e recorded in step S2. As a result, if not satisfied (No), the rebalancing process is terminated.

  On the other hand, as a result of the determination, for example, it is assumed that only the condition (3) is satisfied (Yes). In this case, in step S4, the rebalancing unit 18e compares the deviation / standard deviation of each of the nodes A to C with the deviation threshold value “1.2”, so that the node A has a high load as shown in FIG. It is detected that the node “+11.4” and the nodes B and C are medium load nodes “−7.6” and “−3.6”. As a result, the rebalance unit 18e prepares for rebalancing to correct the load bias by transferring the transfer area in the area in charge of the high load node A to the other nodes B and C.

  At this time, in step S5, the rebalance unit 18e obtains the size (= load amount) of the transfer area in the assigned area of the transfer source node. The load bias to be eliminated at the high load node A is “+11.4” when all the deviations of the node A are assumed. The assigned size of the virtual node necessary to eliminate this deviation can be calculated by the deviation “+11.4” ÷ average load amount per virtual node “5.375”. That is, 11.4 ÷ 5.375 = 2.12, and it is necessary to transfer the transfer area in the area in charge of 2.12 virtual nodes.

  Next, in step S6, the rebalance unit 18e obtains an allowable size of the area in charge of the transfer destination nodes B and C for transferring the transfer area in the area in charge of the transfer source node A. The transfer destination node accepts transfer from the node B having the lowest load. The allowable area size of the transfer destination node B is the deviation “−7.6” ÷ the average load amount per virtual node “5.375. ”Is 1.41 virtual nodes. In addition, the allowable area size of the transfer destination node B is 0.67 virtual nodes because of the deviation “−3.6” ÷ the average load amount per virtual node “5.375”.

  Next, in step S7 shown in FIG. 12, the rebalancing unit 18e determines whether there are transfer destination nodes B and C that can transfer the transfer target area of the transfer source node A, that is, transfer areas in the transfer area. Determine whether. This determination is performed by detecting whether or not the allowable size of the assigned area of the transfer destination nodes B and C can transfer the transfer area in the assigned area of the transfer source node A. As a result, if there are no transfer destination nodes B and C that can be transferred (No), the rebalancing process is terminated.

  On the other hand, if there are transfer destination nodes B and C that can be transferred (Yes), the rebalancing unit 18e transfers the transfer area in the area in charge of the transfer source node A to the transfer destination node B in step S8. In this transfer, for example, 2.12 assigned areas are transferred from the transfer source node A to the transfer destination node B having a vacant area of 1.41 assigned areas. 1.41 = 0.71 virtual nodes remain. In this case, the transfer is performed in units of virtual nodes.

  This will be described by replacing it in the case where data is transferred in units of data in the hash space shown in FIG. For example, “8” load data of the virtual node A1 in the transfer source node A is transferred to the low load virtual node B3 of the load amount “2” in the transfer destination node B. In this case, the transfer source node A has a load amount of 11.4−8 = 3.4.

  Thereafter, in step S9 shown in FIG. 12, the rebalance unit 18e determines whether or not there is a remaining transfer area in the assigned area of the transfer source node A. As a result, if there is no remaining (No), in other words, if all of the transfer target areas have been transferred, the rebalancing process ends.

  On the other hand, if there is a remainder (for 0.71 above) in the determination of step S9 (Yes), the rebalancing unit 18e can transfer the remaining transfer area of the transfer source node A in step S10. It is determined whether there is a transfer destination node. As a result, if there is no transfer destination node (No), the rebalancing process is terminated.

  As a result of the determination in step S10, if there is a transfer destination node to which the remaining transfer area can be transferred (Yes), in step S11, the rebalance unit 18e determines the remaining transfer source node A (for 0.71). Are transferred to the transfer destination node C whose existence is recognized in step S9. With this transfer, 0.71-0.67 = 0.04 remains in the transfer source node A.

  Thereafter, returning to step S9, the rebalancing unit 18e determines whether there is a remaining transfer area in the assigned area of the transfer source node A. In this case, since there is a remaining portion of 0.04 (Yes), in step S10, the rebalancing unit 18e determines whether or not there is a transfer destination node to which the remaining transfer area of the transfer source node A can be transferred. Determine. In this case, since there are no transferable nodes other than the nodes B and C (No), the rebalancing process is terminated. Here, although the rebalancing process ends, since only 0.04 loads of virtual nodes remain in the transfer source node A, the load divergence between the nodes A to C is reduced. .

  In the above description, the transferable load amount of the transfer source node A is a difference from the average load amount (for example, “+11.4” in the node A), but may be arbitrarily determined. For example, the transferable load amount may be “+5”.

<Effects of First Embodiment>
The distributed system 10 according to the first embodiment described above includes a plurality of nodes 15 to which information from the client 11 is distributed by a load balancer 13 connected to a plurality of client machines 11 using a communication service via a network 12. Configured.

  A feature of the first embodiment is that the node 15 includes a rebalance unit 18e. The rebalance unit 18e calculates the standard deviation and the deviation / standard deviation of the load amounts of all the nodes 15. Next, the rebalancing unit 18e includes a condition (1) indicating that the load on the node 15 exceeds a predetermined upper limit value, and the standard deviation of the loads on all the nodes 15 exceeds a predetermined first threshold value. It is determined whether or not any one of the condition (2) indicating presence or the condition (3) indicating that the deviation / standard deviation for each node exceeds a predetermined second threshold is satisfied. When it is determined that the load balance is satisfied, the rebalancing unit 18e allows the node 15 to hold a predetermined load capacity transfer area among the assigned areas in which the load can be held, so that the deviation of the load amount between the nodes 15 is suppressed. A process of transferring from a transfer source node having a high load (for example, node A) to a transfer destination node having a low load (for example, node B) is performed. In addition to the above, the rebalance condition may satisfy any one of the conditions (1) to (3).

  As a result, when any one or a plurality of the conditions (1) to (3) described above are satisfied, the transfer area having a predetermined load capacity in the assigned area of the node is transferred from the node having a high load. Since the transfer is performed to a node having a low load, it is possible to reduce the divergence of the load amount between the nodes without taking time and effort. Therefore, rebalancing can be performed efficiently.

  In addition, when there is a transfer destination node B having a load capacity capable of transferring the transfer area of the assigned area of the transfer source node A, the rebalance unit 18e transfers the assigned area to the transfer destination node B.

  As a result, since the assigned area is transferred only when there is a transfer destination node B having a load capacity capable of transferring the transfer area, it is possible to determine whether or not to perform rebalancing without taking time and effort. Can do.

  When the rebalancing unit 18e transfers the transfer area of the transfer area of the transfer source node A to the transfer destination node B, the transfer area is completely transferred, or the transfer destination node b to which the transfer area can be transferred is The transfer was done until it no longer exists.

  Thereby, it is possible to eliminate or reduce the deviation of the load amount between the nodes 15.

<Configuration of Second Embodiment>
A distributed system according to a second embodiment of the present invention will be described. The difference between the distributed system of the second embodiment and the distributed system 10 of the first embodiment is the difference in the rebalancing process performed by the distributed system load rebalancing unit 18e.

A feature of the second embodiment is that the rebalance unit 18e performs the following processing. That is, even though the total amount of resources (total load) of each node 15 of the distributed system 10 is sufficient with respect to the used resource amount (used load amount), the used resource amount is biased. . At this time, the rebalancing unit 18e does not specify the target node to transfer the transfer area in the assigned area or the appropriate transfer size of the corresponding node 15 as in the first embodiment, and the virtual that each node 15 has Rebalancing is performed so that the number of nodes is reset to the number of virtual nodes that is required for each node 15 according to the current load status for each node 15.
At this time, the rebalancing unit 18e calculates the number of virtual nodes of each node 15 according to the following equation (1) according to the load condition, and rebalances based on the calculated number of virtual nodes of each node 15.

  In this rebalancing, the virtual node ID and the distribution ID space are remapped from the top of the distribution ID space of each virtual node based on the calculated number of virtual nodes. In the remapping, the total extension (total size) of the assigned area is divided by the calculated number of virtual nodes to obtain a distribution ID space size per virtual node, and a new distribution ID space size from the top of the distribution ID space. Each time, the number of virtual nodes for each virtual node ID is reset.

Number of virtual nodes after rebalancing = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of relevant nodes) (Equation 1)
However, the “actually measured load value of the corresponding node” in the equation (1) is an actually measured value of the load of the node having the current virtual node. In addition, Expression (1) is held in a storage unit (not shown) of the rebalance unit 18e.

The rebalancing process of the second embodiment will be specifically described.
First, the rebalancing unit 18e resets the number of virtual nodes that the node 15 has. This resetting process will be described with reference to FIGS. 13 (a) and 13 (b). However, the virtual node IDs “Node 1-1” and “Node 1-2” illustrated in FIGS. 13A and 13B are virtual nodes 1-1 subordinate to the node 1 (not shown) having the node ID “Node 1”. , 1-2 (not shown). The same applies to other virtual node IDs. For example, the virtual node IDs “Node5-1”, “Node5-2”, and “Node5-3” are subordinate to the node 5 (not shown) of the node ID “Node5”. It corresponds to virtual nodes 5-1, 5-2, 5-3 (not shown).

  In the distribution ID table 19b shown in FIG. 13A, the virtual node ID “Node1-1” corresponds to the data size “0 to 199 (D = 200)” as the assigned area of the assigned ID space. A data size of “200 to 399 (D = 200)” is associated with “Node 1-2”. The same applies to other virtual node IDs.

  Further, the number of virtual nodes of each of the nodes 1 to 5 is that the number of virtual nodes of node 1 is 2, the number of virtual nodes of node 2 is 1, the number of virtual nodes of node 3 is 1, and the number of virtual nodes of node 4 is Two and the number of virtual nodes of the node 5 is two.

  Under such conditions, the rebalancing unit 18e resets the number of virtual nodes of the own nodes 1 to 5 to a necessary load amount according to the current load status. Hereinafter, the resetting process will be described.

  First, the rebalance unit 18e changes the number of virtual nodes of each of the nodes 1-5. For example, as shown in FIG. 13A, the current number of virtual nodes for each of the nodes 1 to 5 is two for node 1, one for node 2, one for node 3, two for node 4, 5 is a total of 8 pieces. As shown in FIG. 13B, this corresponds to one node 1, two nodes 2, two nodes 3, two nodes 4, depending on the current load of each node 1 to 5. Node 5 changes to a total of 10 nodes.

  The above formula (1) is used when changing the number of virtual nodes. For example, the number of virtual nodes is changed as shown in FIG. 13B, for example, when the number of nodes 1 to 5 shown in FIG. 13A is “2, 1, 1, 2, 2” = 8. The number of nodes 1 to 5 is changed to “1, 2, 2, 2, 3” = 10.

  In the current state of FIG. 13A, the hash space size (size of the assigned area) of all the nodes 1 to 5 is 1600 of “0 to 1599” and the number of virtual nodes is 8, so the assigned area per virtual node is The size D is 1600 ÷ 8 = 200. Of the size of the assigned area of D = 200, the actual measured value of the load of the corresponding node 1 is, for example, “80” or “150”. Since the average value of the loads of all the nodes 1 to 5 is obtained from the actually measured values of all the nodes 1 to 5, the average value and the actually measured value are substituted into the above formula (1).

  For example, in the assigned area (size D = 200) of the distribution ID space of virtual node ID = “Node 1-1”, the actual load value is “140”, and in “Node 1-2”, the actual load value is “160”. If there is, the measured value of the load of the node 1 is “300”. At this time, the average value of the loads of all the nodes 1 to 5 is “150”. In this case, the number of virtual nodes after the rebalancing of node 1 is 2 × (150/300) = 1. Similarly, the number of virtual nodes after rebalancing is obtained in the other nodes 2 to 5, and the number of virtual nodes of each node 1 to 5 is changed to the obtained number of virtual nodes. However, when the calculation result fitted to the formula (1) includes a decimal point such as 1.6, it is determined in advance that rounding up, rounding down, and rounding off are performed.

  Next, the rebalance unit 18e changes the hash space size per virtual node. In the current state shown in FIG. 13A, as described above, the hash space size D per virtual node is 1600/8 = 200.

  If the number of virtual nodes after change = 10 is used, the hash space size per virtual node is 1600/10 = 160. Using this hash space size, as shown in FIG. 13B, the hash space size D of each virtual node is set to “160”.

  Next, the rebalance unit 18e allocates the virtual nodes having the changed hash space size D = “160” by the number of virtual nodes of the respective nodes 1 to 5 after the above change. That is, since there is one changed virtual node in the node 1, as shown in FIG. 13B, in the node 1, the changed virtual node of the hash space size D = “160” {virtual node ID “Node1- 1 "} is allocated.

  Similarly, since there are two virtual nodes after the change in node 2, two virtual nodes {virtual node IDs “Node2-1, Node2-2”} of size D = “160” are allocated. Thereafter, as shown in the figure, two to three virtual nodes after the change are allocated to the nodes 3 to 5.

<Effects of Second Embodiment>
The feature of the distributed system of the second embodiment described above is that the rebalancing unit 18e of the node 15 holds the above-described formula (1). Further, the rebalance unit 18e obtains the current number of virtual nodes, the average value of the loads of all the nodes 15, and the actual measured value of the load of the node 15 having the current virtual node according to the load status of each node 15. By applying these calculated numerical values to the equation (1), the number of virtual nodes for each node 15 is calculated. Then, the rebalancing unit 18e changes the load capacity per virtual node of the current virtual node, and the corresponding node 15 so that there are as many virtual nodes having the changed load capacity as the calculated number of virtual nodes. Rebalance to be performed.

  As a result, the rebalancing unit 18e determines the number of virtual nodes that each node 15 has for each node 15 without specifying the target node 15 to which the transfer area in the assigned area is transferred or the appropriate transfer size of the corresponding node 15. Depending on the current load status, the number of virtual nodes for obtaining the required load amount can be obtained. That is, rebalancing suitable for the required load amount of the node 15 can be performed.

  In addition, about a concrete structure, it can change suitably in the range which does not deviate from the main point of this invention.

DESCRIPTION OF SYMBOLS 10 Distributed system 11 Client machine 12 Network 13 Load balancer 14 Cluster 15 Node 18 Control part 18a Node identifier management part 18b Distribution part 18c Signal processing part 18d Node load measurement part 18e Distributed system load rebalancing part 19 Storage part 19a Node identifier management Table 19b Distribution ID table 19c Data 19d Node load measurement data 19e Distributed system load summary data 19f Call control status flag

Claims (3)

A distributed system having a plurality of nodes to which information from a plurality of client machines using a communication service is distributed via a network,
The node stores the following formula (1) in the storage unit,
Number of virtual nodes = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of nodes having current virtual nodes) (1)
In accordance with the load status of each node, the current number of virtual nodes, the average value of the loads of all the nodes, and the actually measured value of the load of the node having the current virtual node are obtained, and these obtained numerical values are represented by the formula (1). ) To calculate the number of virtual nodes for each node,
The size of the assigned area per virtual node of the current virtual node is changed, and a virtual node having the size of the assigned area after the change exists in the corresponding node so that there are as many virtual nodes as the calculated number of virtual nodes. A distributed system comprising a rebalancing unit for performing rebalancing. A load balancing method having a plurality of nodes to which information from a plurality of client machines using a communication service is distributed via a network,
The node is
The following mathematical formula (1) is stored in the storage unit,
Number of virtual nodes = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of nodes having current virtual nodes) (1)
In accordance with the load status of each node, the current number of virtual nodes, the average value of the loads of all the nodes, and the actually measured value of the load of the node having the current virtual node are obtained, and these obtained numerical values are represented by the formula (1). ) To calculate the number of virtual nodes for each node,
The size of the assigned area per virtual node of the current virtual node is changed, and a virtual node having the size of the assigned area after the change exists in the corresponding node so that there are as many virtual nodes as the calculated number of virtual nodes. And a step of performing rebalancing to allocate a load balancing method. A computer storing the following formula (1) in a storage unit as a node of a distributed system having a plurality of nodes to which information from a plurality of client machines using a communication service is distributed via a network,
Number of virtual nodes = current number of virtual nodes × (average value of loads of all nodes / actual value of loads of nodes having current virtual nodes) (1)
In accordance with the load status of each node, the current number of virtual nodes, the average value of the loads of all the nodes, and the actually measured value of the load of the node having the current virtual node are obtained, and these obtained numerical values are represented by the formula (1). ) To calculate the number of virtual nodes for each of the nodes,
The size of the assigned area per virtual node of the current virtual node is changed, and a virtual node having the size of the assigned area after the change exists in the corresponding node so that there are as many virtual nodes as the calculated number of virtual nodes. A program to make it function as a means of rebalancing.

Images