KR101042908B1 - Method, system, and computer-readable recording medium for determining major group under split-brain network problem - Google Patents

Abstract

The present invention relates to a method, system, and computer readable recording medium for determining a major group in a network separation failure situation occurring in a network-based distributed computing environment. According to an aspect of the present invention, a method for determining a major group of at least two groups in which an initial group consisting of a plurality of nodes is separated from each other in a network-based distributed environment, the number of nodes of the first group and the second group Comparing the number of nodes of the group, obtaining history information of the node of the first group and the node of the second group, and using the at least one of the node number comparison result and the history information; A method is provided that includes determining a major group as a group or the second group.

Description

METHOOD, SYSTEM, AND COMPUTER-READABLE RECORDING MEDIUM FOR DETERMINING MAJOR GROUP UNDER SPLIT-BRAIN NETWORK PROBLEM}

The present invention relates to a method, system, and computer readable recording medium for determining a major group in a split-brain failure situation that occurs in a network-based distributed computing environment (or distributed environment). More particularly, the present invention relates to a method, system, and computer readable recording medium for determining a major group by referring to historical information of nodes in a network separation failure situation.

In today's IT environment, there is a need for efficient management and control of various network-based distributed systems. In a distributed environment, it is common to connect a large number of computers physically or logically, and to solve huge computational problems by using the processing power of the plurality of connected computers. At this time, if there are dozens of computers or processes participating in the distributed environment, the distributed environment may be properly maintained by an operator managing the distributed environment. However, if the distributed environment includes a scale of thousands or tens of thousands of computers or processes, For this reason, a management system, which is a solution that can synchronize and coordinate overall, is essential.

The most important condition in the management system is the guarantee of fault tolerance, especially in the case of failure in which the network is abnormally separated due to the instability of the link between each node constituting the cluster. It is one of the important things.

1 is a diagram illustrating a network separation failure situation by way of example. In FIG. 1, the distributed environment management system includes a cluster composed of five servers, and each server constituting the cluster is illustrated assuming that the same state information is maintained by replicating data in real time.

If a network isolation failure occurs in the state of FIG. 1 and the cluster is divided into two server groups, the servers in group # 1 will think that all servers in group # 2 are down (ie, disabled), At the same time, servers in group # 2 may think that servers in group # 1 are down. Thus, the two groups on the network determine themselves as a major group, and clients connect to each of the two main groups to judge themselves. For example, client group # 1 accessing group # 1 will read or write as usual, and client group # 2 accessing group # 2 will also read or write.

However, in the above situation, when the network separation failure is recovered and divided into two groups to be merged again, there is no way of knowing which group of data is correct data and which group of data should be deleted. That is, according to the related art, there is a disadvantage that a problem occurs in service continuity or data consistency in a network separation failure situation.

It is an object of the present invention to solve all the problems described above.

It is another object of the present invention to provide a solution that can collectively synchronize and coordinate multiple computers participating in distributed computing.

In addition, another object of the present invention is to ensure maximum service continuity and data consistency even in the event of network separation failure.

Representative configuration of the present invention for achieving the above object is as follows.

According to an aspect of the present invention, a method for determining a major group of at least two groups in which an initial group consisting of a plurality of nodes is separated from each other in a network-based distributed environment, the number of nodes of the first group and the second group Comparing the number of nodes of the group, obtaining history information of the node of the first group and the node of the second group, and using the at least one of the node number comparison result and the history information; A method is provided that includes determining a major group as a group or the second group.

According to another aspect of the present invention, there is provided a distributed environment management system composed of a plurality of nodes, comprising history information of the plurality of nodes, wherein at least one of the plurality of nodes has an initial group divided into at least two groups. In this case, there is provided a distributed environment management system for determining a major group into one of the at least two groups by using at least one of the number of nodes of each group and the history information.

In addition, there is further provided a computer readable recording medium for recording another method, system for implementing the present invention, and a computer program for executing the method.

According to the present invention, there is provided a solution that can collectively synchronize and coordinate a plurality of computers participating in distributed computing.

According to the present invention, even in the event of network separation failure, service continuity and data consistency can be guaranteed to the maximum.

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

DETAILED DESCRIPTION Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention.

[Preferred Embodiments of the Invention]

Configuration of the entire system

2 is a view schematically showing a distributed environment according to an embodiment of the present invention.

As shown in FIG. 2, the entire system according to an embodiment of the present invention includes a communication network 100, a distributed environment management system 200 that provides a locking service in a distributed environment, and a distributed environment management system. It may be composed of a plurality of clients 300 using the services provided by the 200.

First, the communication network 100 may be configured regardless of communication modes such as wired and wireless, and may include a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). Network).

According to one embodiment of the invention, the distributed environment management system 200 may be composed of a number of servers (hereinafter, commonly referred to as "nodes"), a plurality of clients 300 sharing a variety of system resources The lock service may be provided to the client 300 in order to ensure the consistency of data therebetween.

In addition, the distributed environment management system 200 performs a function of creating / maintaining / releaseing a session so that the client 300 can communicate with a specific node within the distributed environment management system 200, and communicates with the client 300. When a failure occurs in a specific node, the client 300 may provide a switching service to communicate with another node that is waiting.

In addition, the distributed environment management system 200 may perform a function of synchronizing a plurality of nodes and processing various events.

According to one embodiment of the invention, the distributed environment management system 200 includes a cluster manager, an event manager, and optionally a client library (not shown). can do.

According to an embodiment of the present invention, the cluster manager may perform a function of managing all nodes in the cluster, which is a set of nodes of the distributed environment management system 200. More specifically, the cluster manager may perform a role of managing nodes in the cluster and a lock operation while communicating with a client library. In addition, when the client 300 attempts a lock operation on a specific path or data, the cluster manager checks whether the requesting client 300 has the authority to perform the lock operation and immediately (so-called non- blocking mode) Returns whether the lock is acquired as a value of True or False. In addition, the cluster manager may perform a function related to the locking operation processing of the client 300 or the event processing associated with the locking operation. For example, depending on the type of the lock, the cluster manager may determine that the path or data is locked. An event for notifying the client 300 may be generated and a function of delivering the event to the event manager may be performed.

According to one embodiment of the present invention, communication with the client library can be made only through the master cluster manager, and other auxiliary cluster managers can maintain a copy of the state information of the master cluster manager in real time. In the event of a failure of the master cluster manager, a new master cluster manager may be selected among the secondary cluster managers.

Next, according to an embodiment of the present invention, like the cluster manager, the event manager is composed of one master event manager and several auxiliary event managers, and manages the list of clients 300 to be notified of the event, When is generated, it may perform a function for delivering a notification about the event to the client (300). More specifically, the event manager can largely perform two events, a lock event and a generic event. Looking at the process of processing the events in more detail, first, in the case of a lock event, the master cluster manager delivers information about the lock event to the master event manager, the event manager to multi-registered multiple client 300 It can be handled by sending an event. Next, in the case of a general-purpose event, regardless of the locking operation, the client 300 may be processed by requesting or registering the event manager to generate an event in a general publish-subscribe manner and receiving an event from the event manager.

In addition, according to an embodiment of the present invention, the client library is a program provided in the form of a library so that the client 300 can easily use the lock service and the event service of the distributed environment management system 200, and at least part of It may be included in the distributed environment management system 200. Of course, if necessary, at least a part of the client library may be included in the client 300.

According to an embodiment of the present invention, the client 300 communicates with the distributed environment management system 200 by using a function provided by the client library. More specifically, the client library is a collection of libraries that can be provided to the client 300 to use the service server 200 remotely. According to an embodiment of the present invention, a user can insert and use this library as a plug-in in his or her own program.In the case of initialization, a proxy object is created to connect to a cluster to form an RPC. You can also perform a lock operation. The client library abstracts the user into various situations, such as master node election, fail-over, and member changes, that occur within a remote cluster, allowing users to use secure and abstracted remote services just as they would with local services. Perform a function to help you.

Provision of lock service

In a distributed environment, many processes or threads often perform a single task collaboratively, and multiple transactions in a process or thread can simultaneously read or read a single piece of data. Since wrong results can be obtained when performing write operations, it is necessary to protect data that can be accessed by multiple transactions in common. The same can be applied to the case where each computer included in the distributed environment shares data with each other. Therefore, even in a distributed environment, a lock function may be required to prevent other transactions from accessing data used by one transaction.

Therefore, hereinafter, the lock service provided by the distributed environment management system 200 to the client 300 will be described in detail according to an embodiment of the present invention.

3 is a diagram exemplarily illustrating a locking function provided by the distributed environment management system 200. Meanwhile, FIG. 3 is assuming that a service is simultaneously executed in three clients 300.

Referring to FIG. 3, when client 1 310, client 2 320, and client 3 330 want to perform a read or write operation on shared data, distributed environment management according to an embodiment of the present invention. It can be seen that the master cluster manager of the system 200 can request a lock on the corresponding data. When Client 1 310 of FIG. 3 requests a lock, if Client 2 320 and Client 3 330 have not requested a lock on their data, Client 1 310 may acquire the lock without difficulty. There will be. However, as shown in FIG. 3, if at least two or more clients competitively requested a lock, the lock may be granted to only one of them.

As a way to prepare for the case where each client 310, 320, 330 competitively requests a lock, the distributed environment management system 200 until each client 310, 320, 330 acquires the lock. You can assume that you are requesting a lock continuously, which can be realized using the following program code:

/ ** Wait in the while loop until the lock is obtained ** /
while (! pleiades.getLock (service_A.LockMode.LOCK_EXCLUSIVE)) {
try {
/ ** Wait 1 second if lock is not obtained ** /
Thread.sleep (1000);
} catch (InterruptedExeption ie) {
return;
}
}

When a lock is granted to a specific client 300, the master cluster manager may record the lock information of the client 300 in a storage device such as a database or a memory.

In addition, as described above, since the lock information updated by the master cluster manager is replicated to the secondary cluster manager periodically or aperiodically, consistency between information recorded in the master cluster manager and information recorded in the secondary cluster manager is maintained. Can be.

According to an embodiment of the present invention, the lock obtained from the distributed environment management system 200 may be licensed only to the process or thread that requested the lock. In addition, according to another embodiment of the present invention, the lock obtained from the distributed environment management system 200 may be licensed to all processes or threads in the client 300 that acquired the lock.

Election of Master Nodes in a Network Separation Failure

As mentioned above, guaranteeing failure prevention or fault tolerance is essential as a condition of distributed computing. According to an embodiment of the present invention, lock information and client 300 of the client 300 participating in the distributed environment may be required. Event notification processing is managed by the cluster manager and the event manager included in the distributed environment management system 200. Therefore, when the cluster manager or the event manager fails, the operations of all the clients 300 participating in the distributed environment Problems may occur that cannot be performed normally.

Accordingly, the distributed environment management system 200 of the present invention includes a plurality of cluster managers, and sets one of them as a master cluster manager to perform direct communication with the client, while other auxiliary cluster managers are configured as master cluster managers. By duplicating and storing the lock information of the manager, even if a failure occurs in the master cluster manager, the service continuity or the data consistency can be guaranteed using one of the auxiliary cluster managers. The above configuration may be equally applied to the event manager (hereinafter, the term "master node" will be used to encompass the master cluster manager and the event cluster manager).

However, the above configuration alone cannot adequately cope with the network separation failure. As described above, each group divided in the network separation failure situation determines that all nodes of the other group are down. This is because a master node is created for each group, and therefore, after a failure is repaired, a problem may occur in the coherence of data between nodes belonging to each group.

Therefore, hereinafter, a method of selecting a master node in a network separation failure situation according to an embodiment of the present invention will be described in more detail.

1. Election of Quorum-Based Master Nodes

According to an embodiment of the present invention, the distributed environment management system 200 selects a master node by applying a quorum algorithm when a network separation failure is detected. Hereinafter, with reference to Figure 4 will be described in more detail.

4 is a diagram illustrating a quorum algorithm applied to a network separation failure situation.

Referring to FIG. 4, it can be seen that a network separation failure occurs in a cluster composed of a total of seven nodes, and thus divided into a group A including four nodes and a group B including three nodes.

In this situation, group A has a node corresponding to a majority of the total number of nodes (that is, seven), so that the quorum satisfies the quorum and may be classified as a major group. Accordingly, one of the nodes included in the major group is designated as the master node, and communication with the client 300 may be performed through the corresponding master node. In addition, the master node may be allowed to read or write data.

On the other hand, group B is classified as a minor group because it does not have a node corresponding to a majority of the total number of nodes. Master nodes will not be elected from these minor groups.

In order to determine whether each group is a major group or a minor group in a network isolation failure situation, the information on how many nodes existed in the initial group before the network isolation failure occurred is displayed. According to an embodiment of the present invention, since the information stored in the master node is replicated to the corresponding secondary node, the above request is stored by storing the above information in the master node. Will be achieved.

Next, in a situation where a network separation failure occurs as shown in FIG. 4, a network separation failure occurs once again, so that group A is divided into a group A1 including three nodes and a group A2 including one node, and group B is divided into two. Suppose the situation is divided into a group B1 containing 1 node and a group B2 containing 1 node, since no group contains more than 4 nodes corresponding to a majority of the number of nodes that belonged to the initial group. There may be a problem that can not specify a group that satisfies.

In order to solve this problem, according to an embodiment of the present invention, the quorum may be dynamically reset according to the change in the number of nodes constituting the group. More specifically, in order to implement dynamic quorum, first set a reset time of the group. According to one embodiment of the invention, the reset time of the group may be set to 30 seconds, in which case the group is reset based on this time point if the number of nodes in the group has not changed for 30 seconds. As the group is reset, the number of nodes in the group on which the quorum is calculated is initialized, and thus the quorum can be changed.

In the above example, the quorum is set to 4 since the total number of nodes included in the group is 7 before the first network detach failure occurs. Then, when the first network separation failure occurs, the quorum is reset to 3 because four nodes are included in group A, and the quorum is reset to 2 because three nodes are included in group B. Accordingly, even when a second network separation failure occurs, group A1 and group B1 can satisfy the quorum.

However, group B1 was classified as a minor group at the time of the first network separation failure. Therefore, even if a second network separation failure occurred, the majority of nodes could not be classified as a major group, and only group A1 was classified as a major group. Can be. Therefore, for this purpose, the master node can store information about how many nodes existed in the group before the network separation failure, as well as whether the group was a major group or a minor group before the network separation failure occurred. have.

Subsequently, the process in the case where the network separation failure is recovered will be described with reference to FIG. 4, and the nodes included in the group B, which is the minor group, perform the update by obtaining the latest state information from the node included in the group A, which is the major group, Thus, a total of seven nodes could again function as a group.

In this manner, data consistency can be guaranteed even in a network separation failure situation.

However, if only the quorum algorithm is adopted as described above, in some cases, there is a disadvantage in that service continuity or data consistency cannot be guaranteed. For example, if a group consisting of six nodes in total is divided into two groups each containing three nodes, no group can contain a majority of the nodes and thus service cannot continue. In addition, due to the continuous network separation failure, the service cannot be continued even when a group including one node is created. That is, when quorum is applied in a static manner, in particular, the following restrictions are applied in terms of service continuity.

Total number of nodes in initial group: 2N + 1

Minimum number of quorums: N + 1

Minimum number of nodes that can write to = number of quorums = N + 1

(N is an integer)

2. Election of history-based master node

According to an embodiment of the present invention, the distributed environment management system 200 may further enhance service continuity while ensuring data consistency using a history table.

According to an embodiment of the present invention, the history table may include node ID information, group joining point information, node state information, and group state information at the time when a group is first formed. The table below shows the data stored in the history table as an example.

ID Joining point Node status Group status Node  #One 2008/07 / 23_12: 12: 33 Master Major Node # 2 *** _ 12:12:35 Slave Minor Node # 3 *** _ 12:12:38 Slave Minor ...

Such a history table may be created when the group is initially configured, and may be maintained until the function of the group ends. Node status information, which is the third field in the history table, is changed to master or slave whenever a new master node is selected, and group status information is changed to major or minor as network disconnection failure occurs. minor).

The distributed environment management system 200 according to an exemplary embodiment of the present invention may determine a major group using the history table when one group is divided into two groups having the same number of nodes due to a network separation failure. (Of course, the history table may be used not only when determining a major group among two groups having the same number of nodes but also when determining a major group among two groups where the difference in the number of nodes is less than a predetermined reference value. It will be apparent to those skilled in the art). More specifically, a group including a node belonging to a major group before the network separation failure occurs and having the earliest time to join the group may be determined as a new major group. In addition, according to an embodiment of the present invention, the node having the earliest time of joining the group may be determined as the master node of the group. After all, if there is no unique group that satisfies the quorum, then the group with the group state immediately before the network isolation failure is major and the node with the most recently elected master (that is, the node with the earliest joining the group). This new major group can be decided.

5 is a diagram exemplarily illustrating a process of solving a network separation failure by applying a dynamic quorum method based on a history table. Hereinafter, the dynamic quorum method based on the history table will be described in detail by describing a process of changing the history table according to the steps shown in FIG. 5.

Step (1): A group of six nodes, before a network isolation failure occurs, all nodes belonging to the major group. Since Node # 1 joins the group the quickest, it is elected as the master node.

ID Joining point Node status Group status Node  #One *** _ 12:12:33 Master Major Node # 2 *** _ 12:12:35 Slave Major Node # 3 *** _ 12:12:38 Slave Major Node # 4 *** _ 12:12:39 Slave Major Node # 5 *** _ 12:12:40 Slave Major Node # 6 *** _ 12:12:42 Slave Major

Step (2): Network separation failure occurs, splitting into two groups with the same number of nodes. Node # 1 is the fastest node to join the group among the major groups before the failure, so the left group is determined as the major group and the right group is changed to the minor group.

ID Joining point Node status Group status Node  #One *** _ 12:12:33 Master Major Node # 2 *** _ 12:12:35 Slave Minor Node # 3 *** _ 12:12:38 Slave Major Node # 4 *** _ 12:12:39 Slave Minor Node # 5 *** _ 12:12:40 Slave Major Node # 6 *** _ 12:12:42 Slave Minor

(3) to (4) steps: During step (2), the major group has been reset to a group containing Node # 1, Node # 3, and Node # 5 (thus, the quorum is reset to 2), After that, the network separation failure occurred again and separated into a group containing Node # 1 and a group containing Node # 3 and Node # 5. The group containing Node # 3 and Node # 5 is determined to be a major group because it satisfies the quorum, and the group containing Node # 1 is changed to a minor group. Meanwhile, Node # 3 is elected as the new master node because the group joining time is the fastest.

ID Joining point Node status Group status Node # 1 *** _ 12:12:33 Slave Minor Node # 2 *** _ 12:12:35 Slave Minor Node  # 3 *** _ 12:12:38 Master Major Node # 4 *** _ 12:12:39 Slave Minor Node # 5 *** _ 12:12:40 Slave Major Node # 6 *** _ 12:12:42 Slave Minor

(5) step: Network separation failure occurs again in the group consisting of two nodes (Node # 3, Node # 5) and divided into two groups. As in step (2), among the nodes previously included in the major group, the group to which Node # 3 with the fastest group joining time belongs is determined as the new major group.

By utilizing the history table, it is possible to guarantee the continuity of services even if the group is divided into two groups containing the same node, or even if there is only one node including one node due to the continuous failure. .

Embodiments according to the present invention described above may be implemented in the form of program instructions that may be executed by various computer components, and may be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the process according to the invention, and vice versa.

Although the present invention has been described by specific embodiments such as specific components and the like, but the embodiments and the drawings are provided to assist in a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations can be made from these descriptions.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the appended claims, fall within the scope of the spirit of the present invention. I will say.

1 is a diagram illustrating a network separation failure situation by way of example.

2 is a view schematically showing a distributed environment according to an embodiment of the present invention.

3 is a diagram exemplarily illustrating a locking function provided by the distributed environment management system 200.

4 is a diagram illustrating a quorum algorithm applied to a network separation failure situation.

5 is a diagram exemplarily illustrating a process of solving a network separation failure by applying a dynamic quorum method based on a history table.

<Brief description of the major reference numerals>

100: network

200: distributed environment management system

300: client

Claims (22)

A method for determining a major group of at least two groups in which an initial group consisting of a plurality of nodes is separated from each other in a network-based distributed environment, Comparing the number of nodes in the first group with the number of nodes in the second group, Obtaining history information of the nodes of the first group and the nodes of the second group, and Determining a major group as the first group or the second group using at least one of the node number comparison result and the history information Including, The plurality of nodes each have specific data, The history information includes information regarding a point in time at which each node of the first group and each node of the second group joined the initial group, And a master node belonging to the major group may perform read and write operations on the specific data owned by the master node. delete The method of claim 1, The major group is determined to be a group including a node having the earliest point in time of joining the initial group among the first group and the second group. The method of claim 1, The major group is determined to be a group including a node that was a master node in the initial group of the first group and the second group. The method of claim 1, Acquiring the history information and determining the major group are performed when a difference between the number of nodes in the first group and the number of nodes in the second group is equal to or less than a predetermined reference value. The method of claim 5, If the difference exceeds the threshold, determining the major group as a group comprising more nodes of the first group and the second group. The method of claim 1, The first group and the second group are created as the initial group is separated at least twice, The major group is determined as the first group or the second group only when the group to which the first group and the second group recently belonged is a major group. delete The method of claim 1, The specific data is lock information in the distributed environment. The method of claim 1, The history information is information included in a history table replicated in each of the plurality of nodes. The method of claim 10, The history table includes information about when each of the plurality of nodes joined the initial group. A computer-readable recording medium for recording a computer program for executing the method according to any one of claims 1, 3 to 7, and 9 to 11. In a system having an initial group consisting of a plurality of nodes in a network-based distributed environment, The plurality of nodes each have specific data, When the initial group is separated to generate at least two groups, At least one of the plurality of nodes compares the number of nodes of the first group with the number of nodes of the second group, obtains historical information of the nodes of the first group and the nodes of the second group, and the node. A major group is determined as the first group or the second group using at least one of a number comparison result and the history information. Wherein the history information includes information regarding a time point at which each node of the first group and each node of the second group joined the initial group, And a master node belonging to the major group can read and write the specific data owned by the master node. delete The method of claim 13, At least one of the plurality of nodes determines the group including the node having the earliest time of joining the initial group of the first group and the second group as the major group. The method of claim 13, At least one of the plurality of nodes determines a group including a node that was a master node in the initial group of the first group and the second group as the major group. The method of claim 13, At least one of the plurality of nodes obtains the history information and determines the major group when a difference between the number of nodes in the first group and the number of nodes in the second group is equal to or less than a predetermined reference value. The method of claim 17, And when the difference exceeds the threshold, at least one of the plurality of nodes determines a major group as a group comprising more nodes of the first group and the second group. The method of claim 13, The first group and the second group are created as the initial group is separated at least twice, The major group is determined by the at least one of the plurality of nodes as the first group or the second group only if the group to which the first group and the second group recently belonged was a major group. The method of claim 13, The specific data is lock information in the distributed environment. The method of claim 13, The history information is information included in a history table replicated in each of the plurality of nodes. The method of claim 21, The history table includes information about when each of the plurality of nodes joined the initial group.

Images