JP4818349B2 - Distributed system and multiplexing control method for the same - Google Patents

Description

  The present invention synchronously operates n computers (n is an integer satisfying 4 <n ≦ m) among m computers (m is an integer larger than 4) connected via a network, and (n−f ) The present invention relates to a distributed system that guarantees multiplexing at or above (f is the maximum integer satisfying 3f <n), and particularly suitable for a case where a failure occurs in any of the n computers. The present invention relates to a multiplexing control method.

  In recent years, computer technology and network technology have been remarkably improved, and along with this, computerization of business has been widely performed. In addition, depending on the contents of the business, there are many things that cannot be interrupted due to a failure or the like, and recently, it is becoming common to construct a distributed system in which a plurality of computers are connected via a network. One of the distributed system operation techniques is multiplexing of execution of a deterministic program using ordered multicast.

  The ordered multicast is a mechanism for delivering the input to the distributed system to all computers, and guarantees that the arrival order of data is the same on all computers.

  As a multiplexing method in a distributed system, for example, Patent Document 1 is based on the principle of split brain in a distributed system in which n computers (n is an integer of 4 or more) (that is, 4 or more computers) are connected by a network. And a method that does not cause interruption of processing when a failure occurs due to a timeout.

In the conventional multiplexing method (prior art) disclosed in Patent Document 1, if a computer is a distributed system having n computers (n is an integer of 4 or more), the n computers Are operated synchronously to guarantee multiplexing on (n−f) of the n computers (where f is the maximum integer satisfying 3f <n called the maximum allowable failure number). In addition, each of the n computers includes input candidate collection means and input candidate selection control means (first input candidate selection control means). The input candidate collecting means collects input data selected as candidates to be processed next by each of the n computers via the network. When there are (n−f) or more input data collected by the input candidate collecting unit, the input candidate selection control unit determines whether or not there are (n−f) or more input data having the same contents. When there are (n−f) or more, the input data is determined as a target to be processed next. As a result, the input data is arranged and multicast. In this way, the input candidate selection control means determines that there are (n−f) or more input data having the same content among the (n−f) or more collected input data. f) It is nothing but an agreement on input data with more than one computer. That is, the input candidate selection control means functions as an agreement means.
Japanese Patent No. 3655263

  According to the above prior art, in a distributed system having n computers (n is an integer of 4 or more) that constitutes multiplexing, computer failures (failures) up to f computers (f is the maximum integer satisfying 3f <n). Stop) is allowed. That is, multiplexing of (n−f) or more units is guaranteed (continued). This means that in the above prior art, if the number of computers that have stopped (failed) has become (f + 1) or more, multiplexing cannot be continued and the distributed system fails. For example, in a distributed system with 7 computers (n = 7) that constitutes multiplexing, since f is 2, failure of up to 2 computers is permitted, but 1 computer fails. When stopped, that is, when three computers fail, the multiplexing cannot be continued.

  The present invention has been made in view of the above circumstances, and its object is to provide three computers that operate normally even when the number of computers in which a failure has occurred is equal to or greater than (f + 1) represented by f at the start of operation. It is an object of the present invention to provide a distributed system that can continue multiplexing if there are more than one, and a multiplexing control method for the system.

  According to one aspect of the present invention, among m computers (m is an integer greater than 4) connected by a network, n computers (n is an integer satisfying 4 <n ≦ m) are operated synchronously. A distributed system is provided. In the distributed system, each of the m computers includes a configuration storage unit, an agreement unit, a failure detection unit, and a configuration determination unit. The configuration storage means stores information for identifying n computers to be operated synchronously as information for identifying computers constituting the distributed system. The agreement means synchronously operates n computers constituting the distributed system identified by information stored in the configuration storage means, and (n−f) of the n computers. (F is the maximum integer satisfying 3f <n) In order to guarantee multiplexing at or above, the input data is arranged and multicasted by agreeing on the input data among the (n−f) computers or more. The failure detection means detects a computer in which a failure has occurred from the status of n computers constituting the distributed system identified by the information stored in the configuration storage means. The configuration determination unit determines that an item to be changed in the configuration of the distributed system has occurred when a computer in which a failure has occurred is detected by the failure detection unit, and an item to be changed in the configuration of the distributed system exists. It is determined whether to change the configuration of the distributed system depending on whether there are (n−f) or more computers that have been determined to have occurred. If it is determined to change the configuration of the distributed system, the configuration is stored in the configuration storage unit. The updated information is updated to show the configuration of the distributed system after the change.

  According to the present invention, it is determined that an item that should change the configuration of the distributed system has occurred because a failure has occurred in one of the n computers that currently constitute the distributed system, and the configuration of the distributed system is changed. By agreeing that there are (n−f) or more computers that have been determined to have occurred, the number of computers that have failed can be reduced by changing the configuration of the distributed system. Multiplexing can be continued if there are three or more computers that operate normally even if it is greater than (f + 1) represented by f

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a distributed system according to an embodiment of the present invention. In FIG. 1, it is assumed that the distributed system 1000 is multiplexed by, for example, seven computers 100-1 (# 1) to 100-7 (# 7). However, in the present embodiment, the number of computers constituting the distributed system 1000 is dynamically changed according to the status of the system.

  Therefore, the number of computers constituting the distributed system 1000 at the start of operation of the distributed system 1000 may be represented by m, and the current number of computers constituting the distributed system 1000 may be represented by n. n is an integer larger than 4 unlike the above-mentioned Patent Document 1. In Patent Document 1, the number of computers constituting the distributed system is not changed. That is, in Patent Document 1, the configuration of the distributed system is not changed according to the status of the system.

  m is an integer greater than 4 (that is, an integer of 5 or more). That is, the number m of computers constituting the distributed system 1000 at the start of the operation of the distributed system 1000 is not limited to 7, and may be larger than 4. As is apparent, n is an integer that satisfies 4 <n ≦ m. In the example of FIG. 1, m = n = 7, but in the present embodiment, n may be changed from 7 to 6 and from 6 to 5.

  The computers 100-1 to 100-7 are connected to the client apparatus 2000 via the network A. The computers 100-1 to 100-7 are also connected to a client device (not shown) other than the client device 2000 via the network A. In the present embodiment, the network A is a public network. The computers 100-1 to 100-7 are connected via a network B. In the present embodiment, the network B is a private network.

  When the computers 100-1 to 100-7 constitute the distributed system 1000, the computers 100-1 to 100-7 are connected via the network A in the same manner as the computers in the distributed system described in Patent Document 1. The input packet (input) received from the client apparatus 2000 is processed in the same order as other computers. An input packet from the client device 2000 is input to any one of the computers 100-1 to 100-7.

  Each of the computers 100-1 to 100-7 has the same application program 3 (see FIG. 2). The computers 100-1 to 100-7 start from the same initial state. Thereafter, the data input from the client apparatus 2000 to the distributed system is always delivered to the computers 100-1 to 100-7 through the ordered multicast in the same order. Thereby, the respective application programs 3 are executed in the computers 100-1 to 100-7.

  The input data string to the application program 3 that each of the computers 100-1 to 100-7 has has the same order by the ordered multicast. For this reason, due to the characteristics of the deterministic program as described in Patent Document 1, the states of the computers 100-1 to 100-7 are kept the same, and the output data strings are all the same. That is, program execution is multiplexed.

  In the distributed system described in Patent Document 1, when the number of computers constituting the multiplexing is n (n is an integer of 4 or more), in the distributed system, f is the largest integer satisfying 3f <n (maximum (Allowable failure count), up to f failure stops are allowed. In other words, in a distributed system composed of n computers, multiplexing of (n−f) or more computers is guaranteed, and the multiplexed program is executed on at least (n−f) computers. The Therefore, also in this embodiment in which n is an integer greater than 4, multiplexing of (n−f) or more computers is guaranteed.

  FIG. 2 is a block diagram showing a configuration of the computer 100-i (i = 1, 2,..., 7). In FIG. 2, an input packet transmitted from the client device 2000 to the computer 100-i via the network A and received by the input reception queue unit 1 of the computer 100-i is queued in the input reception queue unit 1. Is done. The input packet queued in the input reception queue unit 1 is sent to the application program 3 or the later-described ordered multicast unit 2 by the ordered multicast unit 2 (the agreement unit 262 included in the later-described input packet confirmation determination unit 26). To the configuration determining unit 210. The input packet queued in the input reception queue unit 1 includes an input packet (specific input packet) sent from the configuration determining unit 210 in addition to the input packet from the client device 2000.

  The application program 3 receives the delivered input packet, processes the input packet according to the state stored in the program state management unit 4, and generates an output packet. The generated output packet is selected by the output filter unit 5 and then returned to the client device 2000 via the network A (output).

  Next, the configuration of the ordered multicast unit 2 of the computer 100-i will be described. Similar to the ordered multicast unit described in Patent Document 1, the ordered multicast unit 2 has an input sequence number storage unit 21, an input packet journal storage unit 22, a protocol data transmission / reception unit 23, a step number storage unit 24, and a candidate packet storage. The unit 25, the input packet determination determination unit 26, the maximum determination input sequence number storage unit 27, the delay storage unit 28, and the skip determination unit 29 are well-known.

  The input sequence number storage unit 21 stores the sequence number of the next input packet to be delivered to the computer 100-i by the ordered multicast (that is, the latest sequence number assigned serially to the ordered multicast). The input packet journal storage unit 22 stores a certain amount of the input packet sequence that is confirmed to be delivered to the computer 100-i by the ordered multicast from the latest one. The protocol data transmission / reception unit 23 exchanges protocol data with the protocol data transmission / reception unit 23 of another computer via the network B.

  In the present embodiment, the network to be used is switched between data exchange between the client apparatus 2000 and the computer 100-i and data exchange between the computers 100-i. This reduces the network load. However, the configuration may be such that data exchange between the client apparatus 2000 and the computer 100-i and data exchange between the computers 100-i are performed via the network A, for example. Further, the network A is not necessarily a public network.

  The step number storage unit 24, the candidate packet storage unit 25, and the input packet confirmation determination unit 26 are used in an algorithm that determines the next input packet to be delivered to the computer 100-i by ordered multicast. The step number storage unit 24 stores a step number indicating a protocol step. The candidate packet storage unit 25 stores a total of n input packets that are “input candidates” for each computer in that step.

  The input packet determination determination unit 26 determines the determination of the input packet from the information in the candidate packet storage unit 25 and determines the “input candidate” in the next step. Unlike the input packet determination determination unit described in Patent Document 1, the input packet determination determination unit 26 further determines whether the input packet is to be passed to the application program 3 or the configuration determination unit 210. For this determination, information indicating the processing type (processing type information) is added to the input packet queued in the input reception queue unit 1. The input packet confirmation determination unit 26 includes an input candidate collection unit 261 and an agreement unit 262.

  FIG. 3 shows an example of the data structure of data (input packet data) queued in the input reception queue unit 1. As shown in FIG. 3, the input packet data includes fields of processing type and input packet. An input packet is stored (set) in the input packet field, and process type information is stored (set) in the process type field.

  In the present embodiment, the processing type information is determined by the input packet confirmation determination unit 26 (the agreement unit 262) as to whether the input packet stored in the input packet field is to be passed to the application program 3 or the configuration determination unit 210. The processing type for Therefore, the process type indicated by the process type information is divided into (1) application and (2) configuration. When the processing type is “application”, it also indicates that the input packet is input from the external client device 2000. When the processing type is “configuration”, the input packet is one of the constituents of the distributed system 1000. It also indicates that the information is input from the computer configuration determination unit 210 to the input reception queue unit 1 of any one of the computers.

  Referring again to FIG. 2, the maximum confirmed input sequence number storage unit 27 stores the maximum input sequence number that is known to have been delivered, including other computers. The delay storage unit 28 has (n−1) delay flags (6 in the present embodiment where n = 7) indicating whether or not the delay storage is delayed from the other (n−1) computers (n = 7). Number of flags). The skip determination unit 29 determines the necessity of the skip operation from the information in the delay storage unit 28 and executes the skip operation.

  FIG. 4 shows an example of the data structure of the maximum confirmed input sequence number storage unit 27. In the data structure example shown in FIG. 4, “50000” is stored as the maximum confirmed input order number. In this case, the maximum confirmed input sequence number storage unit 27 indicates that the maximum input sequence number that is known to have been confirmed for delivery, including other computers, is “50000”.

  In the following description, the input sequence number stored in the input sequence number storage unit 21 is referred to as a corresponding input sequence number, and the step number stored in the step number storage unit 24 is referred to as a corresponding step number. Of the n “input candidates” stored in the candidate packet storage unit 25 included in the ordered multicast unit 2 of the computer 100-i, the “input candidates” corresponding to the computer 100-i itself (own computer) Is called a self-candidate, and “input candidates” other than the self-candidate are called other candidates.

  In the present embodiment, the ordered multicast unit 2 is different from the ordered multicast unit described in Patent Document 1 in that a new computer for dynamically changing a computer constituting the distributed system 1000 according to the status of the system 1000 is provided. Including the configuration. That is, the ordered multicast unit 2 further includes a configuration determining unit 210, a configuration storage unit 211, a maximum sequence number delay tolerance storage unit 212, and another computer maximum fixed input sequence number storage unit 213, as shown in FIG.

  The configuration determination unit 210 determines the computers that make up the distributed system 1000 according to the status of the system 1000 (in more detail, the status of the computers 100-1 to 100-7 that currently make up the system 1000). To decide. The configuration storage unit 211 stores information indicating computers constituting the distributed system 1000 determined by the configuration determination unit 210. In the present embodiment, an identifier (ID) for uniquely identifying the computer is used as information indicating the computer constituting the distributed system 1000. The configuration determination unit 210 is guided by the number n of computers constituting the multiplexing in the distributed system 1000 and the n based on information stored in the configuration storage unit 211 (information indicating the computers configuring the distributed system 1000). The maximum allowable failure number f is determined. The configuration determination unit 210 includes a failure detection unit 210a that detects a failure of another computer.

  FIG. 5 shows an exemplary data structure of the configuration storage unit 211. In the data structure example shown in FIG. 5, IDs (ID = 1 to ID = 7) of the computers # 1 to # 7 are stored in the configuration storage unit 211. In this case, the configuration storage unit 211 indicates that the distributed system 1000 includes computers # 1 to # 7, n is 7 (n = 7), and f is 2 (f = 2).

  The maximum sequence number delay allowable value storage unit 212 is an input sequence number of the maximum determined input sequence number of a computer (other computer) other than the computer 100-i, which is stored in another computer maximum determined input sequence number storage unit 213 to be described later. A value (maximum sequence number delay allowable value) L that allows a deviation (delay) with respect to the corresponding input sequence number (that is, the latest input sequence number managed in the computer 100-i) stored in the storage unit 21 is stored. . This value L is arbitrarily set by the user, for example. However, L is an integer greater than 1.

  FIG. 6 shows a data structure example of the maximum sequence number delay allowable value storage unit 212. In the example of the data structure shown in FIG. 6, “3000” is stored in the maximum sequence number delay allowable value storage unit 212 as the maximum sequence number delay allowable value L. In this case, a delay with respect to the corresponding input sequence number of the maximum confirmed input sequence number of another computer is allowed up to “3000” in terms of the difference of the sequence numbers. That is, the configuration determining unit 210 determines that the other computer is normal as long as the deviation of the maximum confirmed input sequence number of the other computer from the corresponding input sequence number does not exceed “3000”. In other words, when the deviation of the maximum confirmed input sequence number of another computer from the corresponding input sequence number exceeds “3000”, the configuration determining unit 210 determines (detects) that the other computer is abnormal.

  In the present embodiment, abnormality detection of the computer 100-i (that is, the distributed system 1000 of the distributed system 1000) using the above-described maximum fixed input sequence number (maximum fixed input sequence number stored in the other computer maximum fixed input sequence number storage unit 213). The distributed system 1000 is reconfigured in response to changes in the situation.

  The other computer maximum confirmed input sequence number storage unit 213 stores the maximum confirmed input sequence number corresponding to computers (other computers) excluding the computer 100-i itself (local computer) among the set of computers constituting the distributed system 1000. To do.

  FIG. 7 shows an example of the data structure of the computer maximum determined input sequence number storage unit 213 of the computer 100-1 (# 1) when the computer 100-i is the computer 100-1 (# 1). In the data structure example shown in FIG. 7, the maximum confirmed input sequence number (50000) of the computers 100-2 (# 2) to 100-7 (# 7) other than the computer 100-1 (# 1) is the computer 100. -2 (# 2) to 100-7 (# 7) are stored in the other computer maximum confirmed input sequence number storage unit 213 in association with IDs, for example.

Next, protocol data transmitted / received by the protocol data transmitting / receiving unit 23 will be described.
FIG. 8 is a diagram showing a layout of protocol data. As shown in FIG. 8, the protocol data transmitted / received by the protocol data transmitting / receiving unit 23 includes fields of type, sender, input sequence number, step number, maximum confirmed input sequence number, processing type, and input packet. The protocol data shown in FIG. 8 is different from the protocol data described in Patent Document 1 in that the processing type field is added.

  As is well known, the protocol data is divided into the following three types depending on the type field at the head.

  (1) Candidate type: In the input sequence number field, step number field, input packet field, and processing type field, the corresponding input sequence number, corresponding step number, own candidate, Stores the processing type (processing type information) assigned to the candidate.

  (2) Determined type: Indicates that the input packet corresponding to the input sequence number (the input sequence number stored in the input sequence number field) is in the input packet journal storage unit 22 at the time of transmission by the sender. The packet field and the processing type field store the input packet and the processing type (processing type information) attached to the input packet, respectively. In this case, the step number field is not used.

(3) Delay type: Indicates that there is no input packet corresponding to the input sequence number in the input packet journal storage unit 22 at the time of transmission by the sender. In this case, the step number field, the input packet field, and the process type field are not used.

  In any type of protocol data, the maximum confirmed input order number field stores the corresponding maximum confirmed input order number when protocol data is transmitted from the sender (sender computer). Further, the corresponding maximum confirmed input sequence number in the receiving computer of the protocol data includes the sequence number of the input packet confirmed in the receiving computer, the maximum confirmed input sequence number in the protocol data received by the receiving computer, Of these, it is updated to the largest one.

Next, the operation procedure of the ordered multicast unit 2 of the computer 100-i will be described with reference to the flowcharts of FIGS.
First, in the initial state, the input sequence number storage unit 21 stores an initial input sequence number (for example, 1). The input packet journal storage unit 22 is empty, and the step number storage unit 24 stores an initial step number (for example, 1). The candidate packet storage unit 25 is empty, the maximum determined input sequence number storage unit 27 stores the initial input sequence number, and all the flags in the delay storage unit 28 are reset.

FIG. 9 and FIG. 10 are flowcharts showing an operation procedure of a basic part that performs one delivery of ordered multicast.
The input candidate collection unit 261 included in the input packet confirmation determination unit 26 in the ordered multicast unit 2 is an input selected as a candidate (input candidate) to be processed next by each of the computers 100-1 to 100-7 (n = 7). A candidate list creation process for collecting packets (input data) is executed (step A1 in FIG. 8).

  In the candidate list creation process, the input candidate collection unit 261 receives an input packet in the input reception queue unit 1 when the corresponding step number stored in the step number storage unit 24 is an initial value (YES in step B1 in FIG. 10). It is determined whether or not (more specifically, input packet data including input packets and processing type information) exists (step B2 in FIG. 10).

  If there is an input packet (YES in step B2 in FIG. 10), the input candidate collection unit 261 advances the corresponding step number stored in the step number storage unit 24 (step B3 in FIG. 10). Then, the input candidate collection unit 261 stores the input packet as a self-candidate in the candidate packet storage unit 25, and the self-candidate is set in the input packet field, and the processing type information attached to the self-candidate is the processing type. The protocol data of the candidate type set in the field is transmitted to all other computers via the network B by the protocol data transmitting / receiving unit 23 (step B4 in FIG. 10). In step B4, the input candidate collection unit 261 empties all other candidates in the candidate packet storage unit 25.

  On the other hand, when the corresponding step number is not the initial value (NO in step B1 in FIG. 10) or there is no input packet in the input reception queue unit 1 (NO in step B2 in FIG. 10), the input candidate collection unit 261 It is determined whether or not the protocol data transmitting / receiving unit 23 has received candidate type protocol data having an input sequence number (input sequence number field in which the input sequence number matches the corresponding input sequence number stored in the sequence number storage unit 21). (Step B5 in FIG. 10). If it has been received (YES in step B5 in FIG. 10), the input candidate collection unit 261 has a step number (set in the step number field) in the received protocol data (reception protocol data). Is greater than the corresponding step number (step B6 in FIG. 10).

  If the step number in the reception protocol data is larger than the corresponding step number (YES in step B6 in FIG. 10), the input candidate collection unit 261 receives the corresponding step number stored in the step number storage unit 24. Update to the step number in the protocol data (step B7 in FIG. 10). In this step B 7, the maximum confirmed input sequence number (set in the maximum confirmed input sequence number field) in the reception protocol data is obtained from the corresponding maximum confirmed input sequence number stored in the maximum confirmed input sequence number storage unit 27. If the value is also larger, the corresponding maximum determined input sequence number is updated to the maximum determined input sequence number in the reception protocol data. In the following description, the process in which the corresponding maximum confirmed input sequence number is updated to the maximum confirmed input sequence number in the reception protocol data is referred to as the corresponding maximum confirmed input sequence number update process. In step B7, the maximum confirmed input sequence number stored in the other computer maximum confirmed input sequence number storage unit 213 corresponding to the sender of the received protocol data is updated to the maximum confirmed input sequence number in the received protocol data. The

  Next, the input candidate collection unit 261 stores the input packet (set in the input packet field) in the reception protocol data as a self-candidate in the candidate packet storage unit 25, and sets the self-candidate in the input packet field. Protocol data transmission / reception unit for transmitting the protocol data of the candidate type in which the processing type information attached to the candidate is set in the processing type field (that is, including the input packet field and the copy of the processing type field in the received protocol data) 23, it transmits to all other computers via the network B (step B8 in FIG. 10). In this step B8, the input candidate collection unit 261 uses the input packet in the reception protocol data (in this case, the input packet determined as its own candidate) as another candidate corresponding to the sender of the reception protocol data (indicated by the sender field). Is stored in the candidate packet storage unit 25. That is, in step B8, the input packet in the reception protocol data is set as its own candidate and also set as another candidate corresponding to the sender of the reception protocol data. At this time, the input candidate collection unit 261 discards (empties) all other candidates in the candidate packet storage unit 25 other than the other candidates corresponding to the sender of the received protocol data.

  On the other hand, if the step number in the reception protocol data is equal to the corresponding step number (NO in step B6 in FIG. 10, YES in step B9), the input candidate collection unit 261 determines the input packet in the reception protocol data. Then, it is stored in the candidate packet storage unit 25 as another candidate corresponding to the sender of the reception protocol data (step B10 in FIG. 10). In this step B10, if the maximum confirmed input sequence number in the reception protocol data is larger than the corresponding maximum confirmed input sequence number stored in the maximum confirmed input sequence number storage unit 27, the corresponding maximum confirmed input sequence number update process Is done. In step B10, the maximum confirmed input sequence number stored in the other computer maximum confirmed input sequence number storage unit 213 corresponding to the sender of the received protocol data is updated to the maximum confirmed input sequence number in the received protocol data. The

  When executing step B6 or step B10, the input candidate collection unit 261 determines whether the number of candidates stored in the candidate packet storage unit 25 is (n−f) or more (step B11 in FIG. 10).

  If the number of candidates does not reach (n−f) or more (NO in step B11 in FIG. 10), the input candidate collection unit 261 executes the processing from step B1 again. On the other hand, if the number of candidates is (n−f) or more (YES in step B11 in FIG. 10), the input candidate collection unit 261 ends the candidate list creation process. Even when the determination in step B5 or step B9 is NO, the processing from step B1 is executed again.

  When the candidate list creation process (step A1 in FIG. 9) ends, that is, when the number of candidates (input candidates) (number of non-empty candidates) stored in the candidate packet storage unit 25 becomes (n−f) or more, the input packet The agreement unit 262 in the decision determination unit 26 functions as a first input candidate selection control unit, and whether or not (n−f) or more identical candidates exist in the candidate packet storage unit 25, that is, (n− f) It is determined whether or not there is a candidate for which an agreement has been reached (an agreement has been formed) on at least (n−f) computers (step A2 in FIG. 9). If there are (n−f) or more identical (identical contents) candidates (YES in step A2 in FIG. 9), the agreement unit 262 determines the candidates as input packets in the corresponding input sequence number. (Step A3 in FIG. 9). In step A <b> 3, if the determined input packet exists in the input reception queue unit 1, the agreement unit 262 deletes the input packet from the input reception queue unit 1.

  As described above, when the input packet at the corresponding input sequence number is determined, that is, when the input packet at the corresponding input sequence number is agreed and the ordered multicast is determined, the agreement unit 262 determines the input sequence to proceed to the next process. The corresponding input sequence number stored in the number storage unit 21 is advanced (incremented by 1), and the corresponding step number stored in the step number storage unit 24 is initialized (step A4 in FIG. 9). In step A4, the agreement unit 262 discards (empties) all candidates stored in the candidate packet storage unit 25, and sets all (n−1) delay flags stored in the delay storage unit 28. Reset.

  Next, the agreement unit 262 functions as a candidate output destination switching unit, and determines whether the process type indicated by the process type information attached to the confirmed input packet is “application” (that is, “configuration”). ) Is checked (step A8 in FIG. 9). If it is “application” (YES in step A8 in FIG. 9), the agreement unit 262 passes the determined input packet to the application program 3, and stores the input packet in the input packet journal storage unit 22 (FIG. 9). 9 step A9). Here, the determined input packet is stored so as to be positioned behind the column of input packets currently stored in the input packet journal storage unit 22. If the input packet sequence stored in the input packet journal storage unit 22 reaches a certain amount, the first input packet (that is, the oldest input packet) is discarded.

  On the other hand, if it is not “application”, that is, if it is “configuration” (NO in step A8 in FIG. 9), the agreement unit 262 passes the determined input packet to the configuration determination unit 210 and also performs the input. The packet is stored in the input packet journal storage unit 22 (step A10 in FIG. 9).

  On the other hand, if (n−f) or more identical candidates do not exist (NO in step A2 in FIG. 9), the agreement unit 262 functions as a second input candidate selection control unit, and this time, candidates It is determined whether or not a majority of the same candidates exist in the packet storage unit 25 (step A5 in FIG. 9). If more than half of the same candidates exist (YES in step A5 in FIG. 9), the agreement unit 262 selects the candidate and stores it as a candidate in the candidate packet storage unit 25 (that is, the candidate). ) As the self-candidate in the candidate packet storage unit 25), and this self-candidate is set in the input packet field, and the processing type information attached to the self-candidate is set in the processing type field. Is transmitted to all other computers via the network B by the protocol data transmitting / receiving unit 23 (step A6 in FIG. 9). In step A <b> 6, the agreement unit 262 discards all other candidates stored in the candidate packet storage unit 25.

  On the other hand, if more than a majority of the same candidates do not exist (NO in step A5 in FIG. 9), the agreement unit 262 functions as a third input candidate selection control unit and is stored in the candidate packet storage unit 25. A candidate (input packet) is randomly selected from among the input candidates, and the self candidate is set in the input packet field, and the processing type information attached to the self candidate is displayed in the processing type field. The set candidate type protocol data is transmitted to all other computers via the network B by the protocol data transmission / reception unit 23 (step A7 in FIG. 9). In step A7, the agreement unit 262 discards all other candidates stored in the candidate packet storage unit 25.

  When step A6 (operation as the second input candidate selection control unit) or step A7 (operation as the third input candidate selection control unit) is completed, the agreement unit 262 executes the processing from step A1 again. On the other hand, when step A9 or step A10 (operation as candidate output destination switching means) is completed, one delivery process of the ordered multicast is ended.

  With the above procedure, each computer 100-i proceeds with processing while confirming the match of input packets in (n−f) or more computers.

Next, an operation procedure for eliminating the multiplexing execution delay will be described.
FIG. 11 to FIG. 14 are flowcharts showing an operation procedure for eliminating the multiplexing execution delay. 11 to 14 correspond to FIGS. 7 to 10 of Patent Document 1. FIG.

  When the protocol data transmitter / receiver 23 receives candidate type protocol data having an input sequence number smaller than the corresponding input sequence number, the agreement unit 262 in the ordered multicast unit 2 receives an input packet corresponding to the input sequence number. It is determined whether or not the packet exists in the input packet journal storage unit 22 (step C1 in FIG. 11).

  If an input packet corresponding to an input sequence number smaller than the corresponding input sequence number exists in the input packet journal storage unit 22 (YES in step C1 in FIG. 11), that is, because of a short multiplexing execution delay, If an input packet corresponding to an input sequence number smaller than the input sequence number remains in the input packet journal storage unit 22, the agreement unit 262 sets the input packet in the input packet field and attaches it to the input packet. The protocol data transmission / reception unit 23 causes the protocol data transmission / reception unit 23 to return the received protocol data to the sender of the received protocol data (step C2 in FIG. 11).

  On the other hand, if the input packet corresponding to the input sequence number smaller than the corresponding input sequence number does not exist in the input packet journal storage unit 22 (NO in step C1 in FIG. 11), that is, because of a long multiplexing execution delay, When an input packet corresponding to an input sequence number smaller than the input sequence number has already been discarded from the input packet journal storage unit 22, the agreement unit 262 uses the protocol data transmission / reception unit 23 to send the network B to the protocol type transmission / reception unit 23. To the sender of the received protocol data (step C3 in FIG. 11).

  In steps C2 and C3, if the maximum confirmed input sequence number in the reception protocol data is greater than the corresponding maximum confirmed input sequence number stored in the maximum confirmed input sequence number storage unit 27, the corresponding maximum confirmed input sequence number. Number update processing is performed. In steps C2 and C3, the maximum determined input sequence number stored in the other computer maximum determined input sequence number storage unit 213 corresponding to the sender of the received protocol data is changed to the maximum determined input sequence number in the received protocol data. Updated.

  In addition, when the protocol data transmitter / receiver 23 receives a definite type of protocol data having an input sequence number that matches the input sequence number, the agreement unit 262 uses the input packet in the received protocol data as an input packet. Confirm (step D1 in FIG. 12). In step D1, the agreement unit 262 deletes the input packet from the input reception queue unit 1 if the determined input packet exists in the input reception queue unit 1. In step D1, if the maximum confirmed input sequence number in the reception protocol data is larger than the corresponding maximum confirmed input sequence number stored in the maximum confirmed input sequence number storage unit 27, the corresponding maximum confirmed input sequence number update process is performed. Is done. In step D1, the maximum determined input sequence number stored in the other computer maximum determined input sequence number storage unit 213 corresponding to the sender of the received protocol data is updated to the maximum determined input sequence number in the received protocol data. The

  Next, the agreement unit 262 advances the corresponding input sequence number stored in the input sequence number storage unit 21 (increment by 1) to proceed to the next process, and the corresponding stored in the step number storage unit 24. The step number is initialized (step D2 in FIG. 12). In step D <b> 2, the agreement unit 262 discards all candidates stored in the candidate packet storage unit 25 and resets all (n−1) delay flags stored in the delay storage unit 28.

  Next, the agreement unit 262 functions as a candidate output destination switching unit, and determines whether the process type indicated by the process type information attached to the confirmed input packet is “application” (step D3 in FIG. 12). ). If it is “application” (YES in step D3 in FIG. 12), the agreement unit 262 passes the determined input packet to the application program 3 and stores the input packet in the input packet journal storage unit 22 (FIG. 12). 12 step D4).

  On the other hand, if it is not “application” (NO in step D3 in FIG. 12), the agreement unit 262 passes the determined input packet to the configuration determination unit 210 and also sends the input packet to the input packet journal storage unit 22. Store (step D5 in FIG. 12).

  On the other hand, the skip determination unit 29 in the ordered multicast unit 2 stores the delay type protocol data having an input sequence number matching the corresponding input sequence number in the delay storage unit 28 when the protocol data transmission / reception unit 23 receives it. Among the (n−1) delay flags that have been set, a delay flag corresponding to the sender of the protocol data is set (step E1 in FIG. 13). In this step E1, if the maximum confirmed input sequence number in the reception protocol data is larger than the corresponding maximum confirmed input sequence number stored in the maximum confirmed input sequence number storage unit 27, the corresponding maximum confirmed input sequence number update process. Is done. In step E1, the maximum determined input sequence number stored in the other computer maximum determined input sequence number storage unit 213 corresponding to the sender of the received protocol data is updated to the maximum determined input sequence number in the received protocol data. The

  The skip determination unit 29 also sets the number of set delay flags among the (n−1) delay flags stored in the delay storage unit 28 and the input candidates stored in the candidate packet storage unit 25. It is monitored whether or not the sum with the number has reached (n−f) or more (step F1 in FIG. 14). If the number reaches (n−f) or more (YES in step F1 in FIG. 14), the skip determination unit 29 determines that the number of input candidates stored in the candidate packet storage unit 25 is (n−f). ) It is determined whether it is less (step F2 in FIG. 14). If the number is less than (n−f) (YES in step F2 in FIG. 14), the skip determination unit 29 performs a skip operation (step F3 in FIG. 14). That is, the skip determination unit 29 updates the corresponding input sequence number stored in the input sequence number storage unit 21 to the corresponding maximum determined input sequence number stored in the maximum determined input sequence number storage unit 27, and the step number storage unit The corresponding step number stored in 24 is set as the initial step number. The skip determination unit 29 empties the candidate packet storage unit 25, resets all delay flags in the delay storage unit 28, and notifies the program state management unit 4 of the skip. When the skip is notified, the program state management unit 4 copies the state immediately before the corresponding input sequence number from the program state management unit 4 of another computer. For this reason, the program state management unit 4 holds the state immediately before each input sequence number by a certain amount from the latest one.

  With the above procedure, each computer 100-i realizes a mechanism in which delayed execution catches up so as not to cause split brain.

  Next, an operation procedure (configuration determination process) for determining the configuration of the distributed system 1000, which is executed by the configuration determination unit 210 of each computer 100-i, will be described with reference to the flowchart of FIG. In the present embodiment, the configuration determination process by the configuration determination unit 210 is executed at regular time intervals (that is, periodically), but may be executed at random timing.

  First, the configuration determination unit 210 determines whether the number n of computers that currently constitute the distributed system 1000 exceeds 4 based on the information stored in the configuration storage unit 211 (step G1 in FIG. 15). If n exceeds 4 (that is, 5 or more) (YES in step G1 in FIG. 15), the configuration determination unit 210 includes n computers (here, the computers) that currently constitute the distributed system 1000. Of the computers 100-1 to 100-7), all computers (other computers) 100-j (where j is 1 to 7, except for the computer 100-i (the computer 100-i including the configuration storage unit 211)) For i), the following processing is executed using the failure detection unit 210a (step G2 in FIG. 15).

  First, the failure detection unit 210a includes the corresponding input sequence number stored in the input sequence number storage unit 21 and the computer 100-j that is the current target stored in the other computer maximum confirmed input sequence number storage unit 213. The difference from the maximum confirmed input sequence number of (target computer 100-j) is obtained, and this difference becomes larger than the maximum sequence number delay allowable value L stored in the maximum sequence number delay allowable value storage unit 212. (Step G3 in FIG. 15). Here, the difference indicates the amount of processing delay for the input packet sequence in the target computer 100-j based on the corresponding input sequence number of the computer 100-i.

  If the difference is larger than the maximum sequence number delay allowable value L (YES in step G3 in FIG. 15), the failure detecting unit 210a determines that the target computer 100-j is abnormal. That is, the failure detection unit 210a detects an abnormality (failure) in the target computer 100-j when the difference is larger than the maximum sequence number delay allowable value L. In this case, the configuration determination unit 210 distributes the target computer 100-j (the computer 100-j in which an abnormality has been detected), if more specifically, that it is necessary to change the configuration of the distributed system 1000. It is determined that it is necessary to remove (separate) from the system 1000 (a set of computers constituting the system).

  Therefore, the configuration determining unit 210 makes an agreement (forms an agreement) to remove (separate) the target computer 100-j (the computer 100-j in which an abnormality has been detected) from the distributed system 1000 (a set of computers constituting the computer). ), That is, a process for obtaining an agreement that the configuration of the distributed system 1000 should be changed, using the ordered multicast unit 2 (the agreement unit 262 included in the input packet determination unit 26). (Step G4 in FIG. 15). In other words, the configuration determining unit 210 makes an agreement to remove the target computer 100-j from the distributed system 1000 in the same manner as the ordered multicast for the input packet to be delivered to the application program 3. 2 is used. Whether to remove the target computer 100-j from the distributed system 1000 is determined by agreement among (n−f) or more computers, similarly to the ordered multicast.

  In the present embodiment, removing the target computer 100-j from the distributed system 1000 means deleting information (ID) indicating the computer 100-j from the configuration storage unit 211, as will be described later. That is, to remove the target computer 100-j from the distributed system 1000 is to logically remove the computer 100-j from the set of computers constituting the distributed system 1000, and physically remove the computer 100-j. It is not removed from the distributed system 1000.

  In step G4, the configuration determining unit 210 sends a specific packet requesting to remove the target computer 100-j from the distributed system 1000 to the input reception queue unit 1. Processing type information indicating that the processing type is “configuration” is attached to the specific packet. The specific packet sent from the configuration determining unit 210 to the input reception queue unit 1 is received by the input reception queue unit 1 and is queued in the input reception queue unit 1.

  A specific packet queued in the input reception queue unit 1 (hereinafter referred to as a specific input packet) is obtained from the operation of the ordered multicast unit 2 (within the agreement unit 262) with reference to the flowcharts of FIGS. As will be apparent, the ordered multicast unit 2 of n (n = 7) computers (computers 100-1 to 100-7) including the target computer 100-j is the target of the ordered multicast.

  If the agreement of specific input packets has been reached by (n−f) or more of n (n = 7) computers (computers 100-1 to 100-7) (step of FIG. 9). In the aligned multicast unit 2 of the (n−f) or more computers, the specific input packet is transferred from the agreement unit 262 to the configuration determining unit 210 and stored in the input packet journal storage unit 22. (Step A10 in FIG. 9).

  The configuration determination unit 210 determines that an agreement to remove the target computer 100-j from the distributed system 1000 has been reached by (nf) or more computers when the specific input packet is passed from the agreement unit 262. (YES in step G5 in FIG. 15). In this case, the configuration determining unit 210 removes the target computer from the distributed system 1000 by deleting the information (ID) indicating the target computer 100-j from the configuration storage unit 211 (step G6 in FIG. 15).

  The configuration determining unit 210 (the configuration determining unit 210 of the computer 100-i) performs the processing from the above step G3, excluding the computer 100-i among the n computers that currently constitute the distributed system 1000 (n− 1) Repeat for one computer (other computer) (step G2 in FIG. 15).

  According to the present embodiment, n and f can be scaled down by the processing of the configuration determining unit 210 as described above. Hereinafter, the scale-down of n and f will be described.

  First, in the present embodiment in which the distributed system 1000 includes seven computers 100-1 to 100-7 (n = 7, f = 2) at the start of operation of the distributed system 1000, a certain computer (hereinafter referred to as the first computer). ) Is detected, and an agreement to remove the first computer from the distributed system 1000 has been reached by (n−f) computers (here, 5 computers). In this case, the configuration of the distributed system 1000 is changed to include six computers (n = 6, f = 1).

  Next, in a distributed system 1000 composed of six computers (n = 6, f = 1), a failure of a certain computer (hereinafter referred to as a second computer) is detected, and the second computer is It is assumed that an agreement to be removed from the distributed system 1000 has been reached with (n−f) computers (here, 5 computers). In this case, the configuration of the distributed system 1000 is changed to include five computers (n = 5, f = 1).

  Next, in a distributed system 1000 composed of five computers (n = 5, f = 1), a failure of a certain computer (hereinafter referred to as a third computer) is detected, and the third computer is It is assumed that an agreement to be removed from the distributed system 1000 has been reached with (n−f) computers (here, four computers). In this case, the configuration of the distributed system 1000 is changed to include four computers (n = 4, f = 1).

  As described above, according to the present embodiment, the number of computers included in the distributed system 1000 is reduced from seven (n = 7) to four (n = 4), that is, n and f are reduced. You can scale down from 7,2 to 4,1. In this state, since f = 1 is realized, multiplexing can be continued if there are three or more normally operating computers among the four (n = 4) computers. That is, if at least three computers operate normally even if the number of failed computers is equal to or greater than (f + 1) (here 3) represented by f (2 here) at the start of operation. Multiplexing can be continued, split brain is not generated in principle, and processing is not interrupted when a failure occurs due to timeout.

  Here, the cause of the abnormality of the computer 100-j removed from the distributed system 1000 (the delay exceeding the maximum sequence number delay allowable value L) is the slowdown due to the temporary high load of the computer 100-j. Shall. In this case, the computer 100-j may recover from the slowdown and operate as one of the n computers based on information in the configuration storage unit 211.

However, the content of the configuration storage unit 211 of the computer 100-j is not the content of the configuration storage unit 211 of the computer that currently configures the distributed system 1000, but the computer 100-j is removed from the distributed system 1000 (separated). The contents of the configuration storage unit 211 at the time. Here, if n and f when the computer 100-j is disconnected from the distributed system 1000 are represented by n _old and f _old and the current n and f are represented by n _new and f _new , the computer 100-j , It is necessary to agree on (n _old -f _old ) computers, not (n _new -f _new ) computers. However, n _new computers among the n _old computers constitute the current distributed system 1000. Accordingly, the computer 100-j cannot agree on the (n _old -f _old ) computers including the computer 100-j, and the operation of the n _new computers constituting the current distributed system 1000 is not possible. Will not be adversely affected.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. For example, in the above embodiment, the difference between the corresponding input sequence number and the maximum confirmed input sequence number of the computer 100-j is used to detect an abnormality (failure) in the computer 100-j. However, other methods can be used for detecting an abnormality of the computer 100-j. For example, when a heartbeat timeout algorithm as described in Patent Document 1 as a prior art, that is, when a heartbeat periodically transmitted by each computer cannot be confirmed for a predetermined time or more, an abnormality (failure) of the computer It is also possible to use a method of determining In this method, as pointed out as a problem of the prior art in the above-mentioned Patent Document 1, there is a possibility that a fault is erroneously detected and falls into a split brain. However, for example, when a failure of the computer 100-j is detected by the heartbeat timeout algorithm, it is confirmed that there are n-f or more computers performing the same failure detection, that is, the computer 100-. In principle, the occurrence of split brain can be prevented by agreeing to detect the failure of j with n−f or more computers.

  In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment.

1 is a block diagram showing a configuration of a distributed system according to an embodiment of the present invention. 2 is an exemplary block diagram showing a functional configuration of a computer constituting the distributed system of the embodiment. FIG. The figure which shows the example of a data structure of the data (input packet) queued in the input reception queue part of the embodiment. The figure which shows the example of a data structure of the largest fixed input order number memory | storage part in the embodiment. The figure which shows the data structure example of the structure memory | storage part in the embodiment. The figure which shows the data structure example of the largest sequence number delay tolerance memory | storage part in the embodiment. The figure which shows the example of a data structure of the other computer largest fixed input sequence number memory | storage part in the embodiment. The figure which shows the layout of the protocol data transmitted / received between the computers which comprise the distributed system of the embodiment. The 1st flowchart which shows the operation | movement procedure of the basic part which performs one delivery of the ordered multicast which each computer performs in the same embodiment. The 2nd flowchart which shows the operation | movement procedure of the basic part which performs one delivery of the ordered multicast which each computer performs in the same embodiment. The 1st flowchart which shows the operation | movement procedure for eliminating the delay of multiplexing execution which each computer performs in the same embodiment. The 2nd flowchart which shows the operation | movement procedure for eliminating the delay of multiplexing execution which each computer performs in the same embodiment. The 3rd flowchart which shows the operation | movement procedure for eliminating the delay of multiplexing execution which each computer performs in the same embodiment. FIG. 10 is a fourth flowchart showing an operation procedure for eliminating a delay in multiplexing execution, which is executed by each computer in the embodiment. 6 is an exemplary flowchart illustrating an operation procedure for determining the configuration of a distributed system executed by each computer in the embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Input reception queue part, 2 ... Sort multicast part, 3 ... Application program, 4 ... Program state management part, 5 ... Output filter part, 21 ... Input sequence number memory | storage part, 22 ... Input packet journal memory part, 23 ... Protocol Data transmission / reception unit, 24 ... step number storage unit, 25 ... candidate packet storage unit, 26 ... input packet determination determination unit, 27 ... maximum determination input sequence number storage unit, 28 ... delay storage unit, 29 ... skip determination unit, 100- 1 to 100-7, 100-i ... computer, 210 ... configuration determination unit, 210a ... failure detection unit, 211 ... configuration storage unit, 212 ... maximum sequence number delay tolerance storage unit, 213 ... other computer maximum fixed input sequence number Storage unit 261 ... Input candidate collection unit 262 ... Agreement unit 1000 ... Distributed system 2000 ... Client device A ... N Network, B ... network.

Claims (5)

A distributed system that synchronously operates n computers (m is an integer satisfying 4 <n ≦ m at the start of operation ) among m computers (m is an integer greater than 4) connected via a network. ,
Each of the m computers is
Configuration storage means for storing information identifying n computers to be operated synchronously as information identifying the computers constituting the distributed system;
The n computers constituting the distributed system identified by the information stored in the configuration storage means are operated synchronously, and (n−f) computers (f is 3f <of the n computers). In order to guarantee multiplexing at or above the maximum allowable number of failures that is the largest integer satisfying n), the input data is arranged and multicasted by agreeing the input data with the (n−f) computers or more. Means of agreement,
Fault detection means for detecting a computer in which a fault has occurred from the state of n computers constituting the distributed system identified by information stored in the configuration storage means;
When the failure detection unit detects a computer in which a failure has occurred, the computer determines that an item to change the configuration of the distributed system has occurred and determines that an item to change the configuration of the distributed system has occurred Determines whether to change the configuration of the distributed system depending on whether or not (n−f) or more exist, and changes the information stored in the configuration storage means when it is determined to change the configuration of the distributed system Configuration determining means for updating to indicate the configuration of the later distributed system, and based on the number n of computers constituting the changed distributed system, the maximum that matches the changed distributed system A distributed system comprising: a configuration determining unit that determines an allowable fault count f . When it is determined that an item that should change the configuration of the distributed system has occurred, the configuration determination unit provides specific input data indicating the fact to the agreement unit, and causes the specific input data to be aligned and multicast. When it is determined that there are at least (n−f) computers that have been determined to have changed the configuration of the distributed system and the configuration of the distributed system is determined to be changed, the occurrence of the failure is detected. It has been as computer is disconnected from the distributed system, distributed system of claim 1, wherein updating the information stored in said configuration storage means. The agreement means manages the ordered multicast subject to agreement by a sequence number serially assigned to the ordered multicast,
Each of the m computers further includes another computer maximum sequence number storage means for storing the maximum sequence number of the ordered multicast agreed for each other computer,
The failure detection means includes a difference between the latest ordered multicast order number managed by the agreement means and the largest ordered multicast order number for each other computer stored in the other computer maximum order number storage means. The distributed system according to claim 1, wherein the occurrence of a failure in the other computer is detected depending on whether the value exceeds a threshold value. In a distributed system in which m computers (m is an integer greater than 4) are connected via a network, n computers (n is an integer satisfying 4 <n ≦ m at the start of operation) among the m computers. It is a multiplexing control method that operates synchronously and guarantees multiplexing at (n−f) units (f is the maximum integer satisfying 3f <n),
Each of the m computers includes configuration storage means for storing information identifying n computers to be operated synchronously as information for identifying computers constituting the distributed system, agreement means, failure A detection means and a configuration determination means;
The multiplexing control method includes:
The agreement means of each of the m computers operates n computers constituting the distributed system identified by the information stored in the configuration storage means synchronously, and the n computers In order to guarantee multiplexing at or above (n−f) units (f is the maximum allowable number of faults that is the maximum integer satisfying 3f <n), input data on the (n−f) computers or more. Arranging and multicasting the input data with the agreement of
The failure detection means detecting a computer in which a failure has occurred from the state of n computers constituting the distributed system identified by the information stored in the configuration storage means;
When a computer in which a failure has occurred is detected, it is determined that an item to change the configuration of the distributed system has occurred, and a computer that has determined that an item to change the configuration of the distributed system has occurred (n−f A step in which the configuration determining means determines whether to change the configuration of the distributed system depending on whether or not there are more units;
When it is determined that the configuration of the distributed system is to be changed, the configuration determining unit is configured to allow the maximum allowable value that is suitable for the distributed system after the change based on the number n of computers configuring the distributed system after the change. multiplexing control method characterized by comprising the steps of determining the number of error f, and to update, as shown the structure of the distributed system after the change the information stored in said configuration storage means. Each of the m computers further includes another computer maximum sequence number storage means for storing the maximum sequence number of the ordered multicast agreed for each other computer,
The agreement means manages the ordered multicast subject to agreement by a sequence number serially assigned to the ordered multicast,
The failure detection means includes a difference between the latest ordered multicast order number managed by the agreement means and the largest ordered multicast order number for each other computer stored in the other computer maximum order number storage means. Detects the failure of the other computer depending on whether the value exceeds the threshold
5. The multiplexing control method according to claim 4, wherein:

Images