JP4038510B2 - Cluster node status detection and communication - Google Patents

Description

The present disclosure relates generally to computer networks.
More particularly, the present disclosure relates to a cluster of interconnected computer systems.

A cluster is a parallel or distributed system comprising a collection of interconnected computer systems or servers that are used as a single integrated computing unit.
Cluster members are called nodes or systems.
The cluster service is a collection of software of each node that manages activities related to the cluster.
The cluster service considers all resources as the same object.
Resources can include physical hardware devices such as disk drives and network cards, or logical items such as logical disk volumes, TCP / IP addresses, entire applications, entire databases, among others.
A group is a collection of resources managed as a single unit.
In general, a group includes all of the components necessary to run a particular application and allow a user to connect to services provided by that application.
Operations performed on a group typically affect all resources contained within that group.
Clustering increases system availability, performance, and network system and application capacity by connecting two or more servers together.

Clustering can be used for parallel processing or parallel computing using two or more CPUs simultaneously to execute an application or program.
Clustering is a popular strategy for implementing parallel processing applications because it allows system administrators to utilize existing computers and workstations.
Because it is difficult to predict the number of requests issued to networked servers, clustering distributes processing and communication activities uniformly across the network system so that no single server is overwhelmed. It is also useful for load balancing.
If one server is at risk of being overwhelmed, the request can be forwarded to another clustered server with greater capacity.
For example, a busy website can use two or more clustered web servers to use a load balancing scheme.
Clustering also increases extensibility by allowing new components to be added as the system load increases.
Furthermore, clustering also simplifies the management of groups of systems and their applications by allowing the system administrator to manage an entire group as a single system.
Clustering can also be used to increase the fault tolerance of a network system.
If one server experiences an unexpected software or hardware failure, another clustered server can take over the operation of the failed server.
Thus, if any hardware or software component of the system is compromised, the user may suffer performance degradation but not lose access to the service.

Current cluster services include, among others, Microsoft Cluster Server Server (MSCS), Novell Network Cluster Server, designed by Microsoft for clustering its Windows NT 4.0 operating system and Windows 2000 Advanced Server operating system. (NWCS) and the like are included.
For example, MSCS supports the clustering of two NT servers and provides a single high availability server.

Clustering can also be implemented in a computer network utilizing a storage area network (SAN) and similar network connection environment.
A SAN network allows a storage system to be shared between multiple clusters and / or servers.
A SAN storage device can be structured in a RAID configuration, for example.

To detect system failures, clustered nodes can monitor each other's health using a heartbeat mechanism.
A heartbeat is a signal transmitted from one clustered node to another clustered node.
The heartbeat signal is usually transmitted over Ethernet or a similar network.
However, this network is also used for other purposes.

When an expected heartbeat signal is not received from a node, a failure of that node is detected.
If a node failure is detected, the clustering software can, for example, transfer the entire resource group of the failed node to another node.
Client applications that are affected by the failure can detect session failures and reconnect in the same way as the original connection.

When a heartbeat signal is received from a node of the cluster, that node is typically defined as being in an “up” state.
In the up state, the node is assumed to be operating properly.
On the other hand, when a heartbeat signal is no longer received from a node, that node is typically defined as being in a “down” state.
In the down state, the node is assumed to have failed.

One embodiment disclosed herein relates to a method of communicating status from a node of a cluster of a computer system.
A first status signal is received from the compute node and a default status signal is generated.
These first status signal and default status signal are used to generate a second status signal.

Another embodiment disclosed herein relates to a method of communicating node status within a cluster of computer systems.
A first signal indicating the status of the current node is generated.
A second signal indicating the status of the preceding node is received.
If the current node is in the cluster, the first signal is sent to the next node, and if the current node is removed from the cluster, the second signal is sent to the next node.

Another embodiment disclosed herein relates to an apparatus for communicating status from a node of a cluster of a computer system.
The apparatus includes at least an input, a default signal generator, and an output signal generator.
The input is configured to receive a first status signal from the compute node and the default signal generator is configured to generate a default status signal.
The output signal generator is configured to generate a second status signal using the first status signal and the default status signal.

Another embodiment disclosed herein relates to an apparatus for communicating node status within a cluster of computer systems.
The circuitry is configured to generate a first signal indicating the status of the current node, and the input is configured to receive a second signal indicating the status of the preceding node.
If the current node is in the cluster, send the first signal to the next node, and if the current node is removed from the cluster, send the second signal to the next node, A selection circuit is configured.

  A conventional technique for reporting the status of clustered nodes has been described above. In conventional techniques, a heartbeat mechanism is used and a node is determined to be in either an “up” state or a “down” state.

This conventional technique is inadequate and disadvantageous in various cases.
For example, even if the target critical application is not functioning (ie, the application is down), the node running that application is still sending its heartbeat signal There is a case.

In that case, the cluster will still assume that the node is up, even if an important application is down.
In another example, the cluster may not receive an expected heartbeat signal from a node and may therefore assume that the node is down.
However, the node may actually be up (ie, operating properly) and the missing heartbeat signal may be due to an interconnect failure rather than a down. is there.

In addition, conventional techniques typically utilize existing circuitry to generate and transmit status signals.
This existing circuitry is also used for other communications within the cluster.
In contrast, Applicants have determined that it is advantageous over conventional techniques to use dedicated circuitry specifically designed to robustly generate and transmit status signals.

The efficiency (percentage uptime) of a high availability (HA) cluster is primarily that one of the nodes in the cluster has ceased performing a useful calculation or storage function (ie, the node is effectively down). It can be seen that this is determined by the amount of time it takes for the cluster to recognize.
If the cluster determines that the node is effectively down, the clustering software can perform the tasks necessary to maintain the remaining execution of the node with little interruption to the user task.

However, as mentioned above, the conventional techniques used to determine the state of a cluster node are not accurate in various cases.
Conventional techniques may result in false (unnecessary) failover or detection failure.
A detection failure is when the cluster level software cannot switch when it should switch from a bad node to a good node.
Furthermore, conventional techniques often require an undesirably long time to detect a node down condition.

FIG. 1 is a schematic diagram of a node 100 of a cluster according to an embodiment of the present invention.
Node 100 includes a conventional computing subsystem 102 and signal hardware circuitry 106.
The computing subsystem 102 includes computing elements, which typically include one or more central processing units (CPUs), memory, and the like.
In particular, the computing subsystem 102 generates and outputs a subsystem status signal 104.
The signal hardware circuitry 106 receives the subsystem status signal 104 and outputs a node status signal 108.
The node status signal 108 can be output to the next node in the cluster.
These signals are further described below in connection with subsequent figures.

FIG. 2 is a schematic diagram of the signal hardware 106 according to one embodiment of the present invention.
The signal hardware 106 can include a signal generator 202 and an output signal generator 206.

Signal hardware 106 receives subsystem status signal 104 from compute node 102.
An exemplary timing diagram for subsystem status signal 104 is shown at the top of FIG.
As shown in FIG. 4, the subsystem status signal 104 can be in a GOOD (good) (up) state or a BAD (bad) (down) state.
For example, the GOOD state can be represented by a high (logic 1) signal and the BAD state can be represented by a low (logic 0) signal.
If the computing subsystem 102 is functioning properly (operating properly), the subsystem status signal 104 should be driven to the GOOD state.
If the computing subsystem 102 is not functioning properly, the GOOD state should not be driven on the subsystem status signal 104.
The lack of a GOOD signal means that the system is BAD (down).

The signal generator 202 generates a default BAD (default down) signal 204.
An exemplary timing diagram for the default BAD signal 204 is shown at the bottom of FIG.
As shown in FIG. 4, the default BAD signal 204 comprises a signal that is asymmetric in period (not just logic levels).
For example, as shown, the default BAD signal 204 can comprise an asymmetric toggle pattern signal or a pulse modulated signal.
The toggle pattern shown in FIG. 4 is just an example showing one possibility.
Such toggle patterns can be generated using various electronic circuitry known to those skilled in the art.

Output signal generator 206 is configured to receive both default BAD signal 204 and subsystem status signal 104.
The output signal generator 206 generates and outputs a node status signal 108 using these two signals.

FIG. 3 is a schematic diagram of an output signal generator 206 according to one embodiment of the present invention.
The output signal generator 206 can include a pull-down element 302 and a logic function block 304.

As shown in FIG. 3, the pull-down element 302 is connected to a line that receives the subsystem status signal 104.
When the high level (GOOD in this embodiment) is not driven from the computing subsystem 102, the pull-down element 302 forces the line to a low level (BAD in this embodiment).
Thus, even if the computing subsystem 102 is not generating any signal, the subsystem status signal 104 is conveniently drawn to the level corresponding to the BAD condition.

In an alternative embodiment, a low level of subsystem status signal 104 can correspond to a GOOD state and a high level can correspond to a BAD state.
In that case, a pull-up element can be used to achieve this advantageous effect.
Pull-down circuit elements and pull-up circuit elements (voltage level pull elements) are known to those skilled in the art.

As shown in FIG. 3, logic function block 304 receives default BAD signal 204 along with subsystem status signal 104.
According to one embodiment, the logic function block 304 can comprise an exclusive-or (XOR) gate.
In other embodiments, different functions can be utilized.

An exemplary timing diagram for the node status signal 108 generated by the logic function block 304 is shown in FIG.
In these timing diagrams, the logic function block 304 is an XOR gate, and the signals input to the XOR gate are the signals (104 and 204) shown in FIG.

First, consider the node status signal 108 that is generated when the subsystem status signal 104 corresponds to a BAD state.
In this case, the XOR gate receives the low level of the default BAD signal 204 and the subsystem status signal 104 and performs an exclusive OR operation on these two signals.
The result is a node status signal 108 shown at the top of FIG.
In this example, the node status signal 108 is a periodic signal representing the BAD state.
More specifically, here, the node status signal 108 has the same periodic form (in this example, a toggle pattern or a pulse modulation pattern) as the default BAD signal 204.

Next, consider the node status signal 108 that is generated when the subsystem status signal 104 corresponds to a GOOD state.
In this case, the XOR gate receives the high level of the default BAD signal 204 and the subsystem status signal 104 and performs an exclusive OR operation on these two signals.
The result is a node status signal 108 shown at the bottom of FIG.
In this example, the node status signal 108 is a periodic signal representing the GOOD state.
More specifically, here, the node status signal 108 is a different periodic signal that is a complement to the default BAD signal 204.

FIG. 6 is a schematic diagram of a status passing circuit 600 according to one embodiment of the present invention.
This circuit 600 advantageously allows the node status signal 108 of the preceding node to pass through the current node when the current node is down.

Node N signal hardware 106 generates node N node status signal 108.
For example, the signal hardware 106 and the node status signal 108 may be as described above in connection with the previous drawings.

The selection circuit 602 receives the node status signal 108 of the node N.
Further, the selection circuit 602 also receives the node status signal 108 from the node N-1 (another node in the cluster).
The selection circuit 602 operates on these two signals to generate a status output signal 604 that is transmitted to node N + 1 (the next node in the cluster).
In one embodiment, the selection circuit 602 can comprise a multiplexer (MUX) that selects one of two status signals (via the status output signal 604) and passes it to the next node.
If node N's computing subsystem (computing element) has been previously removed from the cluster (eg, due to node failure, maintenance, or other reasons), the status from node N-1 is passed.
If the node N compute subsystem is currently in use by the cluster, the status of node N is passed.
Thus, even if node N is down, the status of node N-1 is still conveniently evaluated by the system.

Note that if node N-1 is down, the status signal received from node N-1 may originate from node N-2.
If both nodes N-1 and N-2 are down, the status signal received from node N-1 may originate from node N-3.
The same applies hereinafter.

FIG. 7 is a schematic diagram of a node 700 of a cluster according to another embodiment of the invention.
The node 700 in FIG. 7 is the same as the node 100 in FIG.
However, in FIG. 7, node 700 also generates subsystem degradation status signal 702 in addition to conventional subsystem status signal 104.
In combination with the conventional subsystem status signal 104, the subsystem degradation status signal 702 extends the reported state from a simple binary signal to a multi-state (3 states or more than 4 states) signal.

For example, the subsystem degradation status signal 702 can indicate the DEGGRADED (degraded) state or NOT_DEGRADED (non-degraded) state of the computing subsystem 102.
The DEGRADED state can be defined as the time when one or more faces of a node are not “running standard” and as a result the node can be removed from the HA cluster.
For example, the following rules can be used.
Rule D1: The computing subsystem loses more than 50% performance.
Rule D2: A dangerous (some level below critical) chassis code was received.
These rule variants and added rules can also be used to define DEGRADED states for a particular system.
For example, the performance ratio before entering the degraded state may be different from 50%.
The performance ratio may be as high as 75% or as low as 25%.

In one embodiment, the subsystem degradation status signal 702 can be a simple flag that indicates whether the node is degraded or not degraded.
In other embodiments, there may be multiple levels of degradation.
These multiple levels of degradation can be implemented using multiple levels of degradation of the degradation.
In other words, instead of having only a single DEGRADED state, multiple levels of degradation can be defined by rules.
Using multiple levels of degradation advantageously provides additional information for the HA clustering software decision process regarding the cluster's node management method to the HA clustering software.
For example, the degradation level may depend on the performance loss ratio.

In one particular embodiment, the node degradation status signal 704 may comprise a set of lines that digitally provide a degradation state to the next node in the HA cluster.
These lines can be pulled down with resistors.
One embodiment can be as follows.
That is, all of these digital lines being logic 0 can indicate that the node is BAD.
All of these lines being logic 1 can indicate that the node is GOOD.
Other values in between can indicate the degradation level of the node, with higher values indicating more functioning.

FIG. 8 is a schematic diagram of a status passing circuit 800 according to another embodiment of the present invention.
The circuit 800 in FIG. 8 is similar to the circuit 600 in FIG.
However, in FIG. 8, the selection circuit 802 also receives the node degradation status signal 704 from the nodes N and N-1.

The selection circuit 802 operates on the input signal, and generates a status output signal 804 including the added deterioration status information along with the GOOD / BAD status information from either the node N or the node N-1.
Conveniently, this degradation status information can be used by cluster level software such as “matching” with GOOD / BAD status information, resulting in a more reliable set of status information.

The above disclosure includes various advantages over the prior art.
First, dedicated hardware is designed and used to reliably send node status information to the cluster.
This should improve the high reliability of the cluster.
Second, the GOOD state is sent only when the appropriate software on the node is up and running and the GOOD state can be signaled.
As a result, the hardware does not indicate a GOOD state when the software is down.
Third, the above disclosure provides a solution to the problem of distinguishing "no heartbeat" due to a node being down from "lost heartbeat" due to an interconnect failure.
This is done by providing a default BAD signal that the working node can change to a GOOD signal.
Fourth, the above disclosure provides a separate output for degraded type status signals, thus ensuring communication of such degraded conditions.
Furthermore, the degradation status signal allows the cluster level software to use a “voting scheme” to quickly and accurately determine whether a node is actually down.
For example, the voting scheme can utilize three signals including a GOOD / BAD signal, a DEGRADED / NOT_DEGRADED signal, and a normal Ethernet connection provided by the cluster.

In the above description, numerous specific details are given to provide a thorough understanding of embodiments of the invention.
However, the above description of example embodiments of the invention is not intended to be exhaustive or to limit the invention to the precise form disclosed.
Those skilled in the art will appreciate that the invention may be practiced without one or more of the specific details, and that the invention may be practiced with other methods, components, and the like.
In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the invention.
As will be appreciated by those skilled in the art, specific embodiments and examples of the invention are described herein for purposes of illustration, and various equivalent modifications are possible within the scope of the invention.

These changes can be made to the invention in light of the above detailed description.
The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims.
Instead, the scope of the present invention should be determined by the appended claims.
The claims should be construed in accordance with established principles of claim interpretation.

It is the schematic which shows the node of the cluster by one embodiment of this invention. FIG. 2 is a schematic diagram of signal hardware according to an embodiment of the present invention. 1 is a schematic diagram of an output signal generator according to an embodiment of the present invention. FIG. FIG. 6 is a timing diagram of a subsystem status signal and a default BAD signal according to an embodiment of the present invention. FIG. 6 is a timing diagram of a node status signal according to an embodiment of the present invention. It is the schematic of the status passage circuit by one embodiment of this invention. FIG. 4 is a schematic diagram of nodes of a cluster according to another embodiment of the present invention. FIG. 6 is a schematic diagram of a status passing circuit according to another embodiment of the present invention.

Explanation of symbols

100 ... node,
102: Computing subsystem (computing node),
104 ... subsystem status,
106: Signal hardware,
108 ... node status,
104 ... subsystem status,
106: Signal hardware,
108 ... node status,
202... Signal generator,
204 ... default BAD signal,
206 ... Output signal generator,
104 ... subsystem status,
108 ... node status,
204 ... default BAD signal,
304 ... logic function block,

Claims (13)

A method for communicating status from a first node of a cluster of computer systems to a second node comprising :
Receiving a first status signal of a computing node included in the first node ;
Generating a default status signal of the first node ;
Forcing the first status signal to be in a down state if the compute node does not actively drive the first status signal to a state that should be in an up state;
Applying a logic function to the first status signal and the default status signal to generate a second status signal for indicating the status of the first node to the second node. . When the first status signal indicates the up state,
The second status signal is:
Including a first periodic signal indicative of the up state;
When the first status signal indicates the down state,
The second status signal is:
The method of claim 1, comprising a second periodic signal indicative of the down state. If the first status signal does not indicate an up state or a down state,
The method of claim 2, wherein the second status signal defaults to the second periodic signal. The first periodic signal and the second periodic signal are:
4. The method of claim 3, comprising different toggle type signals. The first periodic signal and the second periodic signal are:
The method of claim 4, comprising complementing each other. Receiving a first degradation status signal from the compute node;
Generating a default degradation status signal;
The method of claim 1, further comprising using the first degradation status signal and the default degradation status signal to generate a second degradation status signal. The degradation status signal is
The method of claim 6 comprising multiple levels of degradation. A device for communicating status from a cluster node of a computer system,
  An input configured to receive a first status signal from the compute node;
  A default signal generator configured to generate a default status signal;
  An output signal generator configured to generate a second status signal using the first status signal and the default status signal;
  With
  The output signal generator includes a pull element having a voltage level effective for the first status signal.
  Including the device. The output signal generator includes an exclusive OR circuit effective for the first status signal and the default status signal.
The apparatus of claim 8 comprising: The output signal generator further includes a first periodic signal in which the second status signal indicates an up state when the first status signal indicates an up state, and the first status signal is down When indicating a state, the second status signal is configured to include a second periodic signal indicating a down state
The apparatus according to claim 8. The output signal generator is further configured such that the second status signal becomes the default second periodic signal when the first status signal does not indicate either an up or down state. The
The apparatus according to claim 10. The first and second periodic signals are different toggle type signals
The apparatus of claim 11 comprising: The first and second periodic signals are complementary signals
The apparatus of claim 12 comprising:

Images