WO1998049621A1

WO1998049621A1 - Control message interfacing in a redundant server environment

Info

Publication number: WO1998049621A1
Application number: PCT/US1998/008144
Authority: WO
Inventors: William P. Delaney; Gerald J. Fredin; Andrew J. Spry
Original assignee: Symbios, Inc.
Priority date: 1997-04-25
Filing date: 1998-04-22
Publication date: 1998-11-05
Also published as: AU7149698A

Abstract

Methods and associated apparatus for using a redundant network communication path between networked redundant servers as a fallback communication path for control communications between such redundant servers. A primary control communication path (224) is dedicated between two redundant servers (200 and 210) for purposes of exchanging control and status information between redundant servers (200 and 210) in a networked client/server computing environment. A pair of network communication paths (120 and 122) between the redundant network servers (200 and 210) is used to assure reliable exchange of data among servers and networked clients. A first of the pair of network paths is referred to as the primary network (120) while the other is referred to as the standby network (122). Network connections are typically LAN or WAN connection media while control communication paths are typically RS232 or LAN connection media. Whereas prior architectures added a second control communication path to assure reliable exchange of control and status information between the redundant servers, the architecture of the present invention uses the standby network as a fallback control communication path in case of failure of the primary control communication path. Use of the standby network for fallback control communications obviates the need for a physically separate redundant control communication path which in turn reduces complexity and associated costs of the redundant servers.

Description

CONTROL MESSAGE INTERFACING IN A REDUNDANT SERVER

ENVIRONMENT

1. Technical Field

This invention relates to distributed computing environments and in particular to methods and associated apparatus for using a standby network interface connection between redundant servers for control messages therebetween as a fallback communication path for a failed primary control communication path.

2. Background Art

Distributed computing environments, as the term is used herein, are those in which client processes request services from server processes. In particular, distributed computing environments are often used in conjunction with network communication media and protocols to enable distribution of the various communicating processes across physically remote nodes and locations. A server process may, for example be operable in a server computing node while a client process which requests services from the server process may be operable in the same node or in a remote node connected via communication networks.

Computing nodes in a distributed computing environment connect to the network communication medium via network adapters or network interface cards (also referred to herein as NICs). A NIC provides circuits to receive and transmit data over the network communication medium on behalf of the computing node in which it is housed. As used herein, such a computing system may include general purpose computer systems (e.g., host systems) into which a NIC is inserted as well as peripheral devices with embedded NIC circuits which attach the peripheral to the network medium. A server or service process may be, for example, a file server providing coordinate access to files on behalf of a plurality of client processes, a print server providing printer functions to a plurality of client process, or any other function which provides services on behalf of a client process. A server process may therefore be operable within a general purpose computing node or may be embedded within a special purpose server device. A client process is therefore any process which requests such services from a server process whether operable in a general purpose computing environment or embedded in a special purpose device.

It is known in the art to provide redundancy as a means for improving reliability and availability of a computing application. In a client/server distributed environment, redundant server nodes are often utilized to help assure reliable access to the service(s) provided thereby. Typically, one server node or system provides a particular service while its redundant paired server node remains idle (with respect to provision of the same service). The idle second server node takes over the provision of the service when it senses that the first server has failed in some manner.

In high reliability redundant server environments, each redundant server is connected to redundant networks for the exchange of messages between the servers and requesting client nodes. These redundant network connection are preferably reserved for client/server message exchange pertaining to the intended application. A separate communication path is typically used between the redundant servers for exchange of status and control information regarding the state of operation of each redundant server. The volume of such control and status information is generally low but it is none the less preferred that the control information is segregated to a distinct communication channel. This segregation reduces the overhead load imposed on other nodes of the network. More importantly, segregation of the control information exchange to a separate channel improves the speed of processing the control and status information exchange messages. These messages are isolated from other more general network traffic and may thereby be rapidly processing by the associated processing elements in each server. This control communication path may be any of several well known communication media and standards (e.g., RS232, LAN, etc.). In general, it is a problem in such environments to reliably communicate control information between redundant servers so as to reliably determine the status of each controller by its redundant mate. Prior solutions have applied the same redundancy principles to the control communication path as is applied to the client server information network communication paths. Specifically, prior solutions have used redundant pairs control communication paths distinct from the client/server redundant networks to assure reliable communication of control and status information among the redundant servers. Each communication path used by a server, whether for data exchange or for control and status information exchange, adds complexity and hence cost to each server and the subsystem of which they are components.

It can be seen from the above discussion that a need exists for an improved method for reliable exchange of status and control information between redundant servers that reduces complexity as compared to prior designs.

3. Disclosure of Invention

The present invention solves the above and other problems, thereby advancing the state of the useful arts, by providing methods and associated apparatus for eliminating the distinct redundant communication path for control and status in favor of a single non-redundant control communication path. Methods of the present invention utilize the standby (redundant) network communication path between the redundant servers as a fallback communication path for control and status when the lone control communication path indicates a failure.

Specifically, redundant servers operable in accordance with the present invention are connected via redundant network communication paths used primarily for client/server data exchange. A first of the redundant network paths is designated the primary path while the other network path is designated the standby path. The standby network communication path is essentially unused while the redundant servers are normally operating to exchange information with client nodes over the primary network communication path.

The redundant servers exchange control and status information via the primary (sole) control communication path. As in prior designs, the sole control communication path may use any of several well known communication media and protocols (e.g., RS232, LAN, etc.). When a server detects a possible failure of another server due to loss of communication via the sole control communication path, it verifies the failure by using the standby network communication path for fallback communication to attempt exchange of control and status information with the presumed failed server. If the fallback communication is successful, the failure may indicate a failure of the primary (sole) control communication path rather than a failure of the other server. If the fallback communication is not successful, the server may presume with higher confidence that the other server has indeed failed as distinct from a failure of the control communication path per se.

Use of the methods and associated apparatus of the present invention therefore obviates the need for a redundant control communication path distinct from the client/server network communication paths. Elimination of this communication path reduces complexity (and therefore cost) of each server and thus simplifies the subsystem of which they are a part.

Methods of the present invention preferably utilize special messages for exchange of control information between the servers during fallback communications over the standby network communication path. Typical network communication protocols utilize a variety of header portions in information packets exchanged via the network medium. These header portions are used to identify types of information as well as source and destination addressing. The special messages utilized by the present invention preferably use undefined or resented header blocks values so as to be easily distinguishable by the servers as non-standard network messages. Servers exchanging the special message can quickly identify the processing required in response to receipt of such a special message and other nodes may quickly determine that the special messages are to be ignored. Thus, the messages may be rapidly routed to the proper processing so that servers are minimally impacted by the additional messaging traffic. It is therefore an object of the present invention to provide methods and associated apparatus for reducing the complexity of redundant servers.

It is another object of the present invention to provide methods and associated apparatus for reducing the complexity of redundant servers by obviating the need for a redundant control communication path between the redundant servers.

It is still another object of the present invention to provide methods and associated apparatus for using a communication path nominally reserved for data exchange among the redundant servers as a fallback control communication path. It is a further object of the present invention to provide methods and associated apparatus to eliminate one of a redundant pair of control communication paths between redundant servers by substituting temporary use of a data communication path to serve as a fallback control communication path. The above and other objects, aspects, features and advantages of the present invention will become apparent from the following detailed description and the attached drawings.

4. Brief Description of the Drawings FIG. 1 is a block diagram of a redundant server network environment configured and operable in accordance with known techniques utilizing a redundant pair of control communication paths;

FIG. 2 is a block diagram of a redundant server environment configured and operable in accordance with the improved methods and apparatus of the present invention whereby one of the previously required redundant control communication paths of FIG. 1 is eliminated and replaced by fallback use of a standby network data path for control communication purposes; and

FIG. 3 is a flowchart describing the operation of a redundant server operable in accordance with the present invention to utilize a data communication path as a fallback control communication path between redundant servers.

5. Detailed Description of the Preferred Embodiments

While the invention is susceptible to various modifications and alternative forms, a specific embodiment thereof has been shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that it is not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims.

FIG. 1 is a block diagram of a networked redundant server computing environment operable in accordance with prior techniques devoid of the improvements of the present invention. Server system 100 (server #1) and server system 110 (server #2) of FIG. 1 are operable as a redundant pair of servers, each operable to take over operation of the other in case a failure is sensed in operation of the other.

System 100 and system 110 are each connected to redundant networks 120 and 122 via redundant network interface circuits (NIC) 106, 108, and 116, 118, respectively. In particular, system 100 is connected to its primary network 120 via NIC 106. System 110 acts as a standby server connected to network 120 via its NIC 118. System 100's primary network 120 is therefore system 110's standby network 120. The servers are essentially mirrors of one another. Therefore, system 110 is connected to its primary network 122 via NIC 116 and system 100 is connected to the same network 122 via its NIC 108 as a standby network server. This topology enables each server to assume the operation of the other server by using the other's identity on the other's primary network. In other words, system 100 acts as a redundant server to assume the identity of system 110 on standby network 122 in case an error is detected in operation of system 110. Likewise, system 110 acts as a redundant server to assume the identity of system 100 on standby network 120 in case an error is detected in operation of system 100. In case one system senses a failure of the other, a take over processing component (102 in system 100 and 112 in system 110) performs processing appropriate for the server in which it operates to take over processing responsibility of the other (apparently failed) server system. The server that takes over therefore performs processing of client requests on behalf of the failed server by using its standby network connection and the identity of the failed server thereon.

Systems 100 and 110 exchange control and status information to determine one anothers operational state and to thereby determine when a take over process may be required. For example, a common technique is to exchange so-called "heartbeat" messages between the two systems. Each system awaits periodic receipt of the heartbeat message from the other system. Receipt of the message before a time-out condition arises is a signal that the other system is properly operational. A time-out condition (also referred to as a watchdog time-out) before receipt of an expected heartbeat message is a signal that the other system may have failed.

Such control and status information (e.g., heartbeat or watchdog message exchange) is performed over a primary control communication path 124 connecting the systems 100 and 110. The primary control communication path 124 is often implemented as another network connection or as a simpler RS232 (or other) serial connection. Generally the volume of such message exchange is low but the importance of rapidly processing the messages is obviously critical. Therefore the primary control communication path 124 is typically implemented as a distinct path separate from the redundant network communication paths 120 and 122. When a failure is sensed in message exchange via the primary control communication path 124, the system sensing the failure must determine whether the failure is likely caused by a failed server system at the other end of the control communication path 124 or by a failure of the control communication path 124 per se. To help assure a reliability of such a determination, prior techniques and architectures use a redundant control communication path 126 also separate and distinct from the network communication paths 120 and 122. The redundant control communication path 126 is also typically implemented as a network communication medium or may be a simpler serial link. If a failure is sensed by, for example, system 100 in exchange of control and status information via primary control communication path 124, then system 100 and 110 attempt the same exchange via redundant control communication path 126.

The redundant control communication paths 124 and 126 could be used in a number of manners to achieve the desired reliability in accordance with well known redundancy techniques. For example, the control and status information could be communicated exclusively via the primary control communication path 124 until a possible failure is detected. Both servers 100 and 110 monitor the redundant control communication path 126 in case the other server determines that a possible communication error has occurred and then attempts communication on the redundant path 126. In the alternative, heartbeat messages could be applied simultaneously to both redundant control communication paths 124 and 126. In case of apparent failure of one, messages simultaneously transferred to the other path may be processed instead. Or for example, when a possible failure is sensed on the primary control communication path 124, a specific message may be sent via the redundant control communication path to request that the other server switch to use of the redundant control communication path 126 for continued exchange of control and status information. Regardless of the particular implementation and protocol selected for redundant control communication path 126, its mere presence adds complexity and thereby cost to each server system 100 and 110. In addition, redundant control communication path 126 is infrequently used or required to assure reliable operation of the redundant pair of server systems 100 and 110. FIG. 2 is a block diagram of a networked redundant server computing environment operable in accordance with the present invention. Server system 200 (server #1) and server system 210 (server #2) of FIG. 2 are operable as a redundant pair of servers, each operable to take over operation of the other in case a failure is sensed in operation of the other. System 200 and system 210 are each connected to redundant networks 120 and 122 via redundant network interface circuits (NIC) 206, 208, and 216, 218, respectively. Systems 200 and 210 of FIG. 2 are essentially mirrored images of one another and redundantly connected via their respective NICs to the redundant network communication paths 120 and 122 as discussed above with respect to FIG. 1.

As discussed above with respect to FIG. 1 , systems 200 and 210 exchange control and status information via primary control communication path 224 (e.g., heartbeat or watchdog messaging as above) to determine one anothers' operational state and to thereby determine when a take over process may be required. When a failure is sensed in message exchange via the primary control communication path 224, the system sensing the failure must determine whether the failure is likely caused by a failed server system at the other end of the control communication path 224 or by a failure of the control communication path 224 per se. Prior techniques, as discussed above, utilized a second distinct control communication path to assure redundant communication for control and status information exchange between the redundant server systems 100 and 110. The present invention rather incorporates methods in control and status elements 204 and 214 of each system, 200 and 210, respectively, which use their respective standby network connection to the other server as a fallback control communication path. Specifically, system 200 uses its standby network NIC 208 as a fallback control communication path if a failure is sensed in control message exchange via primary control communication path 224. Likewise, system 210 uses its standby network NIC 218 as a fallback control communication path in case of failure in control message exchange via primary control communication path 224. Use of the standby network communication paths as a fallback control communication path in this manner obviates the need known in prior techniques for a distinct redundant control communication path (e.g., 126 of FIG. 1). The present invention therefore reduces complexity and associated costs as compared to server architectures operable in accordance with prior techniques.

In the preferred embodiment of the present invention, the standby network communication path is used as a fallback communication path only to the extent that a single simple message is exchanged between the servers to verify the apparent failure detected on the primary control communication path. In other words, when a possible failure is detected on the primary control communication path 224, an "are you OK?" inquiry message is sent to the other server via the standby network NIC (e.g., 208 or 218 of FIG. 2). If the other message responds that its is "OK", then both server systems 200 and 210 log an error indication that the primary control communication path 224 has failed. Neither server need initiate take over processing on behalf of the other. Rather, the system of FIG. 2 continues to operate in a degraded mode in that neither server can determine the state of the other via the primary control communication path 224. When an operator repairs the failed communication path, normal operation may be restored. Those skilled in the art will readily recognize that an alternative embodiment may continue to utilize the fallback communication path of the standby network to continue essentially normal operation by exchanging control and status information (in addition to the "are you OK?" inquiry and response). As discussed below, such message exchange over the standby network would preferably utilize header structures that identify the messages as clearly outside the domain of normal network messages for client/server processing. For example, special media access addresses (MAC) may be used or special protocol specific addresses or ports may be used to identify the special nature of the control and status message exchange. As noted above, though the volume of such message exchange is low, the importance of rapidly processing the control and status information is critical.

FIG. 3 is a flowchart describing the operation of methods of the present invention operable within either server system 200 or 210. As noted above, in the preferred embodiment, each server is essentially a mirror image of the other. Each may be operable as a primary server for particular services and/or for particular clients. In addition, each monitors the operational status of the other through exchange of control and status information to determine if it needs to take over client request processing on behalf of the other server. The method described by the flowchart of FIG. 3 is therefore operable identically on either server system 200 or 210. If either server senses a failure in operational status of the other, it initiates processing to take over responsibility for processing client requests on behalf of the other (failed server).

As noted above, control and status information exchange such as heartbeat and/or watchdog message exchange, is performed via primary control communication path 224 between the server systems 200 and 210. When a possible error (failure) is sensed by one of the server systems in the control and status message exchange, that server uses its standby network link (standby NIC) to exchange similar messages over the standby network connection connecting the two servers. If the error persists even when communicated via the standby network, then the server may reliably assume that the other server has failed and may initiate processing to take over client servicing on behalf of the failed server system.

Element 300 is first operable to exchange control and status information with the other server via the primary control communication path. As noted above, such control and status information preferably comprises heartbeat messages which indicate, upon receipt, that operational health of the sending server. Element 300 therefore operates to receive such a heartbeat message from the other server or await a time-out period for such a message.

Element 302 is then operable to determine whether a possible failure was detected by operation of element 300. For example, a possible error may be detected if element 300 times out awaiting the next heartbeat message or if the message received includes supplemental information indicating a failure state of the other server. If no such possible error is detected, processing continue by looping back to element 300 to await the next expected heartbeat message from the other server. Those skilled in the art will recognize that the processing depicted in FIG. 3 is but a small portion of the overall processing that is performed within the server. Obviously, other processing pertaining to satisfaction of client requests is performed to provide the requisite services. In addition, processes are present and operable within each server to generate the requisite heartbeat messages for transmission to the other server. Such heartbeat and watchdog messaging techniques are well known to those skilled in the art as noted above with respect to the operation of FIG. 1.

Element 304 is operable in response to element 302 detecting a possible failure condition of the other server. As noted, a time-out awaiting a next expected heartbeat message or a particular supplemental status received with, or in lieu of, the heartbeat message may be indicative of such a failure. Element 304 therefore attempts to determine the operational status of the other server by inquiring of the other server "are you OK?" through an inquiry message sent via the standby network as a fallback communication path.

Element 306 is then operable to determine if the possible error condition detected by operation of element 302 is in fact accurate. If the attempt to verify the detected error so verifies the error, then the server will take over processing responsibilities for the other server. If, for example, another time-out occurs while awaiting a reply to the "are you OK?" inquiry, then element 308 is operable to initiate take over processing and assume the identity of the apparently failed other server. The server performing the take over process therefore provides the sen/ice it is originally configured to provide as well as the services provided by the failed server for which it has taken over.

If element 306 determines that the other server is operational (e.g., an appropriate response is received to the "are you OK?" inquiry), processing continues with element 310 to log the fact that the primary control communication path has failed. As noted above, in the preferred embodiment of the present invention, the fallback communication path is used to verify the failure of either the primary control communication path or of the other server. The failure of the control communication path does not cause either server to take over processing of the other. Rather, the error is logged and brought to the attention of an operator. Normal operation of the redundant servers may resume after the operator intervenes to repair or replace the failed primary control communication path.

Element 312 is then operable to determine whether the servers have been optionally configured to continue using the standby network communication path as a redundant replacement for the failed primary control communication path. If not, processing continues by looping back to element 300. The processing of element 300-312 may then continue repetitively until the operator intervenes to repair the failed control communication path. If element 312 determines that the servers have been configured to use the standby network as a redundant control communication path, elements 316 and 318 are next operable in lake manner to elements 300 and 302 above to exchange control and status information via the standby network as a redundant (fallback) control communication path. If element 318 determines that an error is sensed in the fallback communication of element 316, then processing loops back to element 300 to attempt a return to normal processing using the primary control communication path. As above, when the operator intervenes to repair the primary control communication path, normal processing will resume.

As noted above, heartbeat messages exchanged via the standby network fallback communication path are encoded and/or formatted in such a manner as to reduce the processing time required by the receiving server to receive and recognize the heartbeat message. For example, in typical networking protocols, messages include header information which identifies particular types of records as well as source and destination addresses. Such header information is preferably used by the methods of the present invention to identify the heartbeat messages as a special type outside the domain of standard network message types. This special message formatting allows the heartbeat messages to be rapidly routed through the networking modules for timely processing by the control and status modules of the sever software (e.g., 204 and 214 of FIG. 2). In the preferred embodiment of the present invention, control and status modules of each server (e.g., 204 and 214 of FIG. 2) are implemented using well known RPC (remote procedure call) network protocols. Use of this network protocol allows the message exchange of control and status information to utilize essentially any communication path for the control operations. Specifically, the RPC protocols may be as easily applied to an RS232 or LAN based primary control communication path (e.g., 224) as to a network communication path (e.g., 120 or 122).

In addition, those skilled in the art will recognize that either network link (the primary or standby network e.g., 120 or 122) may be utilized as a fallback communication path for the exchange of control and status information between the servers.

While the invention is susceptible to various modifications and alternative forms, a specific embodiment thereof has been shown by way of example in the drawing and has been described in detail. It should be understood, however, that it is not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims.

Claims

CLAIMS What is claimed is:

1. In a networked redundant server computing environment having a plurality of servers interconnected via a plurality of network communication medium, an apparatus for reliable control communication between said two of said plurality of servers comprising: a control communication path between said two of said plurality of servers for communication of control and status information between said two of said plurality of servers; and fallback means within said two of said plurality of servers to use a standby network of said plurality of network communication medium for fallback communication of control and status information between said two of said plurality of servers.

2. The apparatus of claim 1 further comprising: means for detecting an error in control communication over said control communication path wherein said fallback means is operable in response to the detection of said failure.

3. The apparatus of claim 1 wherein said fallback means includes: means for exchanging special control messages over said standby network distinct from standard network messages exchanged thereon.

4. The apparatus of claim 1 wherein said control communication path is a network communication medium.

5. The apparatus of claim 1 wherein said control communication path is a serial communication path.

6. A system comprising: a first server system; a second server system; a primary network connecting said first server system and said second server system nominally used for exchange of data between said first server system and said second server system; a standby network connecting said first server system and said second server system nominally used for exchange of data between said first server system and said second server system in response to failure of said first system; a control communication path between said first server system and said second server system for exchange of control and status information between said first server system and said second server system; means in said first server system for detecting a failure in exchange of said control and status information over said control communication path; and fallback means in said first server system to use said standby network for exchange of said control and status information in response to detection of said failure.

7. The system of claim 6 wherein said control communication path is a network communication medium.

8. The apparatus of claim 6 wherein said control communication path is a serial communication path.

9. In a system including a first server system and a second server system interconnected by a primary network and by a standby network and further connected by a control communication path, a method for exchanging control and status information between said first server system and said second server system comprising the steps of: exchanging said control and status information between said first server system and said second server system via said control communication path; detecting a failure in the exchange of said control and status information via said control communication path; and exchanging said control and status information between said first server system and said second server system via said standby network in response to detecting said failure.

10. The method of claim 9 wherein said control communication path is a network communication medium.

11. The method of claim 9 wherein said control communication path is a serial communication path.