JP2012173996A - Cluster system, cluster management method and cluster management program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent unnecessary service stop when split-brain occurs in a cluster system.SOLUTION: A cluster system 301 comprises: multiple cluster servers 151 and 152 for providing service to one or more clients CT1 to CTn and mutually transmitting and receiving a heartbeat signal through a heartbeat communication channel 50; and a quorum server 201 having a quorum. When failing to normally receive the heartbeat signal from the other cluster server, the cluster server 151 or 152 transmits a securement request for securing the quorum to the quorum server 201. When receiving the securement request from the other cluster server within a predetermined time of receiving the securement request from the cluster server, the quorum server 201 transmits success notification to the cluster servers 151 and 152 in response to the securement requests.

Description

  The present invention relates to a cluster system, a cluster management method, and a cluster management program, and more particularly, to a cluster system, a cluster management method, and a cluster management program that take measures against split brain.

  Conventionally, there is a cluster system as a technique that enables a server to continue a service provided to a client by another server in the event of a server failure.

  In such a cluster system, a plurality of servers constituting the cluster system generally detect a failed node, that is, a server by performing communication called a heartbeat with each other.

  Heartbeat communication refers to the monitoring of each server's aliveness by exchanging heartbeats to indicate that the server function has not stopped between multiple servers that are in a failover or business takeover relationship. It is a technology to do. While the heartbeat communication is performed, it is determined that the other server is operating normally, and no failover is performed.

  On the other hand, when the heartbeat communication is interrupted, it is determined that the partner server is down, and services such as business applications provided by the partner server are taken over by the own device. As a result, the entire cluster system looks like a single server to clients using the service. Therefore, even when the processing is switched from the active server to the standby server, the client does not know from which server the service is provided.

  However, if multiple servers that make up the cluster system are isolated due to a failure in the communication path for heartbeat communication, the same service is started on the multiple servers and data is destroyed. Will occur. Such a state is called split brain.

  In order to avoid this split brain, a majority method using a quorum is known. In this majority method, it is necessary to secure more than 50% quorum in order to reconfigure the cluster.

  For example, in a two-node cluster, each server holds one quorum and one shared disk, and when split brain occurs, each server secures quorum on the shared disk as a tie-break process. Try. Since the server that has secured the quorum first has a majority (2/3) quorum, it is reconfigured as a regular cluster. That is, the server takes over the service running on the other server. The server that could not secure the quorum on the shared disk only secures its own quorum (1/3) and cannot secure the majority, so it panics and stops its operation.

  Oracle SunCluster 3.2 Release Note for Solaris OS Sun Cluster Quorum Server [online], [Search February 1, 2011], Internet <URL: http://docs.sun.com/app/docs/doc/ 819-6611 / gcvsc? A = view> (Non-patent Document 1) In a commercial UNIX (registered trademark) environment, a quorum is held in a node not belonging to a cluster instead of a shared disk. There is a cluster product that provides a method. In such a technique, since physical disk writing does not occur, a surviving server can be determined at high speed, and there is a merit that it can be controlled only by software without using a disk-dependent function.

  Japanese Patent Laid-Open No. 2006-48477 (Patent Document 1) discloses the following technique. That is, when communication between two servers (nodes) constituting a cluster system is interrupted, the start time of the next allocation time is determined by the node control unit in each node based on the definition information on the allocation time and its priority. Is calculated. Then, activation of a predetermined service process is delayed until the calculated start time. When the allocation time starts, the service process is started. If the service can be executed within the allocation time, the service process is continued. If the service cannot be executed, the operation is stopped.

  Japanese Unexamined Patent Application Publication No. 2009-223519 (Patent Document 2) discloses the following technique. That is, the cluster management unit of each node detects a node failure by performing heartbeat communication with each other. Then, each weighting processing unit checks the state related to the start of the service of the node, and updates the weight of the node itself in the weight information stored in the shared storage device according to the checked state. The tie breaker mechanism of each node determines whether the priority of the node itself is the highest based on the weight indicated by the updated weight information, and selects the node itself as the master node when the priority is the highest.

Oracle SunCluster 3.2 Release Note for Solaris OS Sun Cluster Quorum Server [online], [searched on February 1, 2011], Internet <URL: http://docs.sun.com/app/docs/doc/819- 6611 / gcvsc? A = view>

JP 2006-48477 A JP 2009-223519 A

  Services that were running on a server that could not have a majority of quorums are reconfigured to run on a server that could have a majority of quorums. For this reason, it is possible to continue providing services on different servers, but this operation temporarily stops the service.

  In addition, since there are many services such as databases that need to be checked for consistency by performing recovery processing or the like before startup, it takes time and cost to provide services when starting a cluster service.

  There are two possible causes for heartbeat interruption between cluster nodes or servers: when the server itself or the cluster management component hangs or slows down, and when there is no problem with the server but the heartbeat communication path fails. It is done. In the former case, it is unavoidable, but in the latter case, although the server can operate normally, the service is temporarily suspended.

  In other words, when split brain occurs in a cluster system, in the conventional method, in order to maintain the consistency of the cluster system, one cluster node (server) is activated and the other cluster node (server) is stopped. To perform the correct operation. For this reason, services are provided on the server to be stopped, and even when the server can operate normally, it is necessary to restart the service on the server to be operated. There was a problem that it was.

  Here, the heartbeat communication path is often multiplexed in order to improve reliability. However, there are cases where multiplexing cannot be performed due to cost and server configuration restrictions, and there are cases where a part of the communication path is not multiplexed, and the heartbeat may be interrupted due to a single failure in the communication path. For example, in an environment where a plurality of ports of one physical NIC (Network Interface Card) must be used and an environment where a network device on the path must be shared, a part of the communication path is not multiplexed.

  The techniques described in Patent Literature 1 and Patent Literature 2 are both methods for selecting a normal server that can start a service when a split brain occurs due to a heartbeat communication path failure and starting the service on a more normal node. It is.

  With these methods, even when the service can operate on a normal node, the service may fail over to a priority or more reliable server, which can cause unnecessary service stop and restart. There is sex.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a cluster system, a cluster management method, and a cluster system capable of preventing an unnecessary service stop when a split brain occurs in the cluster system. It is to provide a cluster management program.

  In order to solve the above problems, a cluster system according to an aspect of the present invention provides a plurality of clusters for providing a service to one or a plurality of clients and transmitting / receiving heartbeat signals to / from each other via a heartbeat communication path. A cluster system comprising a server and a quorum server having a quorum, wherein the cluster server is unable to normally receive the heartbeat signal from another cluster server, and a securing request for securing the quorum To the quorum server, and when the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, the quorum server sends a success notification for the reservation request. Send to the multiple cluster servers.

  In order to solve the above problems, a cluster management method according to an aspect of the present invention provides a plurality of services for providing a service to one or a plurality of clients and transmitting / receiving heartbeat signals to / from each other via a heartbeat communication path. A cluster management method in a cluster system comprising a cluster server and a quorum server having a quorum, wherein the quorum is secured when the cluster server cannot normally receive the heartbeat signal from another cluster server. Transmitting a reservation request to the quorum server, and when the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, Multiple success notifications for the reservation request And a step to be sent to the cluster server.

  In order to solve the above problems, a cluster management program according to an aspect of the present invention provides a plurality of services for providing a service to one or a plurality of clients and transmitting / receiving heartbeat signals to / from each other via a heartbeat communication path. A cluster management program used in a cluster system including a cluster server and a quorum server having a quorum, when the cluster server cannot normally receive the heartbeat signal from another cluster server. A step of transmitting a securing request for securing the quorum to the quorum server, and the securing request from another cluster server within a predetermined time after the quorum server receives the securing request from the cluster server. When you receive The success notification for the serial reservation request is a program for executing the steps of transmitting to the plurality of servers in a cluster.

  According to the present invention, it is possible to prevent an unnecessary service stop when a split brain occurs in a cluster system.

It is a figure which shows the structure of the cluster system which concerns on embodiment of this invention. It is a schematic block diagram of the quorum server which concerns on embodiment of this invention. It is a figure which shows the main information managed with the configuration information in the cluster system according to the embodiment of the present invention. It is a flowchart which shows the operation | movement procedure at the time of the quorum server concerning embodiment of this invention performing a system management process. It is a flowchart which shows the operation | movement procedure at the time of the quorum server concerning embodiment of this invention performing a system management process. It is a figure which shows the example of a change of the service management table of each server in the cluster system which concerns on embodiment of this invention. It is a figure which shows the state of the cluster system which concerns on embodiment of this invention. It is a figure which shows the state of the cluster system which concerns on embodiment of this invention. It is a figure which shows the state of the cluster system which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and description thereof will not be repeated.

[Configuration and basic operation]
FIG. 1 is a diagram showing a configuration of a cluster system according to an embodiment of the present invention.

  Referring to FIG. 1, a cluster system 301 includes servers (cluster servers) 151 and 152 that are examples of information processing apparatuses, a quorum server 201 that is an example of an information processing apparatus, a shared storage apparatus 40, and a service providing service. And a communication path 60. The server 151 includes a cluster management unit 11, a service management unit 12, and configuration information 13. The server 152 includes a cluster management unit 21, a service management unit 22, and configuration information 23.

  The cluster system 301 is a two-node cluster system including servers 151 and 152, for example. Note that the cluster system 301 may be configured to include three or more servers. Hereinafter, a group of servers that provide services in the cluster system 301 is referred to as a cluster. Each of the servers 151 and 152 corresponds to a cluster node constituting the cluster.

  The servers 151 and 152 provide services to the clients CT1 to CTn (n is an integer of 1 or more), and transmit / receive heartbeat signals to / from each other via the heartbeat communication path 50.

  More specifically, the shared storage device 40 can be accessed from the server 151 and the server 152.

  The server 151 and the server 152 are connected to each other via a heartbeat communication path 50 for life and death confirmation (heartbeat). The server 151 and the server 152 periodically transmit and receive heartbeat signals using the heartbeat communication path 50 in order to monitor each other's life and death.

  The server 151 and the server 152 are connected to a service providing communication path 60 for providing services to the clients CT1 to CTn, and the quorum server 201 is disposed on the communication path.

  The quorum server 201 and the servers 151 and 152, which are information processing apparatuses according to the embodiment of the present invention, typically have a basic structure of a computer having a general-purpose architecture and execute a program installed in advance. Thus, various functions as described later are provided. In general, such a program is stored in a recording medium such as a flexible disk and a CD-ROM (Compact Disk Read Only Memory) or distributed via a network or the like.

  The program according to the embodiment of the present invention may be provided by being incorporated in a part of another program such as an OS. Even in this case, the program itself according to the embodiment of the present invention does not include a module included in the other program as described above, and the process is executed in cooperation with the other program. That is, the program according to the embodiment of the present invention may be in a form incorporated in such another program.

  Alternatively, some or all of the functions provided by program execution may be implemented as a dedicated hardware circuit.

  FIG. 2 is a schematic configuration diagram of a quorum server according to the embodiment of the present invention. The configuration of the servers 151 and 152 is the same as that of the quorum server 201, for example.

  Referring to FIG. 2, a quorum server 201 includes a CPU (Central Processing Unit) 101 that is an arithmetic processing unit, a main memory 102, a hard disk 103, an input interface 104, a display controller 105, and a data reader / writer 106. And a communication interface 107. These units are connected to each other via a bus 121 so that data communication is possible.

  The CPU 101 performs various operations by developing programs (codes) stored in the hard disk 103 in the main memory 102 and executing them in a predetermined order. The main memory 102 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and stores data indicating various arithmetic processing results in addition to programs read from the hard disk 103. To do. The hard disk 103 is a non-volatile magnetic storage device, and stores various setting values in addition to programs executed by the CPU 101. The program installed in the hard disk 103 is distributed in a state of being stored in the recording medium 111 as will be described later. In addition to the hard disk 103 or instead of the hard disk 103, a semiconductor storage device such as a flash memory may be employed.

  The input interface 104 mediates data transmission between the CPU 101 and an input unit such as a keyboard 108, a mouse 109, and a touch panel (not shown). That is, the input interface 104 receives an external input such as an operation command given by the user operating the input unit.

  The display controller 105 is connected to a display 110 that is a typical example of a display unit, and controls display on the display 110. That is, the display controller 105 displays the result of image processing by the CPU 101 to the user. The display 110 is, for example, an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube).

  The data reader / writer 106 mediates data transmission between the CPU 101 and the recording medium 111. That is, the recording medium 111 circulates in a state where a program executed by the quorum server 201 is stored, and the data reader / writer 106 reads the program from the recording medium 111. Further, the data reader / writer 106 writes the processing result in the quorum server 201 to the recording medium 111 in response to the internal command of the CPU 101. The recording medium 111 is, for example, a general-purpose semiconductor storage device such as CF (Compact Flash) and SD (Secure Digital), a magnetic storage medium such as a flexible disk, or a CD-ROM (Compact Disk Read Only). Memory).

  The communication interface 107 mediates data transmission between the CPU 101 and other personal computers and server devices. The communication interface 107 typically has an Ethernet (registered trademark) or USB (Universal Serial Bus) communication function. Instead of installing the program stored in the recording medium 111 in the quorum server 201, a program downloaded from a distribution server or the like via the communication interface 107 may be installed in the quorum server 201.

  Further, the quorum server 201 may be connected to another output device such as a printer as necessary.

[Control structure]
Next, a control structure for providing a life and death monitoring function in the quorum server 201 will be described.

  FIG. 1 also shows a control structure provided by each server according to the embodiment of the present invention. Each block in the quorum server 201 and the servers 151 and 152 shown in FIG. 1 is provided by developing a program (code) stored in the hard disk 103 in the main memory 102 and causing the CPU 101 to execute it. Note that part or all of the control structures of the quorum server 201 and the servers 151 and 152 shown in FIG. 1 may be realized by dedicated hardware and / or a wiring circuit.

  Referring to FIG. 1, quorum server 201 includes tie breaker unit 32 as its control structure. The server 151 includes a cluster management unit 11 and a service management unit 12 as its control structure. The server 152 includes a cluster management unit 21 and a service management unit 22 as its control structure. The quorum server 201 includes a quorum storage area 31. The server 151 includes cluster configuration information 13. The server 152 includes cluster configuration information 23.

  In the quorum server 201, the quorum storage area 31 provides a lock function. In addition, the tie breaker unit 32 arbitrates quorum securing requests from the servers.

  When the tie breaker unit 32 receives a quorum securing request (hereinafter also referred to as a securing request) from the server 151 and the server 152 constituting the cluster in the cluster system 301, the quorum (lock) is acquired in the quorum storage area 31. Mark it so that other servers can recognize it.

  Further, when the quorum (lock) is not secured, the tie breaker unit 32 returns a quorum securing success notification (hereinafter also referred to as a success notification) to the server that has transmitted the securing request. On the other hand, the tie breaker unit 32 returns a quorum reservation failure notification (hereinafter also referred to as failure notification) to the server that has transmitted the reservation request when the other server has already secured the quorum and cannot secure the quorum. Normally, when a securing request arrives at the quorum server 201 almost simultaneously from a plurality of servers, the server that sent the securing request that arrived first secures the quorum, and the server that sent the securing request that arrived later secures the quorum. Fail.

  The quorum server 201 does not immediately secure a quorum even if a reservation request is received from a server as in the prior art, but waits for a reservation request from another server within an allowable time. In the quorum server 201, when there are reservation requests from a plurality of servers within the allowable time, control is performed so that the services are not started redundantly on each server, and then the reservation request arrived first is transmitted. Server gets the quorum and returns a success notification to each of the other servers.

  The servers 151 and 152 include cluster management units 11 and 21, service management units 12 and 22, and cluster configuration information 13 and 23 that are storage areas for recording various types of cluster information, as functions for managing clusters. .

  The cluster management units 11 and 21 monitor the state of the cluster and manage cluster configuration information (hereinafter also referred to as configuration information) 13 and 23, respectively. The cluster management units 11 and 21 perform life and death confirmation by transmitting and receiving heartbeat signals to and from the cluster management units 21 and 11 in other servers via the heartbeat communication path 50.

  The cluster management units 11 and 21 are responsible for managing the configuration information 13 and 23, respectively, and hold the same configuration information between servers running in the cluster. The cluster management units 11 and 21 are responsible for reconfiguring the cluster when a server in the cluster is started and joined as a member, and when the heartbeat is interrupted. Here, the cluster reconfiguration is to change the state of the members in the cluster and restart the service on an appropriate server.

  The cluster management units 11 and 21 of each server hold one quorum. Since there is one quorum on the quorum server 201, there are a total of three quorums in the cluster system 301.

  When the heartbeat signal is interrupted, the cluster management units 11 and 21 attempt to acquire the quorum of the quorum server 201.

  When the cluster management units 11 and 21 receive a success notification from the quorum server 201, the cluster management units 11 and 21 have secured a quorum of a majority (2/3) or more of the system, and thus reconfigure themselves as regular clusters. Take over the services of other servers.

  On the other hand, if the cluster management unit 11 or 21 receives a failure notification from the quorum server 201, the quorum to be held is only its own quorum (1/3) and less than the majority, Stop (panic).

  In addition, the cluster management units 11 and 21 need to obtain a majority quorum by making a reservation request to the quorum server 201 even when the server is activated.

  When the cluster management units 11 and 21 fail to secure the quorum of the quorum server 201, it is necessary to wait for the server that secures the quorum of the quorum server 201, and then the activation is permitted. At this time, synchronization (copying) of the configuration information of the server securing the quorum of the quorum server 201 is performed. When the configuration information 13 and 23 are synchronized, the cluster management units 11 and 21 release the quorum of the quorum server 201.

  The service management units 12 and 22 cooperate with the cluster management units 11 and 21, respectively, to control service start, stop, and monitoring based on the configuration information 13 and 23.

  FIG. 3 is a diagram showing main information managed by the configuration information in the cluster system according to the embodiment of the present invention.

  Referring to FIG. 3, the configuration information includes, for example, a server participating in the cluster and a cluster member table 90 that holds the status, a service component, a service status, and a server on which the service is operating. And a service management table 91 that holds

  The service components generally include an application name which is a main element, a disk area used by the application (hereinafter also referred to as LUN), and a virtual IP address (hereinafter referred to as VIP) for waiting for a request from a client. Also called). If the application does not require disk and network functions, LUNs and VIPs can be removed from the service components.

[Operation]
Next, the operation of the information processing apparatus according to the embodiment of the present invention will be described with reference to the drawings. In the embodiment of the present invention, the cluster management method according to the embodiment of the present invention is implemented by operating the quorum server 201 and the servers 151 and 152. Therefore, the description of the cluster management method according to the embodiment of the present invention is replaced with the following description of the operation of the quorum server 201 and the servers 151 and 152. In the following description, FIG. 1 is referred to as appropriate.

  As illustrated in FIG. 1, when the heartbeat on the heartbeat communication path 50 is interrupted while the service 1 is operating on the server 151 and the service 2 is operating on the server 152, the server 151 and the server 152. The cluster management units 11 and 21 in FIG. 11 detect that the heartbeat has been interrupted. Then, the cluster management units 11 and 21 issue a securing request to the quorum server 201 in order to secure a majority quorum.

  That is, if the servers 151 and 152 cannot normally receive a heartbeat signal from another server, the servers 151 and 152 transmit a securing request for securing a quorum to the quorum server 201.

  When the quorum server 201 receives a securing request from another server within a predetermined time after receiving the securing request from the servers 151 and 152, the quorum server 201 transmits a success notification to the securing request to the servers 151 and 152.

  Further, the servers 151 and 152 have configuration information including information on services provided by other servers.

  The quorum server 201 rewrites the configuration information of the servers 151 and 152 so that the servers 151 and 152 do not take over the services of other servers before sending a success notification for the securing request to the servers 151 and 152.

  Also, the quorum server 201 stores that the server that transmitted the reservation request that arrived first has truly secured the quorum.

  In addition, when the configuration information is rewritten, the quorum server 201 rewrites a service provided by another server into a dummy service in the configuration information of the server that has truly secured the quorum.

  More specifically, when the heartbeat signal between the server 151 and the server 152 is interrupted in the state where the service 1 and the service 2 are operating on the server 151 and the server 152 constituting the cluster, Attempts to secure the quorum of the server 201. The tie breaker unit 32 in the quorum server 201 considers that both servers can operate the service when the reservation requests from the servers 151 and 152 reach the quorum server 201 within the allowable time. The tie breaker unit 32 performs exclusive control so that the consistency of the cluster is not impaired, and notifies the server 151 and the server 152 that the quorum has been secured, and is operating on both servers. Continue to operate the service.

  Specifically, the tie breaker unit 32 in the quorum server 201 performs arbitration based on the flowcharts shown in FIGS. 4 and 5 below. Here, a case will be described in which a reservation request from the server 151 first arrives at the quorum server 201.

  4 and 5 are flowcharts showing an operation procedure when the quorum server according to the embodiment of the present invention performs system management processing.

  Referring to FIGS. 4 and 5, when tie breaker unit 32 receives a secure request from server 151 (step A1), tie breaker unit 32 waits for a secure request from server 152 within an allowable time (step A2).

  Next, when receiving a securing request from the server 152 within the allowable time, the tie breaker unit 32 requests the server 152 to close the heartbeat NIC (step A3). The tie breaker unit 32 takes the NIC for heartbeat offline by using a remote shell, for example. As a result, when the heartbeat communication path failure is an intermittent failure, the heartbeat is restarted again while the processing of the flowcharts shown in FIGS. 4 and 5 is being performed, causing inconsistency in consistency. Can be prevented.

  Next, after closing the heartbeat NIC in the server 152, the tie breaker unit 32 requests the cluster management unit 11 in the server 151 to change the configuration information 13 of the server 151. The content of this request is that the service 2 running on the server 152 in the service management table 91 is a dummy service, that is, an application that does not constitute LUNs and VIPs and that simply performs simple calculations periodically. It is to be replaced (step A5).

  Next, the cluster management unit 11 in the server 151 that has received the request changes the service management table 91 of the configuration information 13 (step A6).

  FIG. 6 is a diagram showing a modification example of the service management table of each server in the cluster system according to the embodiment of the present invention.

  Referring to FIG. 6, in service management table 91 in server 151, service 2 operating on server 152 is changed to a dummy service, resulting in service management table 91a.

  By changing the service management table 91 in this way, it is possible to prevent contention with resources such as LUNs and VIPs used by the service 2 operating in the server 152 when the process reaches step A10 described later. it can. Note that by changing the dummy service to be activated, it becomes possible to recognize the operation status of the dummy service or the like at the time of system operation after step A10 or post-analysis.

  Next, after changing the configuration information 13 of the server 151, the tie breaker unit 32 requests the server 152 to change the configuration information 23 (step A7). The content of this request is to delete information on a service operating on a service management table 91 other than its own server.

  Next, the cluster management unit 21 in the server 152 that has received the request changes the service management table 91 of its own configuration information 23.

  Referring to FIG. 6, in service management table 91 in server 152, service 1 that is operating other than server 152 is deleted, resulting in service management table 91b.

  By changing the service management table 91 in this way, when the process reaches a later-described step A10, it is possible to prevent contention with resources such as LUN and VIP used by the service 1 operating in the server 151. it can. Also, when the process reaches step A10, the server 152 is caused to recognize that it is a cluster that should survive, and the operation is continued. Thereafter, the configuration information of the server 152 is the official information. Not used. For this reason, the server 152 is caused to delete information on services other than itself.

  Next, after changing the configuration information 23 of the server 152, the server 151 secures (locks) the quorum of the quorum server 201 (step A9).

  Next, the tie breaker unit 32 issues a success notification to both the server 151 and the server 152. (Step A10).

  Next, the cluster management unit 11 in the server 151 receives the success notification, changes the state of the server 152 in the cluster member table 90 of the configuration information 13 to offline, and operates on the server 152 with respect to the service management unit 12. Instruct the start of the service that is running. Here, since the server 151 changes the configuration information 13 in step A5, the dummy service is actually activated instead of the service 2.

  Further, upon receiving the success notification, the cluster management unit 21 in the server 152 changes the state of the server 151 in the cluster member table 90 of the configuration information 23 to offline, and operates on the server 151 with respect to the service management unit 22. Instruct the activation of the service. Here, since the information of the service operating on the server 151 in the service management table 91 is deleted in step A7, nothing is actually activated.

  FIG. 7 is a diagram showing a state of the cluster system according to the embodiment of the present invention.

  When the process reaches step A10, the cluster system 301 is in a state as shown in FIG. In the processing up to step A10, since the services 1 and 2 that are in operation are not stopped and restarted, the service 1 on the server 151 and the service 2 on the server 152 remain in operation. For this reason, the state in which each client can provide a service is continued.

  After the processing of step A 10 is executed, the tie breaker unit 32 in the quorum server 201 performs alive monitoring of the cluster management unit 21 in the server 152.

  When the server 152 stops the server, the tie breaker unit 32 detects that the server 152 does not respond, and causes the cluster management unit 11 in the server 151 to temporarily set the dummy service offline. Then, the tie breaker unit 32 causes the cluster management unit 11 to replace the service management table 91 before the change in step A5, that is, change the dummy service to the service 2, and then change the service 2 in the server 151. Start.

  In addition, if it is necessary to reconfigure the cluster after the execution of step A10 due to the stop and start of the server, the configuration information 13 of the server 151 that has transmitted the securing request first as the regular cluster configuration information. Is adopted.

  In this case, in step A9, the quorum storage area 31 of the quorum server 201 is locked by the server 151, and the server 151 can obtain a majority quorum. Therefore, the server 151 can be restarted independently. On the other hand, the server 152 cannot secure a majority quorum and cannot be restarted alone.

  Also, the server 151 needs to be activated when the server 152 is restarted, and the configuration information 23 of the server 152 is synchronized (matched) with the formal configuration information 13 of the server 151 when restarted. Only after this synchronization is taken can the server 152 be started as a cluster. When the configuration information is matched between both servers, the quorum lock secured by the server 151 is deleted from the quorum storage area 31 on the quorum server 201.

  Next, in the flowcharts shown in FIGS. 4 and 5, processing when processing is not performed normally will be described.

  If the tie breaker unit 32 cannot reach step A10, the tie breaker unit 32 determines that one of the servers 151 and 152 has become inoperable, such as a hang and a slowdown, rather than a failure of the heartbeat communication path 50, The following operations are performed so that the service can be taken over by an appropriate server.

  If the securing request from the server 152 does not arrive within the allowable time (NO in step A2), the tie breaker unit 32 considers the server 152 to be inoperable. Then, the tie breaker unit 32 issues a success notification to the server 151 (step A22) after securing the quorum by the server 151 (step A21).

  Upon receiving the success notification, the cluster management unit 11 in the server 151 changes the state of the server 152 in the cluster member table 90 of the configuration information 13 to offline, and tries to start the service 2 operating on the server 152.

  When the tie breaker unit 32 fails to close the heartbeat NIC in the server 152 (NO in step A4), the tie breaker unit 32 regards the server 152 as inoperable. The tie breaker unit 32 then issues a success notification to the server 151 and issues a failure notification to the server 152 after securing the quorum by the server 151 (step A41) (step A42).

  Upon receiving the success notification, the cluster management unit 11 in the server 151 changes the state of the server 152 in the cluster member table 90 of the configuration information 13 to offline, and tries to start the service 2 operating on the server 152.

  On the other hand, upon receipt of the failure notification, the cluster management unit 21 in the server 152 tries to stop its operation (panic) because the quorum is less than a majority.

  In addition, when the configuration information change of the server 151 fails (NO in step A6), the tie breaker unit 32 regards the server 151 as inoperable. The tie breaker unit 32 then releases the heartbeat NIC blockage in the server 152 blocked in step A3 in order to allow the server 152 to take over service 1 (step A61).

  Then, the tie breaker unit 32 issues a quorum reservation success notification to the server 152 after issuing a quorum reservation by the server 152 (step A62), and issues a reservation failure notification to the server 151 (step A63).

  Upon receiving the success notification, the cluster management unit 21 in the server 152 changes the state of the server 151 in the cluster member table 90 of the configuration information 23 to offline, and the service operating on the server 151 with respect to the service management unit 22 Request activation of 1.

  On the other hand, the cluster management unit 11 in the server 151 receives the failure notification and panics itself because the quorum is less than a majority.

  In addition, when the configuration information change of the server 152 fails (NO in step A8), the tie breaker unit 32 considers the server 152 to be inoperable. Then, the tie breaker unit 32 restores the configuration information 13 of the server 151 changed in step A5 so that the service 2 can be taken over by the server 151 (step A81).

  The tie breaker unit 32 then issues a success notification to the server 151 and issues a failure notification to the server 152 after securing the quorum by the server 151 (step A41) (step A42).

  Upon receiving the success notification, the cluster management unit 11 in the server 151 changes the state of the server 152 in the cluster member table 90 of the configuration information 13 to offline, and tries to start the service 2 operating on the server 152.

  On the other hand, the cluster management unit 21 in the server 152 receives a failure notification and tries to panic itself because the quorum is less than a majority.

  Next, another operation example of the cluster system according to the embodiment of the present invention will be described. That is, in the cluster system 301, both the service 1 and the service 2 are activated on the server 151, and the operation when the heartbeat is interrupted in a state where there is no service activated on the server 152, two operations are performed. The pattern is described below.

[Pattern 1]
If there is a reservation request from the server 151 to the quorum server 201 first and the server 152 receives a reservation request within the allowable time, the tie breaker unit 32 follows the flowcharts shown in FIGS. To work.

  The tie breaker unit 32 issues a request for closing the heartbeat NIC to the server 152 (step A3).

  Next, the tie breaker unit 32 requests the cluster management unit 11 in the server 151 to change the configuration information 13 of the server 151 (step A5).

  In response to this request, the cluster management unit 11 changes the service running on the server 152 to a dummy service in the service management table 91 of the configuration information 13. However, since there is no service running on the server 152, no change occurs in the service management table 91 of the server 151.

  Next, the cluster management unit 11 requests the cluster management unit 21 in the server 152 to change the configuration information 23 (step A7).

  In response to this request, the cluster management unit 21 in the server 152 deletes the service operating on the server other than its own server from the configuration information. Since there is no service running on the server 152, the service management table 91 on the server 152 is empty.

  Next, after the server 151 secures the quorum of the quorum server 201 (step A9), the tie breaker unit 32 issues a success notification to both the server 151 and the server 152. (Step A10).

  Upon receiving the success notification, the cluster management unit 11 in the server 151 changes the state of the server 152 in the cluster member table 90 of the configuration information 13 to offline, and the service operating on the server 152 with respect to the service management unit 12 Try to start. However, since no service is operating on the server 152, no service is activated.

  Further, upon receiving the success notification, the cluster management unit 21 in the server 152 changes the state of the server 151 in the cluster member table 90 of the configuration information 23 to offline, and operates on the server 151 with respect to the service management unit 22. Try to start the service. However, since the service operating on the server 151 is deleted from the service management table 91 in step A7, no service is activated.

  FIG. 8 is a diagram showing a state of the cluster system according to the embodiment of the present invention.

  When the process reaches step A10, the cluster system 301 is in a state as shown in FIG. In the processing up to step A10, since the service 1 and 2 that are in operation are not stopped and restarted, the service 1 on the server 151 and the service 2 on the server 152 remain in operation. For this reason, the state in which each client can provide a service is continued.

[Pattern 2]
When the quorum server 201 has a reservation request from the server 152 first and the server 151 also has a reservation request within the allowable time, the tie breaker unit 32 follows the flowcharts shown in FIGS. To work.

  The tie breaker unit 32 issues a request for closing the heartbeat NIC to the server 151 (step A3).

  Next, the tie breaker unit 32 requests the cluster management unit 21 in the server 152 to change the configuration information 23 of the server 152 (step A5).

  Upon receiving this request, the cluster management unit 21 changes the service 1 and service 2 operating on the server 151 to dummy services in the service management table 91 of the configuration information 23.

  Next, the tie breaker unit 32 requests the cluster management unit 11 in the server 151 to change the configuration information 13 (step A7). The cluster management unit 11 deletes the service running on the server other than its own server from the configuration information 13. However, since there is no service operating on the server 152, no change occurs in the service management table 91 in the server 151.

  Next, the tie breaker unit 32 issues a success notification to both the server 151 and the server 152 after the server 152 secures the quorum of the quorum server 201 (step A9). (Step A10).

  Upon receiving the success notification, the cluster management unit 11 in the server 151 changes the state of the server 152 in the cluster member table 90 of the configuration information 13 to offline, and the service operating on the server 152 with respect to the service management unit 12 Request activation of. However, since no service is operating on the server 152, no service is activated.

  Further, upon receiving the success notification, the cluster management unit 21 in the server 152 changes the state of the server 151 in the cluster member table 90 of the configuration information 23 to offline, and operates on the server 151 with respect to the service management unit 22. Request activation of the service.

  Here, since the information of the service 1 and the service 2 operating on the server 151 in the configuration information 23 is changed in the step A5, the service 1 and the service 2 are not actually activated and the dummy service is activated. The

  FIG. 9 is a diagram showing a state of the cluster system according to the embodiment of the present invention.

  When the process reaches step A10, the cluster system 301 is in a state as shown in FIG. In the processing up to step A10, since the service 1 and 2 that are in operation are not stopped and restarted, the service 1 on the server 151 and the service 2 on the server 152 remain in operation. For this reason, the state in which each client can provide a service is continued.

  By the way, in the technologies described in Patent Document 1 and Patent Document 2, even if the service can operate on a normal node, the service may be failed over to a server with higher priority or more reliability. Service outages and restarts can occur.

  On the other hand, in the cluster system according to the embodiment of the present invention, if the servers 151 and 152 cannot normally receive the heartbeat signal from another server, a request for securing a quorum is issued to the quorum server. To 201. When the quorum server 201 receives a securing request from another server within a predetermined time after receiving the securing request from the servers 151 and 152, the quorum server 201 transmits a success notification for the securing request to the servers 151 and 152.

  In other words, the tie breaker unit 32 determines that a heartbeat communication path failure has occurred when a quorum securing request is made from a plurality of servers within the allowable time when split brain occurs. The tie breaker unit 32 then notifies the plurality of servers of the quorum reservation success, and continuously operates the plurality of servers and the plurality of services in the split brain state.

  With such a configuration, it is possible to prevent an unnecessary service stop when a split brain occurs in the cluster system. That is, when heartbeat communication between servers is interrupted in the cluster system, the service can be continuously operated without causing unnecessary failover of the service.

  That is, among the components in the cluster system according to the embodiment of the present invention, the minimum configuration including the quorum server 201 and the servers 151 and 152 prevents unnecessary service stop when a split brain occurs in the cluster system. The object of the present invention can be achieved.

  In the cluster system according to the embodiment of the present invention, the servers 151 and 152 have information on services provided by other servers. Then, the quorum server 201 rewrites the information of the servers 151 and 152 so that the servers 151 and 152 do not take over the services of other servers before transmitting the success notification for the securing request to the servers 151 and 152.

  That is, if a quorum securing request is made without delay from each server when split brain occurs, it is determined that service can be continued on each server. Then, after changing the configuration information so that services running on other servers are not started, each server can be operated in the split-brain state by notifying each server of a successful quorum acquisition. Let

  As described above, before the cluster is reconfigured by the server, the cluster configuration information is changed between the servers, thereby preventing resource duplication and data corruption when operating the service in the split brain state. Can do.

  Further, in the cluster system according to the embodiment of the present invention, the quorum server 201 stores that the server that transmitted the reservation request that arrived first has truly secured the quorum.

  As described above, by ensuring the quorum lock on the quorum server in the server holding the regular cluster information, it is possible to determine the server holding the regular cluster information when operating in the split brain state.

  Further, in the cluster system according to the embodiment of the present invention, when rewriting information, the quorum server 201 rewrites a service provided by another server to a dummy service in the information held by the server that truly secures the quorum.

  In this way, when changing the cluster configuration information when a split brain occurs, the configuration information is changed so that the dummy service is started on the server that holds the regular cluster information. The operating status of the dummy service can be recognized from the log or the like.

  The above embodiment should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[Appendix 1]
A plurality of cluster servers for providing services to one or a plurality of clients and for transmitting and receiving heartbeat signals to each other via a heartbeat channel;
A cluster system comprising a quorum server having a quorum,
When the cluster server cannot normally receive the heartbeat signal from the other cluster server, the cluster server sends a securing request for securing the quorum to the quorum server,
When the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, the quorum server transmits a success notification for the reservation request to the plurality of cluster servers. , Cluster system.

[Appendix 2]
The cluster server has information on services provided by other cluster servers,
The quorum server sends the information of the plurality of cluster servers so that the plurality of cluster servers do not take over the services of the other cluster servers before sending a success notification for the reservation request to the plurality of cluster servers. The cluster system according to appendix 1, which is rewritten.

[Appendix 3]
The cluster system according to appendix 1 or 2, wherein the quorum server stores that the cluster server that has transmitted the reservation request that has arrived first has truly reserved the quorum.

[Appendix 4]
The cluster system according to appendix 3, wherein, when rewriting the information, the quorum server rewrites a service provided by the other cluster server to a dummy service in the information of the cluster server that truly secures the quorum. .

[Appendix 5]
A plurality of cluster servers for providing services to one or a plurality of clients and for transmitting and receiving heartbeat signals to each other via a heartbeat channel;
A cluster management method in a cluster system comprising a quorum server having a quorum,
If the cluster server cannot normally receive the heartbeat signal from the other cluster server, sending a securing request to secure the quorum to the quorum server;
When the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, it transmits a success notification for the reservation request to the plurality of cluster servers. And a cluster management method.

[Appendix 6]
The cluster server has information on services provided by other cluster servers,
The cluster management method further includes:
Before the quorum server sends a success notification for the reservation request to the plurality of cluster servers, the information on the plurality of cluster servers is set so that the plurality of cluster servers do not take over the services of the other cluster servers. The cluster management method according to appendix 5, including a step of rewriting.

[Appendix 7]
The cluster management method further includes:
The cluster management method according to appendix 5 or 6, including a step of storing that the cluster server that has transmitted the reservation request that has arrived first has really reserved the quorum.

[Appendix 8]
In the step of rewriting the information of the plurality of cluster servers, the quorum server rewrites a service provided by the other cluster server to a dummy service in the information of the cluster server that has truly secured the quorum. The cluster management method according to appendix 7.

[Appendix 9]
A plurality of cluster servers for providing services to one or a plurality of clients and for transmitting and receiving heartbeat signals to each other via a heartbeat channel;
A cluster management program used in a cluster system comprising a quorum server having a quorum, the computer comprising:
If the cluster server cannot normally receive the heartbeat signal from the other cluster server, sending a securing request to secure the quorum to the quorum server;
When the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, it transmits a success notification for the reservation request to the plurality of cluster servers. A cluster management program for executing steps.

[Appendix 10]
The cluster server has information on services provided by other cluster servers,
The cluster management program further includes:
Before the quorum server sends a success notification for the reservation request to the plurality of cluster servers, the information on the plurality of cluster servers is set so that the plurality of cluster servers do not take over the services of the other cluster servers. The cluster management program according to appendix 9, wherein the rewriting step is executed.

[Appendix 11]
The cluster management program further includes:
11. The cluster management program according to appendix 9 or 10, wherein the quorum server executes a step of storing that the cluster server that has transmitted the reservation request that has arrived first secures the quorum.

[Appendix 12]
In the step of rewriting the information of the plurality of cluster servers, the quorum server rewrites a service provided by the other cluster server to a dummy service in the information of the cluster server that has truly secured the quorum. The cluster management program according to attachment 11.

  According to the present invention, in a cluster system having a plurality of servers as components, it is possible to prevent an unnecessary service stop when each server is disconnected due to a network failure or the like. Therefore, the present invention has industrial applicability.

DESCRIPTION OF SYMBOLS 11 Cluster management part 12 Service management part 13 Configuration information 21 Cluster management part 22 Service management part 23 Configuration information 40 Shared storage apparatus 50 Heartbeat communication path 60 Service provision communication path 90 Cluster member table 91 Service management table 101 CPU
102 Main Memory 103 Hard Disk 104 Input Interface 105 Display Controller 106 Data Reader / Writer 107 Communication Interface 121 Bus 151, 152 Server (Cluster Server)
201 quorum server 301 cluster system

Claims (6)

A plurality of cluster servers for providing services to one or a plurality of clients and for transmitting and receiving heartbeat signals to each other via a heartbeat channel;
A cluster system comprising a quorum server having a quorum,
When the cluster server cannot normally receive the heartbeat signal from the other cluster server, the cluster server sends a securing request for securing the quorum to the quorum server,
When the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, the quorum server transmits a success notification for the reservation request to the plurality of cluster servers. , Cluster system. The cluster server has information on services provided by other cluster servers,
The quorum server sends the information of the plurality of cluster servers so that the plurality of cluster servers do not take over the services of the other cluster servers before sending a success notification for the reservation request to the plurality of cluster servers. The cluster system according to claim 1, wherein the cluster system is rewritten.   The cluster system according to claim 1, wherein the quorum server stores that the cluster server that has transmitted the reservation request that has arrived first has truly reserved the quorum.   4. The cluster according to claim 3, wherein, when rewriting the information, the quorum server rewrites a service provided by another cluster server to a dummy service in the information of the cluster server that truly secures the quorum. system. A plurality of cluster servers for providing services to one or a plurality of clients and for transmitting and receiving heartbeat signals to each other via a heartbeat channel;
A cluster management method in a cluster system comprising a quorum server having a quorum,
If the cluster server cannot normally receive the heartbeat signal from the other cluster server, sending a securing request to secure the quorum to the quorum server;
When the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, it transmits a success notification for the reservation request to the plurality of cluster servers. And a cluster management method. A plurality of cluster servers for providing services to one or a plurality of clients and for transmitting and receiving heartbeat signals to each other via a heartbeat channel;
A cluster management program used in a cluster system comprising a quorum server having a quorum, the computer comprising:
If the cluster server cannot normally receive the heartbeat signal from the other cluster server, sending a securing request to secure the quorum to the quorum server;
When the quorum server receives the reservation request from another cluster server within a predetermined time after receiving the reservation request from the cluster server, it transmits a success notification for the reservation request to the plurality of cluster servers. A cluster management program for executing steps.

Images