JP2012014674A - Failure recovery method, server, and program in virtual environment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual environment in which a failure category can be determined and operation is allowed without causing degradation of the service-providing processing speed due to system switching at the time of failure.SOLUTION: A cluster system of the present invention, which comprises an active machine and a standby machine in a virtual environment, determines a category of a failure location by using a log and, if the failure location is a resource, causes a transition of the cluster status of the active machine to "a status allowing transition to an in-service status." Furthermore, if the determined failure category represents a defective continuity from a guest machine to a host machine, or a network trouble, the system performs a continuity check.

Description

  The present invention relates to a failure recovery method and system in a virtual environment, and in particular, in a redundant system having a virtual function in which a virtual machine monitor (for example, Xen or KVM) that implements a virtualization technology is introduced. The present invention relates to a failure recovery method, a server, and a failure recovery program in a virtual environment for identifying and recovering a cause that a machine is not incorporated in a system.

  As the importance of services increases, so does the demand for systems with low downtime. For this reason, high availability cluster software such as Heartbeat and Pacemaker that enables continuous service by building a redundant cluster system with multiple servers and automatically switching the server when some failure occurs. It has been developed (see Non-Patent Document 1).

  High-availability cluster software monitors resources, networks, shared disks, etc. on the server. If a failure is detected on a server that is in service, the server is switched to a standby server in advance and the service is continued.

  FIG. 1 shows a schematic diagram of a cluster system using high-availability cluster software. The cluster system includes a plurality of servers (active machine and spare machine) connected to a network and a shared disk that is shared and used by the plurality of servers.

  The active machine and the spare machine each have an operating system (OS), high-availability cluster software, and resources that are components necessary for providing a service. The high-availability cluster software detects the occurrence of a failure in the active machine and automatically switches to a spare machine when a failure occurs. The service operating status, resource operating status and failure status in the server are stored in the status storage unit of the internal disk, and detailed information such as the failure location is stored in the log storage unit of the internal disk.

  The active machine and the spare machine are connected to a network called a service LAN and provide a service based on resources to the client. In addition, the current machine and the spare machine are connected to a network called an interconnect LAN, and exchange information such as service operating status, resource operating status, and fault status in the server. Furthermore, the current machine and the spare machine are connected to a network called a management LAN, and can receive commands from the maintenance terminal.

  The active machine and the spare machine can be set with a forced power-off function for forcibly turning off the power of other servers when a failure occurs. The forced power cut-off function cuts off the power of the other server by transmitting an instruction to turn off the power to the hardware control board of the other server via the management LAN.

  The shared disk is a storage device that stores data used for service provision in order to maintain service consistency. With the shared disk, the service can be continued using the same data even after switching from the current machine to the spare machine.

  As described above, since the failure of the resource is monitored by the high availability cluster software, when the resource failure occurs, the service can be continued by switching to the spare machine. After the system is switched to the spare machine, the service is continued on the spare machine (see, for example, Patent Document 1).

Japanese Patent No. 4353005, “System switching method of cluster configuration computer system”

Mitsunori Mitsunori et al., “Development of OSS Middle Heartbeat to Improve Service Availability”, NTT Technical Journal, March 2009, pages 46-49

  The above technology is a process of switching the system to the standby machine when a failure occurs in the active machine, but the virtual environment is not considered, and when a failure occurs, a failure such as a resource failure or network failure occurs. Since the cause cannot be identified, even if the active machine is simply activated depending on the failure location of the active machine, it may not operate normally.

  Furthermore, when the performance of the spare machine is inferior to that of the active machine in the cluster system, the processing speed decreases while the service is provided by the spare machine. Such a decrease in processing speed is not only when the performance of the spare machine is inferior to that of the active machine, but also when a cluster system is composed of two or more active machines and one spare machine. It can happen.

  The present invention has been made in view of the above points, such as Xen et al. Shown in the document "" Open Source Virtualization Technology "NTT Technology Journal 2009 vol. 21. No.8. 2009.8, pp.82-86. Provide a failure recovery method and system in a virtual environment that can determine the category of a failure in a virtual environment based on an open source virtualization product and that can be operated without degrading the performance of the service provision processing speed due to system switching in the event of a failure The purpose is to do.

In order to solve the above-described problems of the present invention, a failure recovery method in a virtual environment according to the present invention (Claim 1) is provided in a cluster system including a working machine and a spare machine in a virtual environment in which a hypervisor is introduced. The active machine and the spare machine have cluster status management means for managing the cluster status indicating the operating status of the active machine and the spare machine service, status management information storage means for storing the cluster status and fault status, and a fault log indicating the fault location A fault recovery method in a virtual environment for identifying the cause of a failure when the spare machine is in service and the active machine is not incorporated in a cluster configuration. ,
A failure location estimating step of estimating a failure location of the current machine based on a result of the failure estimation unit of the current machine being retrieved from the log storage unit of the current machine;
In the failure location estimation step, when it is estimated that the failure location of the active device is a network failure, the continuity confirmation step of confirming the continuity to the router connected to the active device is performed by the continuity confirmation means of the active device. When,
When the continuity confirmation to the router is successful, the cluster configuration activation means of the active device performs a cluster configuration activation step of transitioning the cluster state of the active device to “a state in which the service device can be transited”.

Further, the present invention (Claim 2) is provided in an environment in which a host machine and a guest machine are installed in an active machine.
If the communication check step identifies a communication failure between the host machine and guest machine,
In the continuity confirmation step,
Check the continuity from the guest machine to the host machine,
In the cluster configuration startup step,
When the connection from the guest machine to the host machine is successful, the cluster state of the active machine is changed to “a state in which the service can be changed to operation”.

Further, according to the present invention (Claim 3), when the failure location category is estimated to be a resource in the failure location estimation step,
In the cluster configuration start step, the cluster state of the active machine is changed to “a state in which the service can be changed into operation”.

Further, according to the present invention (Claim 4), in the failure location estimation step, a forced power-off function for forcibly turning off the power of another server in the event of a failure is introduced into the active machine and the spare machine. When an error relating to the forced power-off function is retrieved from the log storage means, it is determined that a serious error has been detected in the working machine,
In the continuity confirmation step,
Check the continuity from the spare machine to the power control means of the current machine,
In the cluster configuration startup step,
If the continuity to the power supply control unit is successful in the continuity confirmation step, the cluster state of the active machine is changed to “a state where the service can be changed to operation”.

The server (working machine) of the present invention (Claim 5) is a cluster system composed of a working machine and a spare machine in a virtual environment in which a hypervisor is introduced, wherein the working machine and the spare machine are the working machine and the working machine. A cluster status management means for managing the cluster status indicating the spare machine service operating status, a status management information storage means for storing the cluster status and the fault status, and a log storage means for storing a fault log indicating the fault location. A server that operates as a working machine for identifying the cause of a failure when the machine is in service and the working machine is not incorporated in the cluster configuration,
Based on the result retrieved from the log storage means, failure location estimation means for estimating the failure location of the working machine,
In the failure location estimation means, when it is estimated that the failure location of the working machine is a network failure, continuity confirmation means for confirming continuity to the router connected to the working machine,
Cluster configuration activation means for transitioning the cluster state of the active machine to “a state where the service can be transitioned to” when the continuity confirmation to the router is successful.

In addition, the server of the present invention (Claim 6) specifies that the communication failure between the host machine and the guest machine is a continuity confirmation means in an environment where the host machine and the guest machine are installed in the server. A means for confirming continuity from the guest machine to the host machine,
The cluster configuration activation means includes means for transitioning the cluster state of the active machine to “a state where the service can be activated” when the continuity confirmation means succeeds from the guest machine to the host machine.

  The server cluster configuration activation means of the present invention (Claim 7), when the failure location is a resource in the failure location estimation means, the cluster status of the server is “a state in which the server can transition to service operation”. It further has a means to make a transition to.

In addition, the server of the present invention (Claim 8) is an environment where a forced power-off function for forcibly turning off the power of other servers in the event of a failure is installed in the active machine and the spare machine.
An error related to the forced power-off function is detected in the cluster configuration activation means in the spare machine, and the continuity confirmation means of the spare machine confirms the continuity from the spare machine to the power control means of the active machine, and the continuity failure If it is found, a means for preventing the cluster state of the server from changing to “a state in which the service can be changed” is included.

  The present invention (Claim 9) is a program for causing a computer to function as each means constituting the server according to any one of Claims 5 to 8.

  As described above, according to the present invention, in the virtual environment, it is easy to cope with recovery by estimating the cause of failure in which the active machine is not incorporated in the cluster configuration.

  In addition, when the estimated failure cause category is a resource failure, it is possible to restart the service on the active machine if the service is active on the spare machine. Can be improved.

1 is a schematic diagram of a cluster system using high availability cluster software. FIG. 1 is a system configuration diagram according to an embodiment of the present invention. It is a detailed block diagram of the principal part in one embodiment of this invention. It is a figure which shows the storage item of the state management information storage part in one embodiment of this invention. It is a figure which shows the initial state and completion | finish state in one embodiment of this invention. It is a figure which shows the outline | summary of the operation | movement in one embodiment of this invention. It is a flowchart (the 1) of a series of operation | movement in one Example of this invention. It is a flowchart (the 2) of a series of operation | movement in one Example of this invention. It is a figure which shows the operation | movement of step 101 of FIG. 7 in one Example of this invention. It is a figure which shows operation | movement of step 101,104,107 of FIG. 7 in one Example of this invention. It is a figure which shows the operation | movement of step 103 of FIG. 7 in one Example of this invention. It is a figure which shows operation | movement of the step 105 of FIG. 7 in one Example of this invention. It is a figure which shows operation | movement of step 108 of FIG. 7 in one Example of this invention. It is a figure which shows the operation | movement of step 109 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 110 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 111 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 112 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 113 of FIG. 8 in one Example of this invention. It is a figure which shows the operation | movement of step 114 of FIG. 8 in one Example of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, terms used in this specification will be defined.

・ Cluster configuration:
This is a technology for connecting a plurality of servers to each other and making the user and other servers behave as if they are one server as a whole, and the plurality of servers can be managed so as to handle one server. Even if one unit stops, the entire system does not stop, and repair and replacement are possible while processing continues.

·resource:
Refers to the components required to provide a service. A resource in a cluster configuration refers to an application to be controlled by the highly available cluster software such as starting, stopping, and monitoring. The application includes a database (DB) and the like.

・ ACT:
The service is running on the server. In a cluster configuration, the state of a server on which a resource providing a service such as a DB is operating is denoted as “ACT”.

・ SBY [online]:
This refers to the state that can transition to "ACT". A server that can switch resources from "ACT" when a system switchover occurs due to a failure in the cluster configuration is described as "SBY [online]".

・ SBY [standby]:
In a cluster configuration, even if a system switchover occurs due to a failure or the like, a server that is suppressed so as not to become “ACT” is described as “SBY [standby]”.

・ SBY [Transition]:
Indicates the state that is going to transition to "ACT" (the state during system switchover). In a cluster configuration, a system switchover occurs due to a failure, etc., and the server that is about to transition to "ACT" is waiting for normal termination of the resource stop processing being performed on the active machine. [Transition]].

・ NONE:
A state in which the server is not incorporated in the cluster configuration. A server that is not included in the cluster configuration due to the server or high availability cluster software stoppage being stopped is described as "NONE".

  FIG. 2 shows a system configuration in an embodiment of the present invention.

  The cluster system shown in the figure includes a plurality of servers having a host machine that is a base of a virtual server in a virtual environment, a guest machine, and a hypervisor that is a control program for controlling the virtual machine (the active machine 10 in this embodiment). , Written as spare machine 20). These servers are connected to the shared disk 30, the client device 1, the service LAN 2, the router 3, the maintenance terminal 4, and the management LAN 5.

  The active machine 10 and the spare machine 20 provide services to the client device 1 via the router 3. The performance of the current machine 10 may be superior to the performance of the spare machine 20. Although one active machine 10 and one spare machine 20 are shown in the figure, two or more current machines and one spare machine 20 may be used.

  The active machine 10 includes a host machine allocation disk 161 allocated for a host machine, a built-in disk 160 including a guest machine allocation disk 162 allocated for a guest machine, a hardware control board 17, and a component between these components. It has an interface for performing communication. The guest machine allocation disk 162 of the internal disk 160 includes a log storage unit 163 and a state management information storage unit 164.

  The host machine 150 of the active machine 10 has a device batch monitoring unit 151 having a disk monitoring function and a network monitoring function. Between the host machine 150 and the hypervisor 170, a management LAN network virtual interface (I / F) 13 and a service LAN network virtual I / F 14.

  The disk monitoring function of the device batch monitoring unit 151 of the host machine 150 of the active machine 10 performs disk management of the internal disk 160 and the shared disk 30, and communicates the conduction results between the host machine 150 and the guest machine 110, Notifying the high availability cluster software 140 of the guest machine via the management LAN network virtual I / F 13 of the host machine, the hypervisor 170, and the management LAN network virtual I / F 15 of the guest machine, and storing them in the log storage unit 163. If an error message output is output from the disk monitoring function, there may be a service LAN communication failure between the active machine 10 and the router 3 or a path failure between the host machine and the guest machine.

  The network monitoring function of the device batch monitoring unit 151 of the host machine 150 of the active machine 10 confirms continuity with the router 3 via the service LAN network virtual I / F 14, the hypervisor 170, and the service LAN network I / F 11. The continuity result is communicated between the host machine and the guest machine, and the guest machine 110 is connected via the service LAN network virtual I / F 16 of the host machine 150, the hypervisor 170, and the service LAN network virtual I / F 16 of the guest machine 110. Is notified to the high availability cluster software 140 and stored in the log storage unit 163. When an error message output is output from the network monitoring function, a service LAN communication failure between the active machine 10 and the router 3 or a path failure between the host machine 150 and the guest machine 110 is considered.

  The guest machine 110 of the active machine 10 has a resource 120, a control execution unit 130, and high availability cluster software 140 for notifying the outside of its own operating state. Between the guest machine 110 and the hypervisor 170, It has a management LAN network virtual I / F 15 and a service LAN network virtual I / F 16. The configuration of the guest machine will be described later with reference to FIG.

  Similarly to the active machine 10, the spare machine 20 is installed with a host machine 150, a guest machine 110, and a hypervisor 270. Further, it has a built-in disk 240 including a host machine allocation disk 261 and a guest machine allocation disk 262 and a hardware control board 27. The guest machine allocation disk 262 includes a log storage unit 263 and a state management information storage unit 264.

  The host machine 250 of the spare machine 20 has a device batch monitoring unit 251 having a disk monitoring function and a network monitoring function, and a management LAN network virtual interface (I / F) between the host machine 250 and the hypervisor 270. 23 and a service LAN network virtual I / F 24.

  The guest machine 210 of the spare machine 20 has a resource 220, a control execution unit 230, and high-availability cluster software 240. Between the guest machine 210 and the hypervisor 270, a management LAN network virtual I / F 25 and a service LAN network It has a virtual I / F 26.

  In the following, there is shown a configuration centered on the guest machine of the active machine 10 that has a possibility of failure and the status is “NONE” and is not incorporated in the cluster configuration.

  FIG. 3 shows a detailed configuration of a main part in one embodiment of the present invention.

  The guest machine 110 of the active machine 10 has a resource 120, a control execution unit 130, and high availability cluster software 140, and a state management information storage unit 164 of an internal disk 160 is connected to the high availability cluster software 140. Although only the active machine 10 is shown here, the spare machine 20 has the same configuration.

  The control execution unit 130 includes a log search unit 131, a continuity confirmation unit 132, a system switching unit 133, a state confirmation execution unit 134, a start execution unit 135, a command execution unit 136, and a failure location estimation unit 137. Is connected to the log storage unit 163 of the internal disk 160.

  When logged in from the maintenance terminal 4, the log search unit 131 searches the log storage unit 163 under the control of the failure location estimation unit 137, acquires an error message, and passes it to the failure location estimation unit 137.

  The continuity confirmation unit 132 executes the Ping process under the control of the failure location estimation unit 137, confirms the continuity to the router 3, and passes it to the failure location estimation unit 137.

  The system switching unit 133 changes the cluster state of the active machine 10 and the spare machine 20.

  The state confirmation execution unit 134 confirms the cluster state acquired by executing the state command of the high availability cluster software.

  The activation execution unit 135 activates the high availability cluster software 140.

  The command execution unit 136 causes the high availability cluster software 140 to execute a command.

  The failure location estimation unit 137 estimates the failure location based on the error message acquired from the log search unit 131 and the continuity confirmation unit 132.

  The high-availability cluster software 140 operates as a “cluster state management unit” and monitors a failure state of a server, a resource start / stop function for starting and stopping a resource based on the cluster state and the failure state, And a state management function for managing the cluster state of the state management information storage unit 164 based on the failure state. As a failure monitoring function, it has a network monitoring result acquisition unit 141 and a disk monitoring result acquisition unit 142, is activated by the activation execution unit 135 of the control execution unit 130, and executes a command under the control of the command execution unit 136. The contents of the state management information storage unit 164 are updated, and the cluster state and the like are notified to the other server (spare machine 20). The network monitoring result acquisition unit 141 obtains the continuity result of the router 3 from the network monitoring function of the device batch monitoring unit 151 of the guest machine via the service LAN network virtual I / F 14, the hypervisor 170, and the service LAN network virtual I / F 16. get. The disk monitoring result acquisition unit 142 monitors the disk from the device batch monitoring unit 151 of the host machine via the management LAN network virtual I / F 13 of the host machine, the hypervisor 170, and the management LAN network virtual I / F 15 of the guest machine 110. Get the result.

  The log storage unit 163 stores failure logs such as disks and networks acquired from the device batch monitoring unit 151 of the host machine. Similarly, a failure log acquired from the device batch monitoring unit 251 of the host machine is also stored in the log storage unit 263 of the spare machine 20.

  As shown in FIG. 4, the state management information storage unit 164 stores flags and values of the cluster state, the number of failures, an error status, and a resource state. These values are stored by the state management function of the high availability cluster software 140. Managed.

  The outline of the operation from the series of failure detection to recovery in the above configuration will be described below.

  First, FIG. 5 shows an initial state and an end state.

  In the initial state, the status check process is executed in the spare machine 20, the failure count of the status management information storage unit 264 is "0", the error status is "0", the cluster status of the spare machine 20 is "ACT", and the active machine When the cluster status of 10 is “NONE”, the power switch of the active machine 10 is turned on and the OS is started up.

  Also, the failure state is a failure estimation process. As a result, when the failure category is “resource” failure, the resource recovery processing is performed, the cluster state of the active device 10 is set to “ACT”, and the standby device. The cluster state of 20 is “SBY [online]”.

  Hereinafter, a series of processes in FIG. 5 will be described.

  FIG. 6 is a sequence chart showing an outline of the operation according to the embodiment of the present invention.

  Step 1) In the “NONE” state where the active machine 10 is not incorporated in the cluster configuration, the OS of the active machine 10 is started up. At this time, the spare machine 20 is in the “ACT” state during service operation.

  The active machine 10 and the spare machine 20 accept login from the maintenance terminal 4 via the management LAN 5, and the log search unit 131 of the control execution unit 130 searches the log storage unit 163 to obtain an error message. Here, it is assumed that an error message indicating a resource error has been acquired.

  Step 2) Furthermore, the control execution unit 130 requests the maintenance terminal 4 to confirm the continuity of the forced power-off function by outputting an error message. As a result, the maintenance terminal 4 logs in to the spare machine 20, and conducts continuity confirmation from the continuity confirmation unit 232 of the execution control unit of the spare machine 20 to the hardware control board 17 of the active machine 10. Here, it is assumed that conduction is successful. Note that even if the continuity check fails, the service continuation is not greatly affected, so that the active device 10 can be incorporated into the cluster configuration.

  Step 3) The failure estimation unit 137 determines that the failure occurrence location is “resource” from the results of Steps 1 and 2 above.

  Step 4) The activation execution unit 135 activates the high availability cluster software 140.

  Step 5) The high availability cluster software 140 sets the cluster state of the active machine 10 in the state management information storage unit 164 to “SBY [online]”.

Step 6) Next, the high availability cluster software 140 changes the cluster state of the spare machine 20 from “ACT” to “SBY [standby]”, and the updated active machine 10 and spare machine 20 are updated with respect to the spare machine 20. Notify the status management information. Thus, in a high availability cluster soft 240 of spare machine 20 updates the contents of the state management information storage unit 264.

  Step 7) The state confirmation execution unit 134 confirms that the cluster state of the standby machine 20 is “SBY [standby]” and the cluster state of the active machine 10 is “SBY [online]”. In order to finish the system switching, the cluster state of the spare machine 20 is returned to “SBY [online]”.

  Step 8) The state confirmation execution unit 134 executes state confirmation processing. If the cluster state of the spare device 20 is not “SBY [online]”, an error is output and the processing is terminated.

  Hereinafter, the processing of the present invention will be described in detail with reference to the drawings.

  7 and 8 are flowcharts of a series of operations in one embodiment of the present invention.

  The “cluster status”, “number of failures”, “error status”, “resource status”, etc. described in the following steps are stored in the status management information storage units 164 and 264 of the active device 10 and the standby device 20. The stored value.

  Prior to the execution of the recovery method, the status check execution unit 134 executes a status check command to read the cluster status stored in the status management information storage unit 164 of the active machine, and the cluster status of the active machine 10 is “NONE”. It may be confirmed that the cluster status of the spare machine 20 is “ACT”. Here, since the high-availability cluster software 140 and 240 exchanges information in the state management information storage units 164 and 264 between the active machine 10 and the spare machine 20, the active machine 10 and the spare machine are used by the status check command. Both states of the machine 20 can be read.

  Step 101) As shown in FIG. 9, the control unit on the management LAN 5 logs in from the maintenance terminal 4 to the control execution units 130 and 230 of the active machine 10 and the spare machine 20. If login to the active machine 10 fails, an error is output to a display device or the like connected to the maintenance terminal 4 and the process ends.

  At this time, the active machine 10 and the spare machine 20 at the time of login are in the following state.

(Current machine status)
Cluster status: "NONE"
Number of failures: 0
Error status: 0
Resource status: 0
(Status of spare machine)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
Step 102) Next, as shown in FIG. 10, a search command as an OS function is executed from the log search unit 131 of the control execution unit 130 of the guest machine 110 of the active machine 10, and a corresponding error message is acquired. When an error message for the router is acquired, the process proceeds to step 103 as an error between the active device 10 and the router 3 to perform continuity confirmation processing.

  If there is no relevant message for the router, search for an error message for the disk. If a message for the disk is obtained, it can be assumed that a problem has occurred in access to the shared disk 30, so an error message is output to the maintenance terminal 4 and the process ends.

  Step 103) When an error message regarding the router is acquired in Step 102, as shown in FIG. 11, in order to improve the accuracy of failure estimation, the continuity confirmation unit 132 of the control execution unit 130 of the guest machine of the active machine 10 A definite diagnosis of continuity is performed by sending PING, which is a command for confirming network communication, to the IP address of the router 3 which is an OS function. Since a temporary failure due to an instantaneous interruption on the network is also assumed, the PING may be transmitted a predetermined number of times every predetermined time. If the continuity is unsuccessful, there is a high possibility that a failure will occur again even if the current machine 10 is incorporated into the cluster configuration, so an error is output to the maintenance terminal 4 and the process is terminated. On the other hand, if the continuity is successful, it is assumed that the router 3 is temporarily broken, and the active device 10 can be incorporated into the cluster configuration.

  Step 104) Next, the failure location estimation unit 137 of the active machine 10 determines whether or not there is a resource failure based on the log search result. As illustrated in FIG. 10, the log search unit 131 of the control execution unit 130 of the guest machine of the active machine 10 searches the log storage unit 163 for an error message. The failure location estimation unit 137 identifies the failure resource from the resource ID by acquiring the resource ID from the retrieved error message. If the identified resource is a DB or the like, the process proceeds to step 106 in order to improve fault tolerance and performance, and fault estimation of the forced power-off function for system switching processing is performed.

  If the forced power-off function has been introduced after identifying the location of failure when the working machine is stopped, the process proceeds to step 105. If not, the process proceeds to step 107. In order to determine the presence or absence of the forced power-off function, a flag indicating the presence or absence of the forced power-off function may be set in the state management information storage unit 164 in advance, and the flag may be referred to. Moreover, the setting state of the forced power-off function may be set for the entire system or may be set for each server. Furthermore, when a failure occurs in the network 10 or the shared disk 30 in the active machine 10 and the spare machine 20, a forced power-off function unit that transmits an instruction to forcibly power off to the power control unit of another server May be provided. When the forced power-off function unit is included, an error is output to the maintenance terminal 4.

  Step 105) When the forced power-off function is introduced, the cause of the failure of the forced power-off function is specified in the spare machine 20, as shown in FIG. In the log search unit 231 of the control execution unit 230 of the guest machine of the spare machine 20, a search command that is an OS function is executed, the log of the corresponding error message is acquired from the log storage unit 263, and the fault location estimation unit 237 Then, it is determined whether the forced power-off process (reset) by the forced power-off function has failed, and the determination result is notified to the maintenance terminal 4.

  Specifically, in the log search unit 231 of the control execution unit 230 of the standby machine 20, a keyword is input and the corresponding error message is searched from the log storage unit 263 of the backup machine 20. When one or more corresponding error messages are output, it is determined that the forced power-off (reset) has failed.

  Step 106) In order to improve the failure estimation accuracy of the hardware control board 17, the continuity confirmation unit 232 of the execution control unit 230 of the spare machine 20 is addressed to the hardware control board 17 of the active machine 10 in accordance with an instruction from the maintenance terminal 4. Then, the PING process is performed via the management LAN 5 to confirm that the continuity is successful, and the process proceeds to step 107. On the other hand, if PING fails, an error is output. In this case, it is estimated that the failure of the forced power-off function of the spare machine 20, the failure of the hardware control board 17 of the active machine 10, or the network failure of the management LAN 5, but the service provision has little influence. It is also possible to incorporate the current machine into the cluster configuration (in this embodiment, the process proceeds to step 107). If the ping process is successful, it is considered a temporary failure due to a momentary network interruption. Therefore, it is assumed that the active machine can be incorporated into the cluster configuration, and the process proceeds to step 107.

  The states of the current machine 10 and the spare machine 20 at this time are as follows.

(Current machine status)
Cluster status: "NONE"
Number of failures: 0
Error status: 0
Resource status: 0
(Status of spare machine)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
Step 107) If the failure estimation unit 137 of the active machine 10 can estimate that a problem has occurred in communication between the host machine 150 and the guest machine 110 based on the log search result, the process proceeds to Step 108. 109.

  Step 108) In the active machine 10, as shown in FIG. 13, the continuity confirmation unit 132 of the control execution unit 130 of the guest machine 110 executes PING, and the management LAN network virtual I / F 13 and service LAN network of the host machine 150 Confirm the continuity to the router 3 via the I / F 11, and if the continuity is successful, the process proceeds to step 109. If it is confirmed as “impossible”, an error is output to the maintenance terminal 4 and the process is performed. Exit.

  Specifically, when the corresponding message indicates that the continuity on the service LAN side used by the client device for access has failed, the continuity confirmation unit 132 of the active device 10 sends the service LAN network I / O of the host machine. Execute the PING command addressed to the IP address of F11 and confirm that continuity is not possible. If the attribute value of the corresponding error message indicates a continuity failure from the guest machine to the host machine, the PING command is executed from the continuity confirmation unit 123 of the active machine 10 to the IP address of the management LAN interface 12 of the host machine. If the connection is successful, the process proceeds to step 109. If it is confirmed that the connection is not possible, an error is output to the maintenance terminal 4 and the process ends.

Step 109) The start execution unit 135 of the control execution unit 130 of the active machine 10 and the high availability cluster software 140 are:
When the failure location is “resource” in step 104;
If the forced power-off function (reset) fails in step 105, but the connection to the hardware control board 17 of the current machine 10 succeeds in step 106 (even if it fails), or to the router 3 in step 107 If continuity is successful and continuity from the guest machine to the host machine is successful;
As shown in FIG. 14, the following recovery processing is performed. The following numbers in parentheses correspond to the processes indicated by the numbers in parentheses in FIG.

  (1) The maintenance terminal 4 instructs the control execution unit 130 of the guest machine 110 of the active machine 10 to start execution.

  (2) The command execution unit 136 of the control execution unit 130 of the active machine 10 activates the high availability cluster software 140 of the active machine 10 by the activation execution unit 135.

  (3) When the high availability cluster software 140 of the active machine 10 is activated, the cluster state of the state management storage unit 164 of the active machine 10 is updated from “NONE” to “SBY [online]”.

  (4) The high availability cluster software 140 notifies the updated cluster state to the high availability cluster software 240 of the standby machine 20.

  (5) The high availability cluster software 240 of the standby machine 20 changes the cluster status of the status management storage unit 264 from “ACT” to “SBY [online]” in response to the notification from the high availability cluster software 140 of the active machine 10.

  The current machine 10 and the spare machine 20 at this time are in the following state.

(Current machine status)
Cluster status: "SBY [online]"
Number of failures: 0
Error status: 0
Resource status: 0
(Status of spare machine)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
Step 110) The active machine 10 executes the status check command of the high availability cluster software 140 in the command execution unit 136 of the control execution unit 130 of the guest machine 110 as shown in FIG. Get the cluster status and confirm that it is "SBY [online]". If it is not in this state, an error is output.

  Step 111) The process shown in FIG. 16 is performed in order to improve the fault tolerance and the performance against the active machine 10. The numbers in parentheses below correspond to the processes indicated by the numbers in FIG.

  (1) The control unit of the maintenance terminal 4 causes the control execution unit 130 of the guest machine 110 of the active machine 10 to execute the system switching process.

  (2) The command execution unit 136 of the control execution unit 130 of the active machine 10 causes the system switching unit 133 to change the cluster state to the active machine 10 “ACT” and the standby machine “SBY [standby]”. 140 executes a system switch command.

  (3) The cluster state of the active machine 10 is changed to “ACT” by changing the cluster state of the standby machine stored in the state management storage unit 164 of the active machine 10 to “SBY [standby]”.

  (4) The high availability cluster software 140 of the active machine 10 notifies the high availability cluster software 240 of the standby machine 20 of the cluster status updated in (3) above.

  (5) Upon receiving the notification, the high availability cluster software 240 of the spare machine 20 updates the cluster state of the active machine and the spare machine in the state management storage unit 264 of the spare machine 20 in the same manner as (3) above.

  (6) The high availability cluster software 240 of the spare machine 20 stops the resource 220 of the spare machine 20.

  (7) The high availability cluster software 140 of the active machine 10 activates the resource 120 of the active machine 10.

  Step 112) As shown in FIG. 17, the control execution unit 130 of the guest machine 110 of the active machine 10 executes the status check command of the high availability cluster software 140, and the cluster status of the status management information storage unit 164 is “ACT”. Confirm that the resource status is "1" and "Started".

  The current machine 10 and the spare machine 20 at this time are in the following state.

(Current machine status)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
(Status of spare machine)
Cluster status: "SBY [standby]"
Number of failures: 0
Error status: 0
Resource status: 0
Step 113) In the active machine 10, the termination process shown in FIG. 18 is performed. The following numbers in parentheses correspond to the processes indicated by the numbers in parentheses in FIG.

  (1) The maintenance terminal 4 connected to the management LAN 5 transmits a system switching processing request to the control execution unit 130 of the guest machine 110 of the active machine 10.

  (2) Since the command execution unit 136 of the control execution unit 130 of the active machine 10 changes the cluster state of the standby machine 20 from “SBY [standby]” to “SBY [online]” by the system switching unit 133, high availability The cluster software 140 is caused to execute a system switching command.

  (3) The high availability cluster software 140 of the active machine 10 updates the cluster status of the standby machine 20 in the status management storage unit 164 of the active machine 10 by a system switching command.

  (4) An update process from “SBY [standby]” to “SBY [online]” is notified from the high availability cluster software 140 of the active machine 10 to the high availability cluster software 240 of the standby machine 20.

  (5) Upon receiving the notification, the high availability cluster software 240 of the spare machine 20 updates the cluster status of the spare machine in the status management storage unit 264 of the spare machine 20 to “SBY [online]”.

  Step 114) As shown in FIG. 19, the active machine 10 executes the status command of the high availability cluster software 140 in the status check execution unit 134 of the control execution unit 130 of the guest machine 110, and the cluster status of the standby machine 20 is “ If it is not “SBY [online]”, an error is output to the maintenance terminal 4 connected to the management LAN 5.

  The current machine 10 and the spare machine 20 at this time are in the following state.

(Current machine status)
Cluster status: "ACT"
Number of failures: 0
Error status: 0
Resource status: 1 (Started)
(Status of spare machine)
Cluster status: "SBY [online]"
Number of failures: 0
Error status: 0
Resource status: 0
Step 115) Log out of the active machine 10.

  As described above, according to the present invention, in the virtual environment, the cause that the active machine is not incorporated in the cluster configuration can be determined by any of the resources of the active machine, the poor continuity from the guest machine to the host machine, the disk, and the network. It is possible to specify whether it is a category failure. As a result, at least in the case of a resource failure, if the service is operating on the spare machine, the service can be restarted on the active machine. In addition, by specifying the error category, the failure location can be easily identified by presenting the error to the maintenance person.

  Moreover, the failure estimation accuracy can be improved by performing the conduction check after the failure location is estimated. Furthermore, it is possible to automatically recover a resource failure that has occurred due to a temporary failure due to an instantaneous network interruption.

  In addition, when the performance of the spare machine is inferior to that of the active machine, or when a cluster system is composed of two or more active machines and one spare machine, the processing speed of the spare machine providing the service speed A decrease can be avoided.

  For convenience of explanation, the system according to the embodiment of the present invention is described using a functional block diagram. However, the system of the present invention may be realized by hardware, software, or a combination thereof. For example, each functional unit of the server (active machine and spare machine) may be realized by software and installed on the operation system. In addition, the functional units may be used in combination as necessary.

  Although the embodiments and examples of the present invention have been described above, the present invention is not limited to the above-described embodiments and examples, and various modifications and applications are possible within the scope of the claims. is there.

1 Client device 2 Service LAN
3 Router 4 Maintenance terminal 5 Management LAN
10 servers (current machine)
11, 21 Service LAN network interface 12, 22 Management LAN network interface 13, 23 Management LAN network virtual interface 14, 24 Service LAN network virtual interface 15, 25 Management LAN network virtual interface 16, 26 Service LAN network virtual interface 17, 27 Hardware Wear control board 20 Server (spare machine)
30 Shared disk 110, 210 Guest machine 151, 251 Device batch monitoring unit 120, 220 Resource 130, 230 Control execution unit 131, 231 Log search unit 132, 232 Continuity check unit 133, 233 System switching unit 134, 234 Status check execution unit 135, 235 Start execution unit 136, 236 Command execution unit 137, 237 Failure location estimation unit 140, 240 High availability cluster software 141, 241 Network monitoring result acquisition unit 142, 242 Disk monitoring result acquisition unit 160, 260 Built-in disk 161, 261 Host machine allocation disks 162 and 262 Guest machine allocation disks 163 and 263 Log storage units 164 and 264 State management information storage units 170 and 270 Hypervisor

Claims (9)

In a cluster system composed of a working machine and a spare machine in a virtual environment where a hypervisor is introduced, a cluster state in which the working machine and the spare machine manage a cluster state indicating an operating state of the working machine and the spare machine Management means, state management information storage means for storing the cluster state and failure state, and log storage means for storing a failure log indicating the failure location, respectively, the spare machine is in service, and the active machine is in a cluster configuration A failure recovery method in a virtual environment for identifying the cause of a failure when not incorporated,
A fault location estimating step of estimating a fault location of the active machine based on a result of the fault search means of the active machine being retrieved from the log storage means of the active machine;
In the failure location estimation step, when it is estimated that the failure location of the active device is a network failure, the continuity confirmation means of the active device confirms continuity to the router connected to the active device. A confirmation step;
A cluster configuration starting step for transitioning the cluster state of the active unit to a “state in which service can be changed” when the continuity confirmation to the router is successful;
A failure recovery method in a virtual environment, comprising: In an environment where a host machine and a guest machine are installed in the current machine,
When it is specified that the communication failure between the host machine and the guest machine in the continuity confirmation step,
In the conduction confirmation step,
Check the conduction failure from the guest machine to the host machine,
In the cluster configuration startup step,
The failure recovery method in a virtual environment according to claim 1, wherein when the connection from the guest machine to the host machine is successful, the cluster state of the active machine is changed to “a state in which the service machine can be changed to service operation”. When it is estimated that the category of the failure location is a resource in the failure location estimation step,
The failure recovery method in a virtual environment according to claim 1 or 2, wherein, in the cluster configuration starting step, the cluster state of the active machine is changed to "a state in which the service can be changed to service operation". In the failure location estimation step, a forced power-off function for forcibly turning off the power of other servers in the event of a failure has been introduced to the active machine and the spare machine, and the forced power-off function from the log storage means of the spare machine If an error regarding is found, it is determined that a serious error has been detected in the working machine,
In the conduction confirmation step,
Conducting confirmation from the spare machine to the power control means of the working machine,
In the cluster configuration startup step,
4. The failure recovery method in a virtual environment according to claim 3, wherein when the connection to the power control means is successful in the continuity confirmation step, the cluster state of the active device is changed to “a state where the service can be shifted to service operation”. . In a cluster system composed of a working machine and a spare machine in a virtual environment where a hypervisor is introduced, a cluster state in which the working machine and the spare machine manage a cluster state indicating an operating state of the working machine and the spare machine Management means, state management information storage means for storing the cluster state and failure state, and log storage means for storing a failure log indicating the failure location, respectively, the spare machine is in service, and the active machine is in a cluster configuration A server that operates as a working machine to identify the cause of failure when it is not installed,
Based on the result retrieved from the log storage means, failure location estimation means for estimating the failure location of the working machine,
In the failure location estimation means, when it is estimated that the failure location of the working machine is a network failure, continuity confirmation means for confirming continuity to the router connected to the active device,
Cluster configuration activation means for transitioning the cluster state of the active machine to “a state in which the service can be transitioned” when the continuity confirmation to the router is successful,
The server characterized by having. The conduction confirmation means
In the environment where the host machine and guest machine are installed on the server, if the communication failure between the host machine and the guest machine is identified, the continuity from the guest machine to the host machine is confirmed. Including means to
The cluster configuration starting means
6. The server according to claim 5, further comprising means for causing the cluster state of the active machine to transition to “a state in which the service can be activated” when the continuity check means succeeds in continuity from the guest machine to the host machine. . The cluster configuration starting means
The server according to claim 5 or 6, further comprising means for causing the cluster state of the server to transition to "a state in which the service can be shifted to operation" when the failure point category is a resource. In an environment where a forced power-off function for forcibly turning off the power of other servers in the event of a failure is introduced to the active machine and the spare machine,
The cluster configuration starting means
When an error related to the forced power-off function is detected in the spare machine, and the continuity confirmation means of the spare machine checks the continuity from the spare machine to the power control means of the working machine, and it is found that the continuity is defective. The server according to claim 7, further comprising means for preventing the cluster state of the server from transitioning to “a state in which a transition to service operation is possible”.   The program for functioning a computer as each means which comprises the server of any one of Claim 5 thru | or 8.

Images