JP2006053728A - Failure handling rule propagation method, failure restoration device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a failure restoration device for sorting out an effective rule while suppressing the management cost of a plurality of failure restoration devices. <P>SOLUTION: In this failure restoration device, an operation means 1115 compares, upon receiving a failure responding rule and environmental information related thereto from another failure restoration device 102, the received environmental information with environmental information for an information processor 310 stored in an environmental information storage means 111 to calculate an application evaluation value of the failure responding rule, and compares the calculated application evaluation value with an application reference value of the own failure restoration device. The failure responding rule is stored in a response storage means 1013 when the application evaluation value is larger than the application reference value, and disposed when the evaluation value is smaller than the reference value. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a failure recovery device and a failure handling rule propagation method having a rule propagation function, and more particularly to a failure recovery device and a failure handling rule propagation method having a rule application function according to different device environments.

  Information communication services such as information provision and product sales via a communication network are rapidly expanding their use because they can provide business efficiency and detailed user services without much time and distance restrictions. As the range of use expands, the impact when a computer that provides these services falls into a fault has increased, and its fault tolerance has become a major issue.

  Japanese Patent Application Laid-Open No. 2004-151561 describes a first conventional technique for a failure recovery apparatus. In this first prior art, failure handling rules are stored in the rule storage means for each of several types of failures, and when a failure occurs in the target device, a failure handling rule corresponding to the type of failure that has occurred. The countermeasure command is taken out from the rule storage means and automatically executed.

A second prior art similar to the first prior art is described in Patent Document 2. However, this second prior art presents an optimum coping method to the worker through the interface screen, and the actual coping is advanced by the operator's own manual work. Further, the second prior art makes it possible to newly add a coping method executed at the operator's own judgment and linked to recovery to the rule accumulating unit as an effective coping method for the current type of failure.
Japanese Patent Publication No. 7-54474 Japanese Patent Laid-Open No. 8-222295

  As can be seen from the first prior art described above, a failure recovery apparatus of a type that uses a failure handling rule to recover from a failure functions effectively with respect to the failure described in the failure handling rule. Does not work. For this reason, how to expand the failure handling rules is particularly important for improving the fault tolerance.

  A typical method for expanding the failure handling rules of the failure recovery apparatus is that the administrator of the failure recovery apparatus creates a new failure handling rule and registers it in the failure recovery apparatus. In the second prior art, it is shown that a countermeasure method executed at the operator's own discretion and linked to recovery is newly added to the rule storage means as an effective countermeasure method for the current fault, and the fault handling rules are expanded. Has been.

  However, in order for the administrator himself to create a new failure handling rule, a wealth of knowledge and experience is required, and the burden on the administrator increases. In particular, when there are a plurality of target failure recovery apparatuses, the burden on the administrator is significantly increased. Specifically, in the case of automatic failure handling, the administrator needs to create a failure handling rule that consists of elements such as a conditional expression that defines the failure state and a response command that is executed when a failure occurs. In the case of a failure recovery device targeted for a processing device, it is necessary for an administrator to grasp the configuration of each information processing device, complete fault handling rules that are effective in each configuration, and set them individually. Therefore, when adding a new failure handling rule or modifying an existing failure handling rule, if the information processing device becomes large-scale, all the characteristics of each device and their combined characteristics will be managed by the administrator. Therefore, the burden on the administrator, such as the setting and correction of failure handling rules, will increase dramatically.

  The present invention has been proposed in view of such circumstances, and an object thereof is to reduce a burden on an administrator required for setting a rule for a failure recovery apparatus.

  According to the first failure handling rule propagation method of the present invention, the application means of the failure recovery device includes a failure handling rule from another failure recovery device, and hardware of the information processing device to be managed by the other failure recovery device. Receiving the environment information related to the failure handling rule among the environment information indicating the hardware configuration and the software configuration, and the received hardware information and the hardware configuration of the information processing apparatus to be managed by the own fault recovery apparatus; Compared with the environment information stored in the environment information storage means for storing environment information indicating the software configuration, the failure handling rule received from the other failure recovery device is the management target of the own failure recovery device A step of calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for an information processing device; b) the control means of the failure recovery device includes: Storing the issued applicability evaluation value is applied a reference value or more when the received troubleshooting rule Action storage means of its own fault recovery apparatus, characterized in that it comprises a.

  In the first failure handling rule propagation method, when a failure handling rule and environment information related to the failure handling rule are propagated from another failure recovery device to a certain failure recovery device, When the application means determines the applicability of the failure handling rule according to the similarity of the environment information and an application evaluation value equal to or higher than a certain application reference value is obtained, the failure handling rule is added to the handling storage means. The Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  In the second failure handling rule propagation method of the present invention, a) the propagation means of the first failure recovery device is the failure handling rule stored in the handling storage means of the own failure recovery device, and the management target of the own failure recovery device Among the environmental information stored in the environmental information storage means for storing the environmental information indicating the hardware configuration and software configuration of the information processing apparatus, the environmental information related to the failure handling rule is one or more second A step of transmitting to the failure recovery device; b) the environment information received from the first failure recovery device by the application means of the second failure recovery device and the information processing device to be managed by the own failure recovery device; The failure received from the first failure recovery device by comparing with the environment information stored in the environment information storage means for storing environment information indicating the hardware configuration and software configuration Calculating an application evaluation value that evaluates the possibility that the processing rule can be applied as a failure handling rule for the information processing apparatus to be managed by the own fault recovery apparatus; c) control means for the second fault recovery apparatus Storing the received failure handling rule in the handling storage means of the own failure recovery device when the calculated application evaluation value is equal to or greater than the application reference value.

  In the second failure handling rule propagation method, a failure handling rule newly registered in the handling storage means of another failure recovery device and environment information related thereto are propagated to a certain failure recovery device. If the applicability of the fault handling rule is determined according to the similarity of the environment information in the application means in the fault recovery apparatus at the propagation destination and an application evaluation value greater than a certain application reference value is obtained, the fault A handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for rule setting for a plurality of failure recovery apparatuses.

  In the third failure handling rule propagation method of the present invention, a) the propagation means of the first failure recovery device is the failure handling rule stored in the handling storage means of the own failure recovery device, and the management target of the own failure recovery device The environment information related to the failure handling rule among the environment information stored in the environment information storage means for storing the environment information indicating the hardware configuration and software configuration of the information processing apparatus is propagated by the own fault recovery apparatus. A step of transmitting to the route selection means; b) an actual value stored in the actual value storage means by the propagation route selection means of the first failure recovery apparatus, the received fault handling rule and the related environment information Transmitting to a second failure recovery device having a reference value equal to or greater than a reference value; c) the environment information received from the first failure recovery device by the application means of the second failure recovery device; Compared with the environment information stored in the environment information storage means for storing the environment information indicating the hardware configuration and software configuration of the information processing device to be managed by the recovery device, and received from the first failure recovery device Calculating an application evaluation value that evaluates the possibility that the failure handling rule that can be applied as a failure handling rule for the information processing apparatus to be managed by the own failure recovery apparatus; d) the second failure recovery apparatus A step of storing the received failure handling rule in a handling storage unit of the own failure recovery device when the calculated application evaluation value is equal to or greater than an application reference value; e) the second failure recovery device; Propagation path selection means manages the failure handling rule received from the first failure recovery apparatus stored in the action storage means of the own fault recovery apparatus A step of transmitting the result applied to the information processing apparatus to the first failure recovery apparatus; f) the failure received by the propagation path selection unit of the first failure recovery apparatus from the second failure recovery apparatus; Updating the result value storage means in accordance with the application result of the handling rule.

  In this third failure handling rule propagation method, a failure handling rule newly registered in the handling storage means of another failure recovery device and environment information related thereto are propagated to a certain failure recovery device. If the applicability of the fault handling rule is determined according to the similarity of the environment information in the application means in the fault recovery apparatus at the propagation destination and an application evaluation value greater than a certain application reference value is obtained, the fault A handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for rule setting for a plurality of failure recovery apparatuses. Further, since the failure recovery device for propagating the failure handling rule is selected by the propagation path selection means based on the past results, useless propagation can be prevented.

  According to the fourth failure handling rule propagation method of the present invention, a) the complementary means of the first failure recovery device is the incomplete failure handling rule input from the user management device, the management target of the own failure recovery device, An incomplete rule supplement request that includes environmental information related to the failure handling rule among the environmental information stored in the environmental information storage unit that stores environmental information indicating the hardware configuration and software configuration of the information processing apparatus. A step of transmitting to the second failure recovery apparatus; b) the complementary means of the second failure recovery apparatus stores environment information indicating the hardware configuration and software configuration of the information processing apparatus to be managed by the own failure recovery apparatus A correspondence table showing a correspondence between the environment information stored in the environment information storage means and the failure handling rules stored in the handling storage means of the self-failure recovery device, and the environment information A failure handling rule related to the same environment information as the environment information included in the uncompleted rule complement request among the failure handling rules stored in the handling storage unit with reference to the storage unit and the handling storage unit And c) content of the completed part in the incomplete fault handling rule included in the incomplete rule complementing request among the extracted fault handling rules by the complementing means of the second fault recovery device. Are selected, the environment information related to the selected failure handling rule is acquired with reference to the correspondence table and the environment information storage means, and the acquired environment information and the selected failure handling rule are A step of transmitting a complementary response to the first failure recovery device; d) before the first failure recovery device application means is included in the complementary response received from the second failure recovery device The environmental information is compared with the environmental information stored in the environmental information storage means for storing the environmental information indicating the hardware configuration and software configuration of the information processing device to be managed by the self-failure recovery device. 2. Calculate an application evaluation value that evaluates the possibility that the failure handling rule included in the complementary response received from the second failure recovery device can be applied as a failure handling rule for the information processing device to be managed by the own failure recovery device E) The control means of the first failure recovery apparatus stores the received failure handling rule in the action storage means of the own failure recovery apparatus when the calculated application evaluation value is greater than or equal to an application reference value. A step.

  In the fourth fault handling rule propagation method, when an incomplete fault handling rule and an environment information related to the incomplete fault handling rule are input to a fault recovery apparatus, the request is transferred to another Sent to the disaster recovery device, based on the similarity of the environmental information and the similarity of the completed part of the rule, the incomplete part of the incomplete fault handling rule is complemented by the complete fault handling rule of the other fault recovery unit Is returned to the original failure recovery apparatus, and then the applicability unit determines the applicability of the failure handling rule according to the similarity of the environment information, as in the first failure handling rule propagation method. When an application evaluation value greater than a certain application reference value is obtained, the failure handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  According to the fifth failure handling rule propagation method of the present invention, a) the complementing means of the first failure recovery device issues an incomplete rule supplement request including an incomplete failure handling rule input from the user management device. A step of transmitting to the second failure recovery device; b) the second failure recovery device complementing means responds to the incomplete rule complement request from the failure handling rules stored in the failure storage device of the own failure recovery device; Environment information that stores the environment information indicating the hardware configuration and software configuration of the information processing apparatus to be managed by the self-failure recovery apparatus by selecting the contents of the completed part in the included incomplete fault handling rules Refer to the correspondence table showing the correspondence between the environment information stored in the storage means and the failure handling rules stored in the handling storage means of the self-failure recovery device, and the selection with reference to the environment information storage means. Acquiring environmental information related to the fault handling rule and transmitting a complementary response including the acquired environmental information and the selected fault handling rule to the first fault recovery device; c) the first fault The environment information indicating the hardware configuration and software configuration of the information processing apparatus to be managed by the own fault recovery apparatus, and the environment information included in the complementary response received by the recovery apparatus from the second fault recovery apparatus Is compared with the environment information stored in the environment information storage means, and the failure handling rule included in the complementary response received from the second failure recovery device is the management target of the own failure recovery device A step of calculating an application evaluation value that evaluates the possibility of application as a failure handling rule for the information processing apparatus; d) a control means of the first failure recovery apparatus; Characterized in that it comprises the step of storing a troubleshooting rule applicable evaluation value the calculated is the received time of the above application the reference value in addressing the storage means of its own failure recovery device.

  In the fifth failure handling rule propagation method, when an incomplete rule supplement request including an incomplete failure handling rule is input to a certain failure recovery device, the request is sent to another failure recovery device. Based on the similarity of the completed part of the rule, the incomplete part of the incomplete fault handling rule is complemented by the complete fault handling rule of the other fault recovery apparatus and the related fault information is added to the original fault recovery apparatus. After that, similar to the first failure handling rule propagation method, the applicability of the failure handling rule is determined according to the similarity of the environment information in the application unit, and an application evaluation value equal to or higher than a certain application reference value is determined. If obtained, the failure handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  According to a sixth failure handling rule propagation method of the present invention, in any one of the first to fifth failure handling rule propagation methods, the control unit receives the reception when the calculated application evaluation value is equal to or more than an application reference value. Instead of performing the process of storing the determined failure handling rule in the handling storage means of the own failure recovery apparatus, the user management is performed on the calculated application evaluation value, the failure handling rule that is the target of the evaluation, and the environment information related thereto. When the storage instruction is displayed on the device and the storage instruction is received from the user management device, the failure handling rule is stored in the handling storage means of the own failure recovery device.

  In the sixth failure handling rule propagation method, if the administrator who has confirmed the failure handling rule propagated from another failure recovery device, its application evaluation value, and related environment information inputs a storage instruction, Since the failure handling rules that have been stored can be stored in the handling storage means, it is possible to reduce the burden on the administrator for setting the rules for the failure recovery apparatus.

  According to a seventh failure handling rule propagation method of the present invention, in any one of the first to sixth failure handling rule propagation methods, the application means defines an application evaluation value corresponding to a pattern of the type of matching environment information. The application evaluation value is determined with reference to the application evaluation value table.

  In the seventh failure handling rule propagation method, the application evaluation value can be derived in detail.

  A first failure recovery apparatus according to the present invention includes a handling storage unit that stores a failure handling rule including a conditional expression and a handling command, a state detection unit that detects a state of an information processing device to be managed, and the detected A handling candidate search unit that selects from the handling storage unit a fault handling rule having a conditional expression that matches the status, a handling execution unit that executes a handling command of the selected fault handling rule on the information processing device, Receiving environmental information storage means for storing environmental information indicating the hardware configuration and software configuration of the information processing apparatus; a fault handling rule transmitted from another fault recovery apparatus; and environment information related to the fault handling rule; Comparing the received environment information with the environment information stored in the environment information storage means, the propagated fault handling rule is Applying means for calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for the computer, and when the calculated application evaluation value is equal to or greater than an application reference value, the propagated failure handling rule is stored in the handling storage means. And a control means for storing in the storage.

  In this first failure recovery device, when a failure handling rule and its related environment information are propagated from the other failure recovery device to the failure recovery device, the application means determines the failure according to the similarity of the environment information. When the applicability of the handling rule is determined and an application evaluation value greater than a certain application reference value is obtained, the fault handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  The second failure recovery apparatus according to the present invention is a correspondence table that holds correspondence between the failure handling rules stored in the handling storage means and the environment information stored in the environment information storage means in the first failure recovery apparatus. The environment information related to the failure handling rule to be propagated to other failure recovery devices among the failure handling rules stored in the handling storage means is acquired from the environment information storage means with reference to the correspondence table and propagated Propagation means for transmitting to a fault recovery device together with a fault handling rule.

  In this second failure recovery device, the failure handling rule stored by the administrator in the response storage means and the environment information related thereto are propagated to other failure recovery devices by the propagation means, and the propagation destination failure recovery device As with the first failure recovery apparatus, when the applicability of the failure handling rule is determined according to the degree of similarity of the environment information in the application means, and an application evaluation value greater than a certain application reference value is obtained. The failure handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for rule setting for a plurality of failure recovery apparatuses.

  The third failure recovery apparatus of the present invention is a correspondence table that holds correspondence between the failure handling rules stored in the handling storage means and the environment information stored in the environment information storage means in the first failure recovery apparatus. The environment information related to the failure handling rule to be propagated to other failure recovery devices among the failure handling rules stored in the handling storage means is acquired from the environment information storage means with reference to the correspondence table and propagated Propagation means to be transmitted together with the failure handling rule, actual value storage means for holding the actual value for each other fault recovery device, failure handling rule received from the propagation means and environment information related thereto are stored in the actual value storage means. The actual value stored is transmitted to another failure recovery device that is equal to or greater than a reference value, and the actual value is determined according to the application result of the failure handling rule received from the other failure recovery device of the transmission destination. Characterized in that it comprises a propagation path selection means for updating the storing means.

  In this third failure recovery device, the failure handling rule and the environment information related to it stored in the response storage means by the administrator are propagated to other failure recovery devices by the propagation means. As with the first failure recovery apparatus, when the applicability of the failure handling rule is determined according to the degree of similarity of the environment information in the application means, and an application evaluation value greater than a certain application reference value is obtained. The failure handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for rule setting for a plurality of failure recovery apparatuses. Further, since the failure recovery device for propagating the failure handling rule is selected by the propagation path selection means based on the past results, useless propagation can be prevented.

  According to a fourth failure recovery apparatus of the present invention, the first failure recovery apparatus receives an incomplete rule supplement request including an incomplete failure handling rule input from the user management apparatus and environment information related thereto. When a complementary response including a completed fault handling rule and environment information related thereto is received as a response to the incomplete rule supplement request from the other fault recovery device. And a supplementary means for transmitting to the applying means the failure handling rule and environmental information related thereto.

  In the fourth failure recovery device, when the administrator inputs an incomplete rule supplement request including an incomplete failure handling rule and environmental information related to the incomplete failure handling rule, the request is sent to another failure recovery device, When a complementary response including a failure handling rule complemented in a complete format as a response and its related environment information is received, the failure handling is performed according to the degree of similarity of the environment information in the application means, as in the first failure recovery device. When the applicability of the rule is determined and an application evaluation value equal to or higher than a certain application reference value is obtained, the failure handling rule is added to the handling storage unit. Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  According to a fifth failure recovery apparatus of the present invention, in the fourth failure recovery apparatus, when the complement means receives an incomplete rule complement request from another fault recovery apparatus, the environment information storage means stores the Referring to the correspondence table showing the correspondence between the environment information and the failure handling rule stored in the handling storage unit, the failure stored in the handling storage unit with reference to the environment information storage unit and the handling storage unit Out of the handling rules, a fault handling rule related to the same environmental information as the environmental information included in the incomplete rule supplement request is extracted, and included in the incomplete rule supplement request from the extracted fault handling rules An environment related to the selected failure handling rule is selected by referring to the correspondence table and the environment information storage unit. Gets the broadcast, and characterized in that to transmit to the disaster recovery system of unfinished rules complement requesting completion response including a troubleshooting rule that the selection and the acquired environmental information.

  In the fifth failure recovery device, when the administrator inputs an incomplete rule supplement request including an incomplete failure handling rule and environmental information related to the incomplete failure handling rule, the request is sent to another failure recovery device, Based on the similarity of the environmental information and the similarity of the completed part of the rule, the incomplete part of the incomplete fault handling rule is complemented by the complete fault handling rules of other fault recovery devices, and the relevant environmental information is added Then, as in the case of the first failure recovery apparatus, the applicability unit determines the applicability of the failure handling rule according to the similarity of the environmental information, and exceeds a certain application reference value. Is obtained, the failure handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  According to a sixth failure recovery apparatus of the present invention, in the fourth failure recovery apparatus, when the complement means receives an incomplete rule complement request from another fault recovery apparatus, the fault handling stored in the handling storage means From the rules, the one that matches the contents of the completed part in the incomplete fault handling rule included in the incomplete rule supplement request is selected, and the environment information stored in the environment information storage means and the handling storage The environment information related to the selected failure handling rule is acquired with reference to the correspondence table showing the correspondence with the failure handling rule stored in the means and the environment information storage means, and the acquired environment information and the selection A complementary response including the failure handling rule that has been completed is transmitted to the failure recovery device that is the incomplete rule complement request source.

  In the sixth failure recovery apparatus, when the administrator inputs an incomplete rule supplement request including an incomplete failure handling rule, the request is sent to another failure recovery apparatus, and the similarity of the completed part of the rule The incomplete portion of the failure handling rule that has not been completed based on the nature is complemented by the complete failure handling rule of another failure recovery device and returned to the original failure recovery device with the related environment information, and then the first As in the case of the failure recovery apparatus, the applicability means determines the applicability of the failure handling rule according to the similarity of the environment information, and if an application evaluation value greater than a certain application reference value is obtained, A failure handling rule is added to the handling storage means. Thereby, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  According to a seventh failure recovery apparatus of the present invention, in any one of the first to sixth failure recovery apparatuses, the control means is configured to cope with the received failure when the calculated application evaluation value is equal to or more than an application reference value. Instead of performing the process of storing the rule in the handling storage means, the calculated application evaluation value, the fault handling rule subjected to the evaluation, and the related environment information are displayed on the user management device, and user management is performed. When a storage instruction is issued from a device, the failure handling rule is stored in the handling storage means.

  In the seventh failure recovery device, if the administrator who confirmed the failure handling rule, its application evaluation value, and related environment information propagated from another failure recovery device inputs a storage instruction, it propagates Since the failure handling rules can be stored in the handling storage means, it is possible to reduce the burden on the administrator required for setting the rules for the failure recovery apparatus.

  In an eighth failure recovery apparatus of the present invention, in any one of the first to seventh failure recovery apparatuses, the application means defines an application evaluation value that defines an application evaluation value according to a pattern of the type of matching environmental information. The application evaluation value is determined with reference to a table.

  In the eighth failure recovery apparatus, the application evaluation value can be derived in detail.

  ADVANTAGE OF THE INVENTION According to this invention, the burden of the administrator required for the rule setting with respect to a failure recovery apparatus can be reduced. The reason for this is that when a failure recovery rule and its related environment information are propagated from another failure recovery device to the failure recovery device, the failure is determined according to the similarity of the environment information in the application means of the destination failure recovery device. When the applicability of the countermeasure rule is determined and an application evaluation value equal to or higher than a certain application reference value is obtained or when the administrator's approval is obtained, the failure countermeasure rule is added to the countermeasure storage means It is.

  Before describing the embodiment of the present invention, a failure recovery apparatus as a premise of the present invention will be described with reference to FIG. 1, FIG. 2, and FIG.

  Referring to FIG. 1, a failure recovery apparatus 101 as a premise of the present invention includes a state detection unit 1011, a countermeasure candidate search unit 1012, a countermeasure storage unit 1013, a control unit 1014, and a countermeasure execution unit 1015. It is connected to an information processing apparatus 301 that is a target for detection and recovery. In addition, a user management apparatus 201 for inputting instructions and data and outputting processing results is connected to the failure recovery apparatus 101. The failure recovery apparatuses 101 and 102, the information processing apparatus 301, and the user management apparatus 201 are composed of general computer hardware, and are connected to a CPU (Central Processing Unit) that controls the entire apparatus and the CPU. And an input device such as a display device and a keyboard for dialog with the user. A program to be executed by the CPU can be stored in a memory or a hard disk, and information can be transmitted / received via a network by a communication unit. In general, the information processing device 301, the state detection unit 1011 and the countermeasure execution unit 1015 constitute an agent device, and the countermeasure candidate search unit 1012, the countermeasure storage unit 1013 and the control unit 1014 constitute a manager device. An agent type configuration is used. The other failure recovery apparatus 102 shown in FIG. 1 also has the same configuration as the failure recovery apparatus 101 of FIG.

  The information processing apparatus 301 is an apparatus that provides an information communication service such as a Web service or a business service that is managed by the failure recovery apparatus 101. Specifically, a device that executes an application program using a communication line such as a Web server, an application server, a database server, or a mail server, or an SNMP protocol that is connected to a communication line such as a router, a firewall, or load distribution. It is a device that can control the program. FIG. 1 shows an example of the information processing apparatus 301, but it may be two or more.

  The state detection unit 1011 obtains or receives from the information processing apparatus 301 an operation state when a failure occurs in the information processing apparatus 301 or an operation state presumed to be a sign of the failure, to the handling candidate search unit 1011 and the control unit 1014. Send. As an example of a method for acquiring or receiving an operation state when a failure occurs in the information processing device 301 or an operation state presumed to be a precursor of the failure, an SNMP agent is made resident in the information processing device 301 and an SNMP request is periodically sent. There are a method of acquiring by transmitting to the SNMP agent, or a method of receiving an SNMP trap event when an SNMP trap is set in the information processing apparatus 301.

  The handling storage unit 1013 includes a plurality of fault handling rules including a conditional expression indicating an operation state when a failure occurs in the information processing apparatus 301, a handling command for recovering or avoiding the failure in the operation state, and a failure sign A plurality of failure handling rules including a conditional expression for determining the estimated operating state and a handling command for recovering or avoiding the failure when the operating state is reached are stored.

  Here, supplementing the operation state, the conditional expression, and the countermeasure command, the operation state when a failure occurs in the information processing apparatus 301 includes the start / stop state of the information processing apparatus 301 and the application program on the information processing apparatus 301. Examples include a start / stop state, an error log of an application program on the information processing apparatus 301, CPU information, memory information, hard disk information, or a combination thereof. The operation state presumed to be a sign of failure includes the current number of concurrent connections, the current throughput, the current input / output traffic amount, or a combination thereof, in addition to the operation state when a failure occurs in the information processing apparatus 301 Etc.

  The conditional expression for determining the operating state that is presumed to be a sign of failure matches the evaluation formula that determines whether or not the operating state that is presumed to be a sign of failure has exceeded a threshold, and the operating state in which the failure has occurred And an evaluation formula for determining whether or not the operating state is to be performed.

  A command for recovering or avoiding a failure in the operating state when a failure occurs in the information processing apparatus 301, and a countermeasure for recovering or avoiding the failure when the operating state is assumed to be a precursor of the failure The command includes a start / stop command for the information processing apparatus 301, a command start / stop command for the information processing apparatus 301, a command for moving or copying data stored in the hard disk to another information processing apparatus 301, and a deletion. A command, a command for moving or copying a program operating on the information processing apparatus 301 to another information processing apparatus, or a combination thereof.

  FIG. 2 shows an example of a failure handling rule including such a conditional expression and a handling command. The failure handling rule of rule number 1 in FIG. 2 is composed of a conditional expression that represents a case where the application of the information processing apparatus 301 is not responding, and a handling command that restarts the application when the condition is satisfied. . Similarly, the failure handling rule of rule number 2 in FIG. 2 is a conditional expression for considering a state where the memory usage rate of the information processing apparatus 301 is 90% or more as a failure, and an operating system when the condition is satisfied. It consists of a countermeasure command to restart. Further, the failure handling rule of rule number 3 in FIG. 2 is composed of a conditional expression representing a case where the memory usage rate of the information processing apparatus 301 is 80% or more and a handling command for restarting the application.

  The handling candidate search unit 1012 takes out a fault handling rule having a conditional expression that matches the operation state detected by the status detection unit 1011 from the handling storage unit 1013, and takes a handling command included in the extracted fault handling rule. To 1015.

  The handling execution unit 1015 interprets the handling command of the fault handling rule received from the handling candidate search unit 1012 and executes the interpreted handling command on the information processing apparatus 301. In addition, a result of executing the handling command on the information processing apparatus 301 is acquired from the information processing apparatus 301, and the acquired execution result is transmitted to the control unit 1014.

  The control unit 1014 includes an operation state when a failure occurs in the information processing apparatus 301 acquired or received by the state detection unit 1011 or an operation state presumed to be a precursor to the failure, and a failure handling rule stored in the handling storage unit 1013 Is transmitted to the user management apparatus 201. Further, the control unit 1014 acquires the execution result of the countermeasure command executed on the information processing apparatus 301 from the countermeasure execution unit 1015 and transmits the result to the user management apparatus 201. Further, the control unit 1014 receives a request for storing a new failure handling rule or an edited failure handling rule in the handling storage unit 1013 from the user management apparatus 201 together with the failure handling rule to be stored, and the failure handling rule. Is stored in the countermeasure storage means 1013. Further, the control unit 1014 receives a request for deleting the failure handling rule stored in the handling storage unit 1013 from the user management apparatus 201, and also performs a process of deleting the failure handling rule stored in the handling storage unit 1013. .

  The user management apparatus 201 includes an operation state in the case where a failure occurs in the information processing apparatus 301 acquired or received by the state detection unit 1011 or an operation state that is presumed to be a precursor of the failure, and a failure stored in the countermeasure storage unit 1013. The handling rule and the execution result of the fault handling command executed on the information processing apparatus 301 are received from the control means 1014 and displayed to the administrator through an output device such as a display device. Further, the user management apparatus 201 sends a request for storing a new failure handling rule or an edited failure handling rule in the handling storage unit 1013 to the control unit 1014 together with the failure handling rule in accordance with an instruction from the user. Furthermore, the user management apparatus 201 sends a request for deleting the failure handling rule stored in the handling storage unit 1013 to the control unit 1014 in accordance with an instruction from the user.

  Next, the overall operation of the failure recovery apparatus as a premise of the present invention will be described with reference to the flowchart of FIG.

  The state detection unit 1011 detects the operation state of the information processing apparatus 301 (S101 in FIG. 3). The operation state is detected in a form according to the conditional expression of the failure handling rule. In the example of FIG. 2, the normal / no response of the AP response, the memory usage rate, and the like are detected. The handling candidate search unit 1012 receives the current operation state from the state detection unit 1011 and searches for a match with the condition formula of the fault handling rule stored in the handling storage unit 1013 (S102). If there is no matching conditional expression, it is determined that no failure has occurred and the process returns to S101. If there is a matching conditional expression, it is assumed that a failure has occurred and a corresponding countermeasure command is searched. For example, in the detected state, when the memory usage rate is 80% or more, it matches the conditional expression of No. 3 in FIG. 2, and thus AP restart is output as a candidate as a countermeasure command. The countermeasure execution unit 1015 receives this countermeasure command and instructs the information processing apparatus 301 to execute the countermeasure command, whereby the application on the information processing apparatus 301 is restarted (S103).

  As described above, regarding the failure described in the failure handling rule in the handling storage unit 1013, the failure of the information processing apparatus 301 can be automatically recovered or avoided. For this reason, expanding the failure handling rules stored in the handling storage means 1013 is important for improving fault tolerance. For this reason, in the failure recovery apparatus 101 shown in FIG. 1, a failure handling rule newly created by the user is registered in the handling storage means 1013 by a user operation from the user management apparatus 201, or a failure that has already been registered. The coping rules can be edited. However, since there is no mechanism for utilizing each other's failure handling rules among a plurality of failure recovery devices, there has been a problem that the burden on the administrator is heavy regarding the task of setting the failure handling rules.

  Next, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

(First embodiment)
A first embodiment of the present invention will be described with reference to FIGS. 4, 5, 6, 7, 8, and 9.

  Referring to FIG. 4, the failure recovery apparatus 101 according to the first exemplary embodiment of the present invention is an environment information storage that stores environment information of the information processing apparatus 301 compared to the failure recovery apparatus that is the premise of FIG. 1. Means 1111; correspondence table 1112 for storing correspondence between environment information stored in environment information storage means 1111 and failure handling rules stored in handling storage means 1013; and environment information related to failure handling rules for the rules Is transmitted to the other failure recovery apparatus 102, the environment information added to the failure handling rule received from the other failure recovery apparatus 102, and the environment information stored in the environment information storage means 1111 In comparison, the possibility that the failure handling rule received from the other failure recovery device 120 can be applied as a failure handling rule for the information processing device 301 is expressed numerically. And application means 1115 for calculating the value was applied evaluation value is different in including a application evaluation value table 1114 that refers to the application unit 1115 calculates the applied evaluation value. Further, the other failure recovery apparatus 102 has the same configuration as the failure recovery apparatus 101 of FIG. Note that an information processing apparatus managed by another failure recovery apparatus 102 and a user management apparatus that exchanges data and instructions between the other failure recovery apparatus 102 and the user are not shown.

  The environment information storage unit 1111 stores hardware and software configuration information constituting the information processing apparatus 301 to be managed as environment information. Specifically, the environment information storage unit 1111 includes a category, hardware or software name, hardware or software version information, hardware or software setting status, hardware or software operating status, hardware or software status. The performance information is stored as environment information of the information processing apparatus 301. To supplement this environmental information, the category represents, for example, an application or an operating system or a network or hardware. The setting state of hardware or software represents information described in an initial setting file of hardware or software, for example. The operating state of hardware or software represents information representing whether the hardware or software is operating, for example. The hardware or software performance information represents, for example, hardware or software load information, and hardware or software usage status.

  FIG. 5 shows an example of environment information stored in the environment information storage unit 1111. In the environment information storage unit 1111, one piece of environment information is defined by a set of an environment information number, an index, and contents. For example, the environment information of environment information number 1 defines that the category belongs to AP. The environment information of environment information number 2 indicates that the category belongs to the OS, the environment information of environment information number 3 indicates that the AP category is a Web server, and the environment information of environment information number 4 indicates that the AP category has an AP category. The environment information of environment information number 5 indicates that the server is an SMTP server, the Web server version is 1.3.28, the environment information of environment information number 6 indicates that the OS version is 9.0, The environment information of environment information number 7 defines that the OS category is a core OS. Here, the environment information indexes in all the failure recovery apparatuses including the failure recovery apparatus 101 and the failure recovery apparatus 102 are unified. In other words, for example, the environment information defining the OS version uses “OS version” as an index in all failure recovery apparatuses. The same applies to other indexes.

  The correspondence table 1112 stores a correspondence relationship between the environment information stored in the environment information storage unit 1111 and the failure handling rule stored in the handling storage unit 1013. An example of the correspondence table 1112 is shown in FIGS. 6 associates a set of rule numbers and failure information numbers related to the failure handling rules for each failure handling rule stored in the handling storage unit 1013. FIG. For each piece of environment information stored in 1111, a set of the environment information number and a failure handling rule number related to the environment information is associated.

  Referring to FIG. 6, rule number 1 is associated with environment information numbers 1, 2, 3, 4, 5, and 7 in FIG. 5. This means that the failure handling rule with the rule number 1 is a rule that is effective for the information processing apparatus 301 on which the version 1.3.28 Web server, the SMTP server, and the backbone OS that are one of the APs are operating. . Similarly, environment information numbers 1, 4, and 7 are assigned to rule number 2, environment information numbers 1 and 3 are assigned to rule number 3, environment information numbers 1, 2, and 3 are assigned to rule number 4, and rule number 5 is assigned. Are associated with environmental information numbers 2, 7, rule information 6 is associated with environmental information numbers 1, 3, 5 and rule number 7 is associated with environmental information numbers 1, 2, 4, 6, 7.

  Referring to FIG. 7, rule numbers 1, 2, 3, 4, 6, and 7 are associated with environment information number 1. This means that there are rules 1, 2, 3, 4, 6, and 7 as failure handling rules related to the category AP of environment information number 1 in FIG. Similarly, environment information number 2 has rule numbers 1, 4, 5, and 7, environment information number 3 has rule numbers 1, 3, 4, and 6, and environment information number 4 has rule numbers 1, 2, and 7. However, the environment information number 5 is associated with rule numbers 1 and 6, the environment information number 6 is associated with rule number 7, and the environment information number 7 is associated with rule numbers 1, 2, 5, and 7, respectively.

  In the present embodiment, for each failure handling rule stored in the handling storage means 1013, the table of FIG. 6 in which the rule number and a pair of failure information numbers related to the failure handling rule are associated with each other, the environment information For each environment information stored in the storage unit 1111, the correspondence table 1112 is configured by the table of FIG. 7 in which the environment information number and the pair of the failure handling rule number related to the environment information are associated with each other. However, for example, in a two-dimensional matrix in which a failure information number and a failure handling rule number are respectively indexed by rows and columns, a configuration in which a mark is given to the intersection of a row of failure information and a column of failure handling rules that are related to each other It is also possible to use one correspondence table.

  The propagation unit 1113 receives the new failure handling rule registered in the handling storage unit 1013 or the edited failure handling rule from the control unit 1014, and refers to the correspondence table 1112 to edit or edit the new failure handling rule. The environment information related to the failure handling rule is acquired from the environment information storage unit 1111, the environment information related to the new failure handling rule or the edited failure handling rule is added, and transmitted to the other failure recovery apparatus 201. . For the environment information corresponding to the failure handling rule, the environment information number corresponding to the rule number of the failure handling rule is acquired from the correspondence table 1112, and the environment information matching the acquired environment information number is acquired from the environment storage unit 1111. Can be obtained.

  The application unit 1115 receives the failure handling rule and the environment information related to the failure handling rule from the other failure recovery apparatus 102, and receives the received environment information and the environment of the information processing apparatus 301 stored in the environment information storage unit 1111. Compared with information, calculates an application evaluation value that evaluates numerically the possibility that the received failure handling rule can be applied as a failure handling rule for the information processing apparatus 301, and receives the received failure handling rule and its related environment Information and the application evaluation value are transmitted to the control means 1014. For example, the application evaluation value is calculated by checking whether the same environment information as the environment information is stored in the environment information storage unit 1111 for each environment information sent from the other failure recovery apparatus 102 together with the failure handling rule. The application evaluation value can be calculated from the result using the calculation formula. For example, if there are three pieces of received environment information and all of them are stored in the environment information storage unit 1111, the application evaluation value is 100%, and if only two of them are stored, 66%, either one If only one is stored, 33%, and if no environment information storage means 1111 has the same environment information, it can be set to 0%. Further, the evaluation value may be calculated by assigning a weight to each environment information. However, the application unit 1115 in the present embodiment calculates the application evaluation value with reference to the application evaluation value table 1114.

  The application evaluation value table 1114 is a table in which application evaluation values corresponding to patterns of types of matching environmental information are defined in advance, and an example thereof is shown in FIG. The application evaluation value table 1114 of this example includes three pieces of environment information in which the environment information received together with the failure handling rules from other failure recovery apparatuses 102 has indexes of “category”, “AP category”, and “Web server version”, respectively. If the environment information having the same contents as all the environment information is stored in the environment information storage unit 1111, two environment information other than the application standard value 100% and “Web server version” are included in the environment information. When stored in the storage unit 1111, the application reference value is 80%, and when two pieces of environmental information other than “AP category” are stored in the environment information storage unit 1111, the application reference value is 30%. Is. Note that the application standard values for other combinations are not shown. In this way, by using the application evaluation value table 1114 in which application evaluation values corresponding to the matching pattern between the environment information related to the failure handling rule and the environment information stored in the environment information storage unit 1111 are registered in advance, The application evaluation value can be derived in detail.

  The control means 1014 includes a request and a trouble handling rule for propagating a new fault handling rule or an edited fault handling rule to another fault restoring apparatus, in addition to the functions of the control means in the fault restoration apparatus of FIG. Is received from the user management apparatus 201, and this failure handling rule is transmitted to the propagation means 1113. In addition, the control unit 1014 sets an application mode indicating whether to automatically determine and apply a failure handling rule received from another failure recovery apparatus 102 or to request confirmation from the administrator from the user management apparatus 201. An application standard value, which is a determination criterion for automatically storing the fault handling rule received from the other fault recovery apparatus 102 in the handling storage unit 1013, is stored.

  Further, when storing the application mode indicating that it is automatically determined and applied, the control unit 1014 receives the failure handling rule, the environment information related to the failure handling rule, and the application evaluation value from the application unit 1115. When the application evaluation value is larger than the application reference value when received, the failure handling rule is stored in the handling storage means 1013, and the failure handling rule, its associated environmental information, the application evaluation value, and the automatic And a result indicating that it is applied to the user management apparatus 201. On the other hand, if the application evaluation value is smaller than the application reference value, the failure handling rule, the environment information related to it, the application evaluation value, and a result indicating that the application evaluation value cannot be automatically applied are transmitted to the user management apparatus 201. .

  In addition, the control unit 1014 receives the failure handling rule, the environment information related to the failure handling rule, and the application evaluation value from the applying unit 1115 when the application mode indicating that confirmation is requested from the administrator is stored. At the same time, a failure confirmation rule application confirmation request is transmitted to the user management apparatus 201 together with them. If a failure handling rule application request is received from the user management apparatus 201, the failure handling rule is stored in the handling storage unit 1013.

  The user management apparatus 201 adds a new failure handling rule or an edited failure handling rule to another fault recovery apparatus 102 in accordance with an instruction from the user, in addition to the functions of the user management apparatus of FIG. A function of transmitting the request to propagate and the failure handling rule to the user management apparatus 201; The user management apparatus 201 also applies an application mode indicating whether to automatically determine and apply a failure handling rule received from another failure recovery apparatus 102 or to ask the administrator for confirmation in accordance with an instruction from the user. Is transmitted to the control means 1014.

  When the user management apparatus 201 receives from the control means 1014 the failure handling rule, the environment information related to the failure handling rule, the application evaluation value, and the result indicating that it has been automatically applied, the user management device 201 And display as a log. Further, when the failure handling rule, the environment information related to the failure handling rule, the application evaluation value, and the result indicating that the failure handling rule cannot be automatically applied are received from the control means 1014, they are displayed to the administrator and are displayed as a log. Record.

  Furthermore, when the user management apparatus 201 receives from the control means 1014 a failure handling rule, environmental information corresponding to the failure handling rule, an application evaluation value, and an application confirmation request for the failure handling rule, these are sent to the administrator. Both are displayed and recorded as a log. When the administrator determines that the failure handling rule is to be applied and an instruction to that effect is input, a request to apply the failure handling rule is transmitted to the control means 1014.

  Next, the operation of the present embodiment will be described.

  In the failure recovery apparatus 101, the failure storage rule is stored in the response storage unit 1013, the application mode and application reference value is stored in the control unit 1014, the environment information of the information processing device 301 is stored in the environment information storage unit 1111, and the failure response is stored in the application evaluation value table 1114. The comparison evaluation information of the environmental information corresponding to the rule, and the correspondence table 1112 store the correspondence information between the failure handling rule and the environmental information. The same applies to other failure recovery apparatuses 102.

  When the administrator adds a new failure handling rule or an edited failure handling rule to the handling storage unit 1013 of the own failure recovery device 101 through the user management device 201, the failure handling rule is transferred to another failure recovery device 102. When it is desired to propagate, a request for propagating the failure handling rule to another failure recovery device 102 is transmitted from the user management device 201 to the control means 1014 of the failure recovery device 101. This propagation request includes the rule number of the failure handling rule to be propagated. Thereafter, a series of processes as shown in FIG. 9 is started in the failure recovery apparatus 101 and other failure recovery apparatuses 102.

  First, the control unit 1014 of the failure recovery apparatus 101 that has received a request for propagating the failure handling rule to another failure recovery apparatus 102 reads the failure handling rule having the rule number included in the propagation request from the handling storage unit 1013, and It transmits to the propagation means 1113 of the own apparatus 102 with a rule number (S201).

  The propagation unit 1113 of the failure recovery apparatus 101 that has received the failure handling rule retrieves the correspondence table 1112 by using the rule number as a key, acquires an environment information number related to the failure handling rule to be propagated, and further acquires the environment information number. The environment information storage unit 1111 is searched for the key to acquire the environment information (index and content), and the failure handling rule to be propagated and the acquired environment information are transmitted to the other failure recovery apparatus 102 (S202).

  The application unit 1115 of the failure recovery apparatus 102 that has received the failure handling rule and the environment information related thereto can be applied as the failure handling rule for the information processing apparatus managed by the own failure recovery apparatus 102. An application evaluation value obtained by numerically evaluating the property is calculated, and the calculated application evaluation value is transmitted to the control unit 1014 of the own apparatus 102 together with the failure handling rule and the environment information related thereto (S203). The application evaluation value is calculated by checking whether or not the same environment information is stored in the environment information storage unit 1111 for each received environment information, and obtaining an application evaluation value corresponding to the result from the application evaluation value table 1114. . For example, when the received environment information is three environment information having indexes of “category”, “AP category”, and “Web server version”, the same environment information as each environment information is stored in the environment information storage unit 1111. For example, the same environment information is stored in the environment information storage unit 1111 for the environment information of “category” and the environment information of “AP category”, but the environment of “Web server version” If the information is not the same, it matches the pattern in the second row of the application evaluation value table 1114 illustrated in FIG. 8, so that 80% is calculated as the application evaluation value.

  The control unit 1014 of the failure recovery apparatus 102 that has received the propagated failure handling rule and the related environment information and application evaluation value automatically determines the failure handling rule based on the application mode set in the own device 102. It is determined whether the mode to be applied or the mode for requesting confirmation from the administrator is set (S204), and the process is divided according to each mode.

(1) In a mode in which failure handling rules are automatically determined and applied First, the control means 1014 of the failure recovery apparatus 102 first applies the application evaluation value received from the application means 1115 of the own apparatus 102 and the application criteria of the own apparatus 102. The values are compared (S205). When the application evaluation value is equal to or greater than the application reference value, the control unit 1014 is unique in the handling storage unit 1013 unless the same fault handling rule as the propagated fault handling rule exists in the handling storage unit 1013 of the own device 102. A rule number is assigned and stored in the countermeasure storage means 1013, and further, a processing result indicating that it has been automatically applied is transmitted to the user management apparatus 201, including the failure countermeasure rule, environment information related thereto, and application evaluation value. (S206). The user management device 201 displays the received information to the administrator through the display device and records it as a log. Further, the control means 1014 includes the number of the environment information stored in the environment information storage means 1111 among the environment information passed from the application means 1115 together with the trouble handling rule, and the rule number of the stored trouble handling rule. Is registered in the correspondence table 1112.

  On the other hand, when the application evaluation value is smaller than the application reference value, the control means 1014 uses the processing result indicating that it has not been automatically applied, including the propagated failure handling rule, the related environment information, and the application evaluation value. (S207). The user management device 201 displays the received information to the administrator through the display device and records it as a log.

(2) In the case of the mode for confirming the administrator The control means 1014 of the failure recovery apparatus 102 applies an application confirmation request including the failure handling rule received from the application means 1115 of the own apparatus 102, environmental information related thereto, and application evaluation value. Is transmitted to the user management apparatus 201 (S208). The user management apparatus 201 displays information received from the control unit 1014 to the administrator through a display device and records it as a log. The administrator confirms the failure handling rule displayed on the display device, the related environment information and the application evaluation value, and if this failure handling rule is applied to the failure recovery device 102, the storage request is stored in the user management. It is transmitted from the apparatus 201 to the control means 1014.

  In response to the application confirmation request, the control unit 1014 of the failure recovery apparatus 102 determines whether a failure handling rule storage request is received from the administrator through the user management device 201 (S209). Then, a rule number that is unique in the handling storage unit 1013 is assigned to the failure handling rule that has been requested to be stored, and stored in the handling storage unit 1013 of the device 102 (S210). At this time, the control unit 1014 includes the number of the environment information stored in the environment information storage unit 1111 among the environment information passed from the application unit 1115 together with the failure handling rule, and the rule number of the stored failure handling rule. Is registered in the correspondence table 1112.

  As described above, the failure handling rule newly stored in the handling storage unit 1013 of the fault recovery apparatus 101 is propagated to the other fault recovery apparatus 102 and stored in the handling storage unit 1013 of the other fault recovery apparatus 102. In the same manner as the case of the failure recovery apparatus that is the premise of FIG. 1, the stored failure handling rule recovers a failure that occurred in the information processing apparatus to be managed by the failure recovery apparatus 102 or its precursor. It will be used as a failure handling rule for recovery. Since the operation at that time is the same as that of the presumed failure recovery apparatus, description thereof is omitted.

  As described above, according to the present embodiment, by constructing a mechanism for propagating the failure handling rule newly stored in the handling storage unit 1013 of the failure recovery apparatus 101 to the other failure recovery apparatus 102, a conventional administrator It is possible to reduce the task of adding or editing a failure handling rule for each different failure recovery apparatus.

  For example, in the case of managing a plurality of information servers (Web servers) and mail servers (SMTP servers), in a conventional failure recovery apparatus, an administrator sets a failure handling rule as shown in FIG. It was necessary to set according to the configuration. In contrast, in the failure recovery apparatus according to the present embodiment, when the correspondence tables shown in FIGS. 6 and 7 are used, the failure handling rule of rule number 1 is the environment information numbers 3 and 4 shown in FIG. If it is understood that the information is common, and the failure handling rule of number 1 in FIG. 2 is set in either the Web server or the SMTP server, the failure recovery device that manages each server (information processing device) The failure handling rule is propagated, and the same failure handling rule is applied to the other side. As a result, even a server that provides a different service can cope with a common failure of “restarting an AP when the AP (Web or SMTP) becomes unresponsive”. The individual setting work is greatly reduced. In addition, even if there is a failure in setting a fault handling rule on a server that can be propagated, a fault handling rule that can be set up will be presented to the administrator using the same procedure. This reduces the work that has been checked, and can achieve a remarkable effect of preventing a setting failure due to an administrator's mistake.

(Second Embodiment)
Next, a second embodiment of the present invention will be described in detail with reference to FIG. 10, FIG. 11, and FIG.

  Referring to FIG. 10, the failure recovery apparatus 101 according to the second embodiment of the present invention is provided with a complementing unit 1116 as compared with the failure recovery apparatus 101 according to the first embodiment of FIG. Is different. Further, the other failure recovery apparatus 102 has the same configuration as the failure recovery apparatus 101 of FIG.

  When the complement unit 1116 receives an incomplete rule complement request including an incomplete fault handling rule and related environment information from the user management apparatus 201 through the control unit 1014 of the own fault recovery apparatus 101, the complement unit 1116 receives the incomplete rule. When a completion request is transmitted to another failure recovery apparatus 102 and a completion response including a completed failure handling rule and environment information related thereto is received in response to this incomplete rule completion request, the failure included in the completion response It has a function of transmitting the handling rule and the related environment information to the applying unit 1115 of the own failure recovery apparatus 101. The application unit 1115 uses the same failure handling rule and the related environment information transmitted thereto as the failure handling rule and the related environment information transmitted from the other failure recovery apparatus 102 in the first embodiment. Process as follows.

  Further, when the complementing unit 1116 receives an incomplete rule complementing request including an incomplete fault handling rule and related environment information from another fault recovery device 102, the complementing unit 1116 stores it in the handling storage unit 1013 of the own fault recovery device 101. The failure handling rule, correspondence table 1112 and environment information storage unit 1111 are referred to complete the incomplete rule, and the completed response including the completed failure handling rule and the related environment information is received as the incomplete rule. It has a function to transmit to the other failure recovery apparatus 102 of the complement request source. A specific method for completing an incomplete rule is as follows.

  First, the complementing unit 1116 extracts all failure handling rules related to the same environment information as the environment information included in the incomplete rule complementing request from the failure handling rules stored in the handling storage unit 1013. This is done by searching the correspondence table 1112 in FIG. 7 using the environment information number included in the incomplete rule complement request as a key to obtain the rule number corresponding to the environment information number, and then obtaining the rule number. This is done by searching the countermeasure storage means 1013 using as a key. Next, the complement means 1116 completes the complemented rule, that is, the completed rule in the incomplete fault handling rule included in the incomplete rule complementing request among the extracted fault handling rules. Trouble handling rules.

  Here, the incomplete failure handling rule means a failure handling rule in which at least one element of the failure handling rule is not set. All or part of the conditional expression, or all or part of the handling command, or all of the conditional expression and handling command, or part of the conditional expression and handling command, or all of the conditional expression and handling command A fault handling rule in which a part or part of a conditional expression and a part of a handling command are not set is an incomplete fault handling rule.

  FIG. 11 shows an example of a failure handling rule including an incomplete part. Rule number 1 is a failure handling rule with no unfinished part. Rule No. 2 is an incomplete failure handling rule in which a conditional expression for when the CPU usage rate is 90% or more is set, but no handling command is set. Rule number 3 is an incomplete fault handling rule in which AP restart is set as a handling command, but the fault operation of the AP output in the conditional expression is not set. Rule number 4 is an incomplete failure handling rule in which the OS output failure operation is not set and the response command is not set. Rule number 5 is an incomplete failure handling rule in which both the conditional expression and the handling command are not set.

  In addition to the functions of the user management apparatus 201 in the first embodiment, the user management apparatus 201 accepts an incomplete rule supplement request including an incomplete failure handling rule and related environment information from the administrator. And transmitting to the control means 1014.

  In addition to the function of the control unit 1014 in the first embodiment, the control unit 1014 receives, from the user management apparatus 201, an incomplete rule complement request including an incomplete fault handling rule and related environment information. , And a function of transmitting to the complementing means 1116.

  Next, the operation of the present embodiment will be described.

  Similarly to the first embodiment, in the failure recovery apparatus 101, the failure storage unit 1013 previously has a failure handling rule, the control unit 1014 has an application mode and application reference value, and the environment information storage unit 1111 has environment information of the information processing device 301. The application evaluation value table 1114 stores comparison evaluation information of environment information corresponding to the failure handling rule, and the correspondence table 1112 stores correspondence information between the failure handling rule and the environment information. The same applies to other failure recovery apparatuses 102.

  When the administrator adds a new failure handling rule that has not been completed or an edited failure handling rule to the handling storage unit 1013 of the own failure recovery device 101 through the user management device 201, the failure handling rule is added to another failure handling rule. If it is desired to propagate to the recovery device 102, a request to propagate the failure handling rule to another failure recovery device 102 is transmitted from the user management device 201 to the control means 1014 of the failure recovery device 101. The handling rule can be applied to other failure recovery apparatuses 102 as necessary. Since the operation at this time is the same as that in the first embodiment, the description thereof is omitted.

  Next, an operation when an administrator inputs an incomplete rule supplement request including an incomplete failure handling rule and environmental information related thereto from the user management apparatus 201 will be described.

  When an incomplete rule complement request including an incomplete fault handling rule and related environment information is input from the administrator, the user management apparatus 201 outputs the incomplete rule complement request to the control unit of the fault recovery apparatus 101. To 1014. Here, the administrator himself determines the environmental information related to the fault handling rule from the contents of the completed part of the incomplete fault handling rule, the purpose of use of the rule, and the intention to create the rule, and sets the number of the environmental information. Referring to the environment information storage unit 1111, it is included in the incomplete rule supplement request.

  For example, when a new type of Web server is introduced into the information processing apparatus 301, it is certain that a failure will occur if the CPU usage rate is 90% or higher. However, since an appropriate countermeasure method is not known, it is assumed that a failure handling rule number 2 in FIG. In this case, environmental information of numbers 1, 3, and 5 in FIG. 5 can be considered as environmental information related to the failure handling rule.

  Also, since a new SMTP server has been added to the information processing apparatus 301, we want to prepare a failure handling rule for recovery when it becomes a failure. Suppose that a failure handling rule of number 5 in FIG. 11 is created in which both the conditional expression and the handling command are not set. In this case, environmental information of numbers 1 and 4 in FIG. 5 can be considered as environmental information related to the failure handling rule.

  When an incomplete rule supplement request including an incomplete fault handling rule and environmental information related thereto is transmitted from the user management apparatus 201 to the control means 1014 of the fault recovery apparatus 101, the following is shown in FIG. A series of processing is started in the failure recovery apparatus 101 and another failure recovery apparatus 102.

  First, the control means 1014 of the failure recovery apparatus 101 that has received the incomplete rule complement request transmits the request to the complement means 1116 of the own apparatus 101 (S301). The complementing unit 1116 transmits the incomplete rule complementing request received from the control unit 1014 to the other fault recovery apparatus 102 (S302). In response to this incomplete rule complementing request, the completed fault handling rule and the related rule Wait for a supplemental response containing environmental information.

  When the complementing means 1116 of the other fault recovery apparatus 102 receives the incomplete rule supplement request transmitted from the fault recovery apparatus 101 (S303), first, of the fault handling rules stored in the handling storage means 1013, A failure handling rule related to the same environmental information as the environmental information included in the completion rule supplement request is extracted (S304). For example, the environmental information numbers included in the incomplete rule complement request are 1, 3, and 5, and the contents of the environmental information storage unit 1111 and the correspondence table 1112 of the other failure recovery apparatuses 102 are shown in FIGS. In this case, the failure handling rules with the rule numbers 1, 2, 3, 4, 6, 7 are extracted from the handling storage unit 1013. Here, the failure handling rules related to at least one environment information included in the incomplete rule complementation request are extracted, but the failure handling rules related to all or a plurality of environment information included in the incomplete rule complementing request are extracted. You may do it. For example, in the above example, when the failure handling rules related to all of the environment information numbers 1, 3, and 5 included in the incomplete rule supplement request are extracted, the failure handling rules of rule numbers 1 and 6 are obtained.

  Next, the complementing unit 1116 of the fault recovery apparatus 102 selects a fault handling rule that matches the content of the completed part in the incomplete fault handling rule included in the incomplete rule complementing request from the extracted fault handling rules. (S305). For example, the content of the completed part in the incomplete fault handling rule is the conditional expression “memory usage rate> 90%”, and there are two fault handling rules extracted in step S304. When the usage rate> 90% ”and the other conditional expression is“ memory usage rate> 80% ”, the former failure handling rule that matches the conditional expression is selected. In the case of the incomplete failure handling rule of number 5 shown in FIG. 11, since there is no completed part, all the failure handling rules extracted in step S304 are selected in this step S305.

  Next, the complementing unit 1116 of the failure recovery apparatus 102 acquires the number of the related environment information with reference to the correspondence table 1112 for each failure handling rule selected in step S305, and uses the acquired environment information number as a key. Environment information is retrieved from the environment information storage unit 1111 (S306). Then, the complementing unit 102 transmits a complementary response including the failure handling rule and the environment information related to the failure handling rule to the failure recovery apparatus 101 that has transmitted the incomplete rule complementing request (S307).

  When the complementary unit 1116 of the failure recovery apparatus 101 receives a complementary response from another failure recovery apparatus 102 that has transmitted the incomplete rule complementary request (S308), the failure handling rule included in the received complementary response and environment information related thereto Is transmitted to the application means 1115 of the own apparatus 101 (S309). Thereafter, application determination processing S310, which is the same processing as steps S203 to S210 in FIG. 9 in the first embodiment, is executed in the application unit 1115, the control unit 1014, and the user management apparatus 201 in the failure recovery apparatus 101. . As a result, the possibility that the failure handling rule returned from the other failure recovery device 102 as a complementary response can be applied as the failure handling rule for the information processing device 301 is evaluated. Will be registered in its own action storage means 1013 under the confirmation.

  As described above, the incomplete failure handling rule input by the administrator is completed in the other failure recovery device 102, returned to the own failure recovery device 101, and newly stored in the response storage means 1013 of the own device 101. Then, the stored failure handling rule indicates the failure that occurred in the information processing device 301 to be managed by the failure recovery device 101 or its precursor as in the case of the failure recovery device that is the premise of FIG. It will be used as a failure handling rule for recovery or recovery. Since the operation at that time is the same as that of the presumed failure recovery apparatus, description thereof is omitted.

  As described above, according to the present embodiment, the incomplete failure handling rule created by the administrator for the failure recovery apparatus 101 is completed by the failure handling rule existing in the other failure recovery apparatus 102, and then the original By constructing a mechanism for propagating to the failure recovery apparatus 101, it is possible to reduce the work for creating the failure handling rule and the additional work that have been performed by the administrator.

  For example, when considering an information communication service such as providing information to a general mobile phone, the service provider considers the change in the number of service users and the performance of the terminal to be used while continuing the service. The system will be expanded by adding (information processing equipment) or changing to a new model, but with the conventional fault recovery equipment, every time such a configuration change or function change occurs, the fault handling rule is corrected. Had to be done by the administrator. In this case, in order to describe failure detection conditions and response commands in a complete form, the administrator needs to grasp all the detailed characteristics such as the contents of the change and consistency with existing functions. If a failure related to a new failure handling rule occurs, it must be strictly tested whether the response is executed correctly. As the scale of the information processing device that executes the service increases, the administrator In some cases, the burden on the company has increased dramatically, causing secondary failures due to poor management.

  On the other hand, in the failure recovery apparatus according to the present embodiment, for example, the administrator sets uncompleted failure handling rules like the failure handling rules indicated by numbers 2 to 5 in FIG. It is possible to receive completed fault handling rules from other fault recovery apparatuses having similar setting elements and similar configurations. As a result, incomplete elements that were not set by the administrator will be complemented based on the contents that are actually set in other failure recovery devices, greatly increasing the administrator's setting and verification work. Will be reduced.

  Specifically, the failure handling rule of No. 2 in FIG. 11 is a case such as “If the CPU usage rate is 90% or more, it is certain that a failure will occur, but what an appropriate countermeasure cannot be determined”. Since a new type of server has been introduced, there may be situations where the administrator has no knowledge of appropriate handling commands. Similarly, the failure handling rule of No. 3 is an example of “I do not know what kind of error the newly installed program (AP) outputs, but I want to restart when there is an error”. If the operating system (OS) abnormality is not set, but you do not know what to set ", the number 5 is" A new server has been added, but what kind of trouble should be set to deal with it? " I don't know what to do ". These are situations that often occur in actual operational management situations, and each time a heavy burden is placed on the administrator. On the other hand, in the failure recovery apparatus according to the present embodiment, for example, when the failure handling rule of number 2 is set, a failure handling rule effective for this new server is selected from the configuration similarity, The coping command is complemented, but this coping command has been tested for its behavior on other fault recovery devices that are actually in operation. As described above, the failure recovery apparatus according to the present embodiment provides a remarkable effect that it is possible to support the setting of an effective failure handling rule while reducing the administrator's configuration grasping and testing work.

  In the present embodiment, the incomplete rule supplement request includes the environmental information related to the incomplete fault handling rule in addition to the incomplete fault handling rule, but an embodiment in which the related environmental information is omitted is also conceivable. In this case, step S304 in FIG. 12 is omitted, and in step S305, the complementing unit 1116 selects from the handling storage unit 1013 a fault handling rule that matches the completed part of the incomplete fault handling rule included in the incomplete rule complementing request. To do. As a result, the failure handling rules cannot be narrowed down by the similarity of the environment information as in the previous embodiment, and there is a tendency that the number of failure handling rules returned by being included in the complementary response tends to increase. Since it is narrowed down, the same effect as the previous embodiment can be obtained.

(Third embodiment)
Next, a third embodiment of the present invention will be described in detail with reference to FIGS.

  Referring to FIG. 13, the failure recovery apparatus 101 according to the third exemplary embodiment of the present invention has a propagation path selection unit 1117, a propagation, compared to the failure recovery apparatus 101 according to the first exemplary embodiment of FIG. The difference is that an information storage unit 1118 and a performance value storage unit 1119 are provided. Further, the other failure recovery apparatus 102 has the same configuration as the failure recovery apparatus 101 of FIG.

  The propagation information storage unit 1118 transmits the rule number and the other fault recovery device 102 that has transmitted the fault handling rule with respect to the fault handling rule propagated from the other fault recovery device 102 and stored in the handling storage unit 1013 of the own device 101. This is a means for storing a set of information.

  When storing the failure handling rule propagated from another failure recovery apparatus 102 in the action storage means 1013, the control means 1014 records the rule number and information of the failure recovery apparatus 102 of the propagation source in the propagation information storage means 1118. . Further, when the countermeasure execution unit 1015 executes the countermeasure command of the failure handling rule selected by the handling candidate search unit 1012 on the information processing apparatus 301, the execution result is sent to the control unit 1014 together with the rule number of the failure handling rule. It is configured to report. Upon receipt of this report, the control means 1014 searches the propagation information storage means 1118 with the rule number, and if the same rule number is stored, the information of the failure recovery apparatus 102 of the propagation source stored corresponding thereto is stored. Is transmitted from the propagation information storage unit 1118 and the propagation path selection unit 1117 is requested to transmit the execution result of the failure handling rule reported from the handling execution unit 1015 to the failure recovery apparatus 102 of the propagation source.

  The actual value storage unit 1119 is a unit that stores, for each other fault recovery apparatus 102, an actual value corresponding to the application result of the fault handling rule propagated from the own apparatus 101 to the other fault recovery apparatus 102. The actual value takes a larger value than the preset propagation standard value as the failure recovery rules propagated in the past are actually applied and the failure recovery and avoidance are more successful. As the number of cases that have failed, the value is updated to take a value smaller than the propagation reference value.

  The propagation path selection unit 1117 receives from the propagation unit 1113 of the own apparatus 101 the failure handling rule to be propagated to the other failure recovery apparatus 102 and the environment information related thereto, and the other stored in the actual value storage unit 1119 It has a function of referring to the actual value for each failure recovery device 102 and selecting another failure recovery device 102 that propagates the failure handling rule. Further, the propagation path selection unit 1117 receives the execution result of the propagated failure handling rule from the other failure recovery device 102 to which the failure handling rule has been propagated, and the other failure recovery device 102 in the actual value storage unit 1119. Has a function of updating the actual value.

  Further, the propagation path selection unit 1117 has a function of transmitting the execution result of the failure handling rule propagated from the other failure recovery device 102 to the other failure recovery device 102 based on the request from the control unit 1014.

  Next, the operation of the present embodiment will be described.

  Similarly to the first embodiment, in the failure recovery apparatus 101, the failure storage unit 1013 previously has a failure handling rule, the control unit 1014 has an application mode and application reference value, and the environment information storage unit 1111 has environment information of the information processing device 301. The application evaluation value table 1114 stores comparison evaluation information of environment information corresponding to the failure handling rule, and the correspondence table 1112 stores correspondence information between the failure handling rule and the environment information. Further, the propagation reference value is stored in the propagation path selection means 1117. The same applies to other failure recovery apparatuses 102.

  When the administrator adds a new failure handling rule or an edited failure handling rule to the handling storage unit 1013 of the own failure recovery device 101 through the user management device 201, the failure handling rule is transferred to another failure recovery device 102. When it is desired to propagate, a request for propagating the failure handling rule to another failure recovery device 102 is transmitted from the user management device 201 to the control means 1014 of the failure recovery device 101. This propagation request includes the rule number of the failure handling rule to be propagated. Thereafter, a series of processes as shown in FIG. 14 is started in the failure recovery apparatus 101 and other failure recovery apparatuses 102.

  First, the control unit 1014 reads out the failure handling rule having the rule number included in the propagation request from the handling storage unit 1013, and transmits it to the propagation unit 1113 of the own device 102 together with the rule number (S401). Further, the propagation unit 1113 searches the correspondence table 1112 using the rule number as a key to obtain an environment information number related to the failure handling rule to be propagated, and further searches the environment information storage unit 1111 using the environment information number as a key. Then, the environment information (index and contents) is acquired, and the failure handling rule to be propagated and the acquired environment information are transmitted to the propagation path selection means 1117 (S402).

  The propagation path selection unit 1117 compares the actual value for each of the other fault recovery devices 102 stored in the actual value storage unit 1119 with the propagation reference value, and the other fault recovery device whose actual value is larger than the propagation reference value. 102 is selected as a transmission destination (S403). Then, the failure handling rule to be propagated and the environment information related thereto are transmitted to the other failure recovery apparatus 102 selected as the propagation destination (S404). In addition, for other failure recovery apparatuses 102 that have not been selected as the propagation destination because the actual value is less than the propagation reference value, the control unit 1014 is notified that the failure handling rule has not been propagated, and the control unit 1014 notifies the fact. The administrator is notified through the user management apparatus 201 (S405).

  On the other hand, in the other fault recovery apparatus 102 that has received the fault handling rule and the environment information related thereto, an application determination process S406, which is substantially the same process as steps S203 to S210 of FIG. 9 in the first embodiment, is performed. This is executed in the application unit 1115, the control unit 1014, and the user management apparatus 201 in the failure recovery apparatus 102. As a result, the possibility that the failure handling rule propagated from the failure recovery device 101 can be applied as a failure handling rule for the information processing device managed by the failure recovery device 102 is evaluated. Under the confirmation, it is registered in the countermeasure storage unit 1013 of the failure recovery apparatus 102. The control means 1014 stores the propagated failure handling rule in the processing corresponding to steps S206 and S210 in the handling storage means 1013, and stores the rule number and the information of the propagation source failure recovery apparatus 102 in the propagation information storage. Record in means 1118.

  As described above, when the failure handling rule propagated from the failure recovery apparatus 101 is newly stored in the handling storage means 1013 of the other failure recovery apparatus 102, the stored failure handling rule is assumed to be the premise of FIG. In the same way as described in the case of the failure recovery apparatus, it is used as a failure handling rule for recovering or recovering from a failure that occurred in the information processing apparatus to be managed by the failure recovery apparatus 102 or its precursor. The operation at that time will be described below.

  When the operation state of the information processing apparatus detected by the state detection unit 1011 of the failure recovery apparatus 102 matches the conditional expression of a certain failure handling rule stored in the handling storage unit 1013, the handling candidate search unit 1012 A response command corresponding to the occurrence of the failure is searched for, and the searched response command and the rule number of the failure response rule including the response command are output to the response execution unit 1015.

  The handling execution unit 1015 executes the handling command output from the handling candidate search unit 1012 on the information processing apparatus 301 and notifies the control unit 1014 of the execution result together with the rule number of the fault handling rule. The control unit 1014 checks whether or not the notified rule number is stored in the propagation information storage unit 1118. If it is stored, the failure recovery of the propagation source recorded in the propagation information storage unit 1118 together with the rule number is stored. The device information is read, and a request for transmitting the execution result notified from the countermeasure execution unit 1015 to the failure recovery apparatus of the propagation source is transmitted to the propagation path selection unit 1117. Based on the request from the control unit 1014, the propagation path selection unit 1117 transmits the execution result of the failure handling rule to the failure recovery apparatus 101 that is the propagation source of the rule.

  When the propagation path selection unit 1117 in the failure recovery apparatus 101 of the rule propagation source receives the execution result of the propagated failure handling rule from the other failure recovery apparatus 102 that has propagated the failure handling rule, the propagation value selection unit 1117 The actual value of the failure recovery apparatus 102 is updated.

  According to the present embodiment, since the failure recovery device for propagating the failure handling rule is provided with the propagation path selection means 1117 for selecting based on the past results, some failure handling rules have been propagated in the past. It is possible to prevent the failure handling rule from being continuously propagated to other failure recovery apparatuses that are not useful at all, and to prevent the propagation efficiency and search efficiency from being lowered.

  Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various other additions and modifications can be made. Moreover, the failure recovery apparatus of the present invention can be realized by a computer and a failure recovery program as well as by realizing the functions of the failure recovery device in hardware. The failure recovery program is provided by being recorded on a computer-readable recording medium such as a magnetic disk or a semiconductor memory, read by the computer when the computer is started up, etc., and controlling the operation of the computer. In each embodiment, the failure recovery apparatus functions as state detection means, countermeasure candidate search means, countermeasure execution means, control means, propagation means, application means, complement means, and propagation path selection means.

It is a block diagram which shows the structural example of the failure recovery apparatus used as the premise of this invention. It is a figure which shows an example of the failure handling rule stored in a handling memory | storage means. It is a flowchart which shows operation | movement of the failure recovery apparatus used as the premise of this invention. 1 is a block diagram illustrating a configuration example of a first exemplary embodiment of the present invention. It is a figure which shows an example of the environmental information stored in an environmental information storage means. It is a figure which shows an example of the corresponding | compatible table which shows the correspondence of the environmental information stored in an environmental information storage means, and the failure handling rule stored in a countermeasure storage means. It is a figure which shows an example of the corresponding | compatible table which shows the correspondence of the environmental information stored in an environmental information storage means, and the failure handling rule stored in a countermeasure storage means. It is a figure which shows an example of an application evaluation value table. It is a flowchart which shows the operation | movement in the case of propagating the failure handling rule in the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the 2nd Embodiment of this invention. It is a figure which shows an example of the failure handling rule containing the incomplete part of the 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement in the case of complementing the failure handling rule containing an incomplete part in the 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the 3rd Embodiment of this invention. It is a flowchart which shows operation | movement in the case of propagating a failure handling rule in the 3rd Embodiment of this invention.

Explanation of symbols

101, 102 ... Failure recovery device 201 ... User management device 301 ... Information processing device 1011 ... Status detection means 1012 ... Action candidate search means 1013 ... Action storage means 1014 ... Control means 1015 ... Action execution means 1111 ... Environment information storage means 1112 ... correspondence table 1113 ... propagation means 1114 ... application evaluation value table 1115 ... application means 1116 ... complement means 1117 ... propagation path selection means 1118 ... propagation information storage means 1119 ... performance value storage means

Claims (23)

a) The failure recovery device applying means includes a failure handling rule from another failure recovery device, and the failure of the environment information indicating the hardware configuration and software configuration of the information processing device to be managed by the other failure recovery device Environment information storage means for receiving environment information related to a handling rule, and storing the received environment information and environment information indicating a hardware configuration and a software configuration of an information processing apparatus to be managed by the own failure recovery apparatus The failure handling rule received from the other failure recovery device can be applied as a failure handling rule for the information processing device to be managed by the own failure recovery device. Calculating an application evaluation value that evaluates the sex;
b) a step in which the control means of the failure recovery apparatus stores the received failure handling rule in the handling storage means of the own failure recovery apparatus when the calculated application evaluation value is equal to or greater than an application reference value;
A fault handling rule propagation method characterized by comprising: a) The propagation means of the first failure recovery apparatus is configured to determine the failure handling rules stored in the action storage means of the own failure recovery apparatus, and the hardware configuration and software configuration of the information processing apparatus to be managed by the own failure recovery apparatus. Transmitting environmental information related to the failure handling rule among the environmental information stored in the environmental information storage means for storing the environmental information to be indicated to one or more second failure recovery devices;
b) The application unit of the second failure recovery device indicates the environment information received from the first failure recovery device, and the hardware configuration and software configuration of the information processing device to be managed by the own failure recovery device The information processing apparatus which is compared with the environment information stored in the environment information storage means for storing environment information, and the failure handling rule received from the first failure recovery apparatus is a management target of the own failure recovery apparatus Calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for
c) the control means of the second failure recovery apparatus storing the received failure handling rule in the action storage means of the own failure recovery apparatus when the calculated application evaluation value is equal to or greater than an application reference value;
A fault handling rule propagation method characterized by comprising: a) The propagation means of the first failure recovery apparatus is configured to determine the failure handling rules stored in the action storage means of the own failure recovery apparatus, and the hardware configuration and software configuration of the information processing apparatus to be managed by the own failure recovery apparatus. Transmitting the environmental information related to the failure handling rule among the environmental information stored in the environmental information storage means for storing the environmental information to be shown to the propagation path selection means of the own fault recovery device;
b) The propagation path selection means of the first failure recovery apparatus receives the failure handling rule and the environment information related to the received failure handling rule and the second value of which the actual value stored in the actual value storage means is greater than or equal to a reference value. Sending to the disaster recovery device;
c) The application unit of the second failure recovery apparatus shows the environment information received from the first failure recovery apparatus, and the hardware configuration and software configuration of the information processing apparatus to be managed by the own failure recovery apparatus The information processing apparatus which is compared with the environment information stored in the environment information storage means for storing environment information, and the failure handling rule received from the first failure recovery apparatus is a management target of the own failure recovery apparatus Calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for
d) a step in which the control means of the second failure recovery apparatus stores the received failure handling rule in the action storage means of the own failure recovery apparatus when the calculated application evaluation value is greater than or equal to an application reference value;
e) The propagation path selection means of the second failure recovery apparatus uses the failure handling rule received from the first failure recovery apparatus stored in the action storage means of the own failure recovery apparatus as the management target of the own failure recovery apparatus. Transmitting the result applied to the information processing apparatus to the first failure recovery apparatus;
f) The propagation path selection means of the first failure recovery device updates the actual value storage means according to the application result of the failure handling rule received from the second failure recovery device;
A fault handling rule propagation method characterized by comprising: a) The complementary means of the first failure recovery device shows the incomplete failure handling rule input from the user management device and the hardware configuration and software configuration of the information processing device to be managed by the own failure recovery device A step of transmitting an incomplete rule supplement request including environmental information related to the failure handling rule out of the environmental information stored in the environmental information storage means for storing environmental information to the second fault recovery device;
b) The environment stored in the environment information storage means for storing the environment information indicating the hardware configuration and software configuration of the information processing apparatus to be managed by the own fault recovery apparatus as the complement means of the second fault recovery apparatus With reference to the correspondence table showing the correspondence between the information and the failure handling rules stored in the handling storage unit of the own failure recovery apparatus, the environment information storage unit, and the handling storage unit, the information is stored in the handling storage unit Extracting a fault handling rule related to the same environmental information as the environmental information included in the incomplete rule supplement request among the fault handling rules being
c) The complementing means of the second fault recovery device matches the contents of the completed part in the incomplete fault handling rule included in the incomplete rule complementing request among the extracted fault handling rules. Selecting the environment information related to the selected failure handling rule with reference to the correspondence table and the environment information storage means, and providing a complementary response including the acquired environment information and the selected failure handling rule Transmitting to the first failure recovery device;
d) The hardware configuration of the information processing apparatus to be managed by the environmental information included in the complementary response received from the second fault recovery apparatus by the application means of the first fault recovery apparatus and the own fault recovery apparatus And the environmental information stored in the environmental information storage means for storing the environmental information indicating the software configuration, and the failure handling rule included in the complementary response received from the second failure recovery device Calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for the information processing device to be managed by the recovery device;
e) the control means of the first failure recovery apparatus storing the received failure handling rule in the action storage means of the own failure recovery apparatus when the calculated application evaluation value is greater than or equal to an application reference value;
A fault handling rule propagation method characterized by comprising: a) a step of transmitting an incomplete rule complement request including an incomplete fault handling rule input from a user management device to a second fault recovery device by a complement means of the first fault recovery device;
b) The complement means of the second fault recovery device is one of the incomplete fault handling rules included in the incomplete rule complement request among the fault handling rules stored in the handling storage means of the own fault recovery device. The environmental information stored in the environmental information storage means for selecting environmental information that indicates the hardware configuration and software configuration of the information processing apparatus to be managed by the self-failure recovery apparatus, and selects the one with the same content of the completed part Refer to the correspondence table showing the correspondence with the failure handling rules stored in the handling storage means of the own failure recovery device, and obtain the environment information related to the selected failure handling rules with reference to the environment information storage means, Transmitting a complementary response including the acquired environmental information and the selected failure handling rule to the first failure recovery device;
c) The hardware configuration of the environment information included in the complementary response received from the second failure recovery device by the application unit of the first failure recovery device and the information processing device to be managed by the own failure recovery device And the environmental information stored in the environmental information storage means for storing the environmental information indicating the software configuration, and the failure handling rule included in the complementary response received from the second failure recovery device Calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for the information processing device to be managed by the recovery device;
d) a step in which the control means of the first failure recovery apparatus stores the received failure handling rule in the action storage means of the own failure recovery apparatus when the calculated application evaluation value is greater than or equal to an application reference value;
A fault handling rule propagation method characterized by comprising:   When the calculated application evaluation value is greater than or equal to an application reference value, the control unit performs the process of storing the received failure handling rule in the handling storage unit of the own failure recovery apparatus, instead of performing the process of calculating the calculated application evaluation value. Value and evaluation target failure handling rule and related environment information are displayed on the user management device, and when there is a storage instruction from the user management device, the failure handling rule is stored in the handling memory of the own failure recovery device 6. The fault handling rule propagation method according to any one of claims 1 to 5, wherein a process of storing in the means is performed.   The application means determines the application evaluation value with reference to an application evaluation value table that defines application evaluation values corresponding to patterns of types of matching environmental information. The fault handling rule propagation method according to item 1.   Handling storage means for storing a troubleshooting rule including a conditional expression and a handling command, a status detecting means for detecting the status of the information processing apparatus to be managed, and a fault handling having a conditional expression that matches the detected status Handling candidate search means for selecting a rule from the handling storage means, handling execution means for executing a handling command for the selected failure handling rule on the information processing apparatus, and hardware configuration and software configuration of the information processing apparatus Environment information storage means for storing environment information indicating the failure, the failure handling rule transmitted from another failure recovery apparatus and the environment information related to the failure handling rule, the received environment information and the environment information storage means And the propagated failure handling rule as a failure handling rule for the information processing device An application unit that calculates an application evaluation value that evaluates the possibility of use, and a control unit that stores the propagated failure handling rule in the handling storage unit when the calculated application evaluation value is equal to or greater than an application reference value. A failure recovery apparatus comprising:   Correspondence table holding correspondence between the failure handling rules stored in the handling storage means and the environment information stored in the environment information storage means, and other fault recovery among the failure handling rules stored in the handling storage means Propagation means for acquiring environment information related to a failure handling rule to be propagated to a device from the environment information storage means with reference to the correspondence table, and transmitting to the other failure recovery device together with the failure handling rule to be propagated The failure recovery apparatus according to claim 8, wherein:   Correspondence table holding correspondence between the failure handling rules stored in the handling storage means and the environment information stored in the environment information storage means, and other fault recovery among the failure handling rules stored in the handling storage means Environment information related to failure handling rules to be propagated to the device is acquired from the environment information storage means with reference to the correspondence table and transmitted together with the failure handling rules to be propagated, and actual values for each other failure recovery device The actual value storage means for holding the failure handling rule received from the propagation means and the environment information related thereto are transmitted to other failure recovery devices whose actual value stored in the actual value storage means is a reference value or more. And a propagation path selection means for updating the actual value storage means according to the application result of the failure handling rule received from the other failure recovery apparatus at the transmission destination. Failure recovery device according to claim 8, wherein that.   Receives an incomplete rule supplement request including an incomplete failure handling rule input from the user management device and environmental information related to it and transmits it to another failure recovery device, and the incompleteness from the other failure recovery device When a completed response including a completed failure handling rule and related environment information is received as a response to the rule complement request, the failure handling rule included in the complement response and the associated environment information are transmitted to the application unit. The failure recovery apparatus according to claim 8, further comprising a complementing unit.   The complement means indicates correspondence between the environment information stored in the environment information storage means and the fault handling rules stored in the handling storage means when receiving an incomplete rule complement request from another fault recovery device Referring to the correspondence table, the environment information storage means, and the countermeasure storage means, the same environment as the environment information included in the incomplete rule complement request among the fault handling rules stored in the countermeasure storage means Extract failure handling rules related to information, and select from the extracted failure handling rules that match the contents of the completed part in the incomplete failure handling rules included in the incomplete rule supplement request In addition, referring to the correspondence table and the environment information storage unit, environment information related to the selected failure handling rule is acquired, and the acquired environment information and the selected failure handling route are acquired. DOO failure recovery system of claim 11, wherein the complementary response is to send the unfinished rules complement requesting disaster recovery system including.   When the complement means receives an incomplete rule complement request from another fault recovery device, the incomplete fault handling included in the incomplete rule complement request is selected from the fault handling rules stored in the handling storage means. A correspondence table indicating correspondence between the environment information stored in the environment information storage unit and the failure handling rule stored in the handling storage unit; The environment information related to the selected failure handling rule is acquired with reference to the information storage means, and a complementary response including the acquired environment information and the selected failure handling rule is used as the failure recovery of the incomplete rule complement request source The failure recovery apparatus according to claim 11, wherein the failure recovery apparatus transmits to the apparatus.   When the calculated application evaluation value is greater than or equal to an application reference value, the control unit performs the process of storing the received failure handling rule in the handling storage unit, instead of performing the process of storing the calculated application evaluation value and the evaluation value. The target failure handling rule and the environment information related to it are displayed on the user management device, and when there is a storage instruction from the user management device, the failure handling rule is stored in the handling storage means The failure recovery apparatus according to any one of claims 8 to 13.   The application means determines the application evaluation value with reference to an application evaluation value table that defines application evaluation values corresponding to patterns of types of matching environmental information. The failure recovery apparatus according to item 1.   A failure recovery apparatus having a handling storage means for storing a handling rule including a conditional expression and a handling command, and an environment information storage means for storing environment information indicating a hardware configuration and a software configuration of an information processing apparatus to be managed A state detection unit that detects a state of an information processing apparatus to be managed, a countermeasure candidate search unit that selects a failure handling rule having a conditional expression that matches the detected state from the handling storage unit, Receiving a countermeasure execution means for executing a countermeasure command of the selected failure countermeasure rule on the information processing device, a failure countermeasure rule transmitted from another failure recovery device, and environment information related to the failure countermeasure rule; Comparing the received environmental information with the environmental information stored in the environmental information storage means, Applying means for calculating an application evaluation value that evaluates the possibility of being applied as a failure handling rule for the information processing apparatus, and the propagated failure handling rule when the calculated application evaluation value is greater than or equal to an application reference value. A program that functions as a control unit that is stored in the countermeasure storage unit.   Environment information related to a failure handling rule for causing the computer to propagate to another failure recovery device among failure handling rules stored in the handling storage unit, and a failure handling rule stored in the handling storage unit and the environment Refers to the correspondence table holding correspondence with the environment information stored in the information storage means, acquires from the environment information storage means, and functions as a propagation means for transmitting to other failure recovery devices together with the failure handling rule to be propagated The program according to claim 16.   Environment information related to a failure handling rule for causing the computer to propagate to another failure recovery device among failure handling rules stored in the handling storage unit, and a failure handling rule stored in the handling storage unit and the environment Propagation means for acquiring from the environment information storage means with reference to a correspondence table holding correspondence with the environment information stored in the information storage means and transmitting together with the failure handling rules to be propagated, and the failure handling rules received from the propagation means And the environmental information related thereto are transmitted to other failure recovery devices whose actual value stored in the actual value storage means for holding the actual value for each other failure recovery device is greater than or equal to the reference value, And functioning as a propagation path selection unit that updates the actual value storage unit according to the application result of the failure handling rule received from another failure recovery device, According to claim 16 of the program that.   The computer further receives an incomplete rule supplement request including an incomplete fault handling rule input from the user management apparatus and environmental information related to the received incomplete fault rule, and transmits the request to another fault recovery apparatus. When a complementary response including a completed fault handling rule and its related environment information is received as a response to the incomplete rule complement request from the device, the fault handling rule and its related environment information included in the complementary response are 17. The program according to claim 16, wherein the program is made to function as a complement means for transmitting to the application means.   The complement means indicates correspondence between the environment information stored in the environment information storage means and the fault handling rules stored in the handling storage means when receiving an incomplete rule complement request from another fault recovery device Referring to the correspondence table, the environment information storage means, and the countermeasure storage means, the same environment as the environment information included in the incomplete rule complement request among the fault handling rules stored in the countermeasure storage means Extract failure handling rules related to information, and select from the extracted failure handling rules that match the contents of the completed part in the incomplete failure handling rules included in the incomplete rule supplement request In addition, referring to the correspondence table and the environment information storage unit, environment information related to the selected failure handling rule is acquired, and the acquired environment information and the selected failure handling route are acquired. Claim 19 of the program, characterized in that for transmitting bets as incomplete rule complement requesting disaster recovery apparatus complementary response including.   When the complement means receives an incomplete rule complement request from another fault recovery device, the incomplete fault handling included in the incomplete rule complement request is selected from the fault handling rules stored in the handling storage means. A correspondence table indicating correspondence between the environment information stored in the environment information storage unit and the failure handling rule stored in the handling storage unit; The environment information related to the selected failure handling rule is acquired with reference to the information storage means, and a complementary response including the acquired environment information and the selected failure handling rule is used as the failure recovery of the incomplete rule complement request source The program according to claim 19, wherein the program is transmitted to a device.   When the calculated application evaluation value is greater than or equal to an application reference value, the control unit performs the process of storing the received failure handling rule in the handling storage unit, instead of performing the process of storing the calculated application evaluation value and the evaluation value. The target failure handling rule and the environment information related to it are displayed on the user management device, and when there is a storage instruction from the user management device, the failure handling rule is stored in the handling storage means The program according to any one of claims 16 to 21, wherein:   The application means determines the application evaluation value with reference to an application evaluation value table that defines application evaluation values corresponding to patterns of the types of matching environmental information. The program according to item 1.

Images