JP4313823B2 - Fault response system and fault response method - Google Patents

Description

  The present invention relates to a failure handling system and a failure handling method for handling a failure that has occurred in a monitored server.

  Conventionally, when a server goes down due to a failure, a person in charge for failure recovery analyzes the cause of the failure and performs a recovery procedure according to the failure by trial and error.

  When a failure occurs in a server, a failure information notification program (Patent Document 1) is proposed that makes it possible to schedule an expected response work and notify each person in charge of the failure content and the estimated work start time of each person in charge. Has been.

  The failure information notification program described in Patent Literature 1 refers to a recovery information table storing a person in charge and the contents of countermeasures, and extracts a person in charge and details of countermeasures related to an event that has occurred in the monitored server. And a first scheduling step for generating a countermeasure schedule based on the countermeasure contents extracted in the first extraction step, and a first notification step for notifying the person in charge of the countermeasure schedule. Yes.

As a result, since the appropriate person is called at the right time, it takes time for the two-stage call that the inappropriate person is called and the called person calls the appropriate person. It is said that it can solve the problem of insufficient communication of fault information. In addition, it is said that each person in charge can come too early and solve the problem of wasting time until his work order comes.
JP 2004-280171 A

  However, since the schedule table generated by the failure information notification program described in Patent Document 1 is not generated in time for the deadline, there is a possibility that it will not be in time for the deadline even if recovery work is performed as scheduled. It was.

  In addition, since the failure information notification program is performed without considering the recovery rate per unit time when selecting and ordering the response procedures, the recovery rate is not necessarily high even if working according to the schedule, and the time until recovery There was also a problem that it was not a short time.

  The present invention has been made in view of such circumstances, and a failure response system and method for generating a schedule so as to meet the deadline and have a high recovery rate and can be recovered in a short time. Is to provide.

The problems of the present invention can be solved by the following inventions.
That is, the failure response system of the present invention selects a response procedure for recovering the failure when a failure occurs in the monitored server, and generates a schedule table indicating the order in which the response procedures are tried. A fault information database in which information about faults is stored, a fault procedure that is a type of fault that has occurred in the monitored server, a response procedure, a time required for the response procedure, Recovery procedure information database that stores information about whether or not the recovery procedure has been recovered, failure events that occurred in the past, the response procedure that was performed to recover from the failure, and the requirements required for the response procedure A case accumulation database in which the time and the presence or absence of recovery in the corresponding procedure are stored, and the monitoring target server transmitted from the monitoring means built in the monitoring target server Based on the information transmitted from the monitoring means, the fault information receiving means for receiving the information on the fault, the deadline input receiving means for receiving the input of the deadline that is the deadline for which the fault must be recovered, By searching the failure information database, error information analysis means for determining the failure event that has occurred in the monitored server, the recovery procedure information database is searched based on the failure event, the corresponding procedure is read, and the deadline input Schedule analysis generation means for generating a schedule table in which the corresponding procedure is scheduled so that failure recovery is in time for the deadline input from the reception means, and the contents of the case accumulation database each time a failure occurs in the monitored server And the contents of the recovery procedure information database The corresponding history monitoring means new to, further comprising: a is mainly characterized.

  As a result, the schedule is generated in time for the deadline, so that the monitored server can be recovered from the failure in time for the deadline by performing the recovery operation according to the schedule.

  In addition, since the contents of the recovery procedure information database are updated each time a failure occurs, the time required for the response procedure and the recovery rate information are also updated, and a highly accurate schedule is generated when the next schedule is generated. can do.

  Further, in the failure handling system according to the present invention, the schedule analysis / generation unit is required for a restoration rate that is stored in the recovery procedure information database and is a probability of restoration for each handling procedure, and a handling procedure having the restoration rate. The recovery rate per unit time is calculated from the required time, the corresponding procedure is selected in order of the recovery rate per unit time, and the selection is selected at the date and time when the corresponding procedure is scheduled to be performed. Add the time required for the response procedure, perform until the date and time after the addition reaches the deadline, and generate a schedule so that the selected response procedure is executed in order of the recovery rate per unit time. This is the main feature.

  As a result, a schedule is generated so that it can be executed from a procedure with a high recovery rate per unit time, so that recovery work can be carried out efficiently and recovery can be performed with high probability in a short time. be able to.

  Furthermore, in the failure handling system according to the present invention, the schedule analysis generation unit regenerates the schedule table at least once after the schedule table is first generated and before the failure of the monitored server is recovered. When the schedule is regenerated, the main feature is that the schedule is regenerated based on the date and time when the schedule is regenerated, the deadline, and the contents of the recovery procedure information database.

  As a result, even if the implementation of the response procedure ends earlier than scheduled or ends later, a new schedule is generated again accordingly, so that the accuracy of the schedule can always be kept high.

  As described above, according to the failure response system of the present invention, the schedule is generated in time for the deadline, so that the monitoring server can be recovered in time for the deadline by working according to the schedule. Can do.

  In addition, according to the failure handling system of the present invention, the contents of the recovery procedure information database are updated every time a failure occurs, so the time required for the handling procedure and information on the recovery rate are also updated to generate the next schedule In doing so, a more accurate schedule can be generated.

  In addition, since the schedule is generated so that it can be executed from the procedure with a high recovery rate per unit time, the recovery work can be carried out efficiently, and it can be recovered with high probability in a short time. Can do.

  Furthermore, since the schedule is generated once or more after the schedule is generated based on the latest information at that time, the accuracy of the schedule can always be kept high.

Hereinafter, an embodiment of a failure handling system of the present invention will be described in detail with reference to the accompanying drawings.
<Configuration>
FIG. 1 is a configuration diagram of a failure handling system according to an embodiment of the present invention. As illustrated in FIG. 1, the failure handling system includes a monitoring target server 10, a client 20, a network 30, and a monitoring server 40. The monitoring target server 10, the client 20, and the monitoring server 40 are connected to each other via the network 30.

The monitoring target server 10 has a monitoring program 11. The monitoring program 11 monitors the monitoring target server 10, and when a failure occurs in the monitoring target server 10, detects the failure and transmits information regarding the failure to the monitoring server 40 via the network 30. Here, the monitoring program 11 is not necessarily a program, and may be a device that performs the same function. Moreover, the number of monitoring target servers is not limited to one, and there may be a plurality of servers.
The number of clients 20 is not limited to one, and there may be a plurality of clients.

  The monitoring server 40 includes an error information analysis unit 41, a schedule analysis / generation unit 42, a response history management unit 43, an information database 44, a recovery procedure information database 45, and a case accumulation database 46.

  The error information analysis means 41 analyzes the error information transmitted from the monitoring program 11 and investigates which fault corresponds. Specifically, the failure information database 44 in which the failure event and the error number and alert number associated therewith are stored is searched to determine the failure event corresponding to the error information.

  The schedule analysis / generation unit 42 reads all the recovery methods related to the failure event determined by the error information analysis unit 41 from the recovery procedure information database 45. The recovery procedure information database 45 stores information related to the failure event, such as a response procedure for the failure recovery, a time required for the procedure, and whether or not the procedure has been recovered.

  Next, when a deadline that is a time limit for recovering from a failure is input from the client 20 to the monitoring server 40 via the network 30, it is read from the recovery procedure information database 45 in time for the deadline. By combining the recovery methods, a schedule showing the recovery procedure is generated. The deadline can also be input directly to the monitoring server 40.

  If no deadline has been input, a recovery method is selected from the plurality of recovery methods read from the recovery procedure information database 45 in descending order of recovery probability per unit time, and scheduling is performed to generate a schedule. Here, the scheduling method is not limited to scheduling in the order of high recovery probability per unit time, and various methods can be applied. For example, as a whole, scheduling can be performed by combining the recovery methods so that the recovery probability becomes the highest, or the recovery speed can be more important than the recovery probability.

  The response history management unit 43 is based on information sent from the monitoring program 11 every time a failure occurs in the monitored server 10 and information input by the person in charge of failure recovery to the monitoring server directly or via the network. Thus, the failure case information is stored in the case accumulation database 46. The failure case information includes a failure occurrence date and time, a server in which a failure has occurred, a failure event, a recovery method performed for recovery from the failure, a recovery procedure, and the presence or absence of recovery.

  Further, the response history management unit 43 updates the contents of the recovery procedure information database 45 based on the contents of the case accumulation database. As a result, the number of response procedures for recovery can be increased, and the accuracy of the required time for each response procedure can be increased.

  FIG. 2 is a diagram illustrating an example of information stored in the failure information database 44. As shown in FIG. 2, when a failure occurs, the error number or alert number issued by the server where the failure has occurred and the failure event are stored in association with each other. As a result, the error information analysis unit 41 can search for and determine the failure events corresponding to the error numbers and alert numbers included in the failure information received from the monitoring program 11.

  FIG. 3 is a diagram illustrating an example of information stored in the recovery procedure information database 45. As shown in FIG. 3, the response procedure ID, the content representing the failure recovery method indicated by the response procedure ID, the time required to implement the content, and the probability that the system will recover when the content is implemented Is stored in association with the failure event ID. In association with the failure event ID, a response procedure ID, a procedure indicating a more detailed work procedure of the response procedure, and a step ID which is a number assigned to each of the detailed work procedures are also stored.

  Thereby, the schedule analysis / generation unit 42 can search and select a failure recovery method corresponding to the failure event, and further generate a schedule table in consideration of the time required for recovery and the recovery rate. Can do.

  FIG. 4 is a diagram illustrating an example of information stored in the case accumulation database 46. As shown in FIG. 4, the date and time when the failure occurred, the failure server indicating the failed server, the failure event, the response procedure ID corresponding to the failure, and the procedure indicated by the response procedure ID Is stored in association with the required time required when the operation is performed and the presence / absence of recovery indicating whether the recovery is successful or not.

  Based on the contents stored in the case accumulation database 46, the response history management means 43 can update the contents of the recovery procedure information database 45. As a result, every time a failure occurs, data is accumulated in the case accumulation database 46 and the recovery procedure information database 45, and the accuracy of the schedule table generated by the schedule analysis / generation unit 42 is also high.

  FIG. 5 is a diagram showing an example of a schedule table generated by the schedule analysis / generation unit 42. As shown in FIG. 5, the failure recovery method and the countermeasure procedure indicating the order of execution, and the scheduled start time and the planned end time indicating the scheduled start time and the planned end time when each failure recovery method indicated in the countermeasure procedure is performed. In association with the status indicating the start status of each countermeasure procedure, the implementation schedule of each countermeasure procedure is represented by a Gantt chart.

  The primary deadline shown in FIG. 5 is a deadline input from the client, and the schedule analysis / generation unit 42 executes the procedure from the recovery procedure information database 45 so that the recovery from the failure is completed by the primary deadline. Select and order. The deadline shown in FIG. 5 represents the expected recovery date and time when restoration by backup is performed in the case where recovery is not performed even by the procedure performed up to the primary deadline.

<Operation>
Next, the operation of the failure handling system of this embodiment will be described with reference to FIG.
When a failure occurs in the monitoring target server 10, the monitoring program 11 detects the failure and transmits failure information related to the failure such as an error number and an alert number issued by the failed server via the network 30. 40.

  The error information analysis unit 41 that has received the failure information searches the failure information database 44 and determines a failure event corresponding to the failure that occurred in the monitoring target server based on the received failure information. Specifically, the corresponding failure event is searched and selected based on the error number and alert number included in the failure information.

  The failure event determined in this way is passed to the schedule analysis / generation unit 42. The schedule analysis / generation unit 42 searches the recovery procedure information database 45 and selects a corresponding procedure ID corresponding to the passed failure event. At this time, if a deadline is input from the client 20, the schedule analysis / generation unit 42 sets this deadline as a primary deadline, selects a corresponding procedure ID that can be executed by the primary deadline, The execution order of the selected corresponding procedure ID is determined.

  Various methods can be considered for selecting the corresponding procedure ID and determining the execution order. For example, as shown in FIG. 3, by using the required time and recovery rate data for each corresponding procedure ID stored in the recovery procedure information database 45, by dividing the recovery rate by the required time, Find the recovery rate. Next, select the one with a high recovery rate per unit time, and end the selection by adding the time required for the corresponding procedure ID selected from the scheduled restoration work start date and time. Until just before reaching the line. The order of the selected corresponding procedure IDs can be arranged in descending order of the recovery rate per unit time.

  As a result, it is possible to make a plan to perform recovery work from a recovery method with a high recovery rate per unit time, so that the probability of completing the recovery by the primary deadline is increased, and the planned schedule is reduced in a short time. It can be efficient to end the recovery.

  Next, the schedule analysis / generation unit 42 generates a schedule based on the corresponding procedure IDs selected and arranged. As shown in FIG. 5, the schedule can be a Gantt chart in which the start schedule, the end schedule, and the Status are described for each countermeasure procedure, and the schedule up to the primary deadline and the deadline is represented by lines. As a result, the countermeasure procedure and schedule can be grasped at a glance, thereby reducing the burden on the person in charge of restoration and enabling efficient restoration work.

  In addition, the schedule analysis / generation unit 42 can update the schedule table for each progress of the countermeasure procedure or periodically. When the schedule is updated for each progress of the countermeasure procedure, when one of the countermeasure procedures is executed and a result is obtained, the schedule is generated again from that point based on the result. For example, if the result is obtained earlier than the scheduled end and it still does not recover, there is a time before the primary deadline, so it may be possible to add a countermeasure procedure by recreating the schedule.

  Also, if the result comes later than the scheduled end and has not yet been restored, there is no time until the primary deadline from the initial schedule, so the countermeasure procedure may be deleted from the schedule.

When the schedule is regularly updated, the timing for updating the schedule is only periodic, and the content is the same as the update for each progress of the countermeasure procedure described above.
Thus, since the schedule is updated according to the situation, the accuracy of the schedule can always be kept high.

  As shown in FIG. 4, each time a failure occurs, the response history management unit 43 generates a failure occurrence date, failure server, failure event, response procedure ID used for recovery, required time for each response procedure ID, and recovery time. Information about presence / absence is stored in the case accumulation database 46.

  In addition, the response history management unit 43 updates the recovery procedure information database 45 based on the contents stored in the case accumulation database 46. Thereby, since the data of the recovery procedure information database 45 increases, the accuracy of schedule generation is improved.

Next, further description will be given based on the flowchart.
FIG. 6 is a flowchart showing a processing flow in the failure handling system of the present invention.
When a failure occurs in the monitoring target server 10, the monitoring program 11 detects the failure and transmits failure information to the monitoring server 40. The error information analysis means 41 that has received the transmitted failure information analyzes the error information and searches the failure information database 44 to determine which failure event corresponds (S1).

  Next, the schedule analysis / generation unit 42 reads the recovery procedure corresponding to the determined failure event from the recovery procedure information database 45 (S2). Subsequently, the schedule analysis / generation unit 42 checks whether there is an input of deadline information (S3). If there is an input, the schedule analysis / generation unit 42 generates a primary deadline from the deadline information (S4). A schedule is generated so as to be within the deadline (S5). If no deadline information is input, a schedule is generated (S5).

  Next, the schedule generation (S5) will be described in detail with reference to FIG. FIG. 7 is a flowchart showing a schedule generation flow.

  The schedule analysis generation means 42 confirms whether or not a primary deadline has been generated (S11). When the primary deadline has not been generated, all the recovery procedures are set as scheduling targets (S13), and a schedule table for executing the recovery procedures in order of high recovery rate per unit time is generated (S17).

  When the primary deadline has been generated, it is determined whether or not the work excluding the corresponding procedure ID 99 can be performed by the primary deadline (S12). Here, the handling procedure ID 99 is the handling procedure ID 99 shown in FIG. 3 and indicates that the system is restored from a backup file such as a tape. When the system is restored from the backup file, the system state returns to the time when the backup file was generated, so the data after the backup file was created cannot be restored. Therefore, this is a system recovery method on the premise of sacrificing data after the generation of the backup file, which is the last means.

  Returning to FIG. 7, if it is determined in step S12 that the work except the corresponding procedure ID 99 cannot be performed before the primary deadline, only the peer procedure ID 99 is set as a scheduling target (S14), and a schedule is generated (S14). S17).

  In step S12, when it is determined that the work excluding the corresponding procedure ID 99 can be performed by the primary deadline, the recovery rate per unit time of each recovery procedure is calculated (S15), and recovery that can be performed by the primary deadline. A combination that maximizes the recovery probability is selected from among the combinations of procedures (S16).

Here, various methods are conceivable for selecting a combination that maximizes the recovery probability. For example, a combination pattern of recovery means that can be executed up to the primary deadline is extracted, and a combination that maximizes the total recovery rate in each pattern is selected. At this time, it examines in order from a combination including a procedure with a high recovery rate per unit time, and if a combination with a recovery rate exceeding 100% is found, the calculation is terminated there. In this way, the combination that maximizes the recovery probability can be selected. In addition, various methods can be applied.
Subsequently, a schedule for executing the selected restoration procedure in order of the restoration rate per unit time is generated (S17).

  Next, updating of the case accumulation database 46 and the recovery procedure information database 45 will be described with reference to FIG. FIG. 8 is a flowchart showing an update flow of the case accumulation database 46 and the recovery procedure information database 45.

  The recovery procedure described in the generated schedule is executed in the order described (S21). The execution of the procedure can be performed by a person in charge of the recovery work or can be performed by the monitoring server 40.

  Next, the correspondence history management unit 43 confirms whether or not it has been restored (S22). Information for this confirmation can be obtained directly from the monitoring program 11 by the response history management means 43, or can be obtained by a person in charge of recovery work who has input to the monitoring server via the network. .

If not recovered, the response history management unit 43 saves the time required for executing the procedure and the fact that the procedure has not been recovered in the case accumulation database 46 (S23).
Next, a schedule is generated again from the current time and the primary deadline (S24), and it is confirmed whether there is a procedure that can be executed before the primary deadline (S25).

  If there is an executable procedure, the procedure is executed again from S21. If there is no executable procedure, restoration is performed using the backup data of the corresponding procedure ID 99 (S26), and the time required for restoration, the fact that the restoration has been performed, and the like are stored in the case accumulation database 46. This restoration operation can also be performed by the person in charge of the recovery operation, or can be performed by the monitoring server 40. Thereafter, the required time and recovery rate of the recovery procedure information database 45 are updated (S28).

  When it is confirmed in step S22 that the recovery has been made, the response history management means 43 stores the required time for executing the procedure and the fact that it has been recovered in the case accumulation database 46 (S27), and the required time in the recovery procedure information database 45 The recovery rate is updated (S28).

  The present invention can be modified in various ways within the scope of the technical idea of the present invention. For example, the operation performed by the failure handling system of the present invention can also be performed by the CPU of the computer system reading and executing a program on the storage device, and the invention is intended to be solved by obtaining exactly the same effects. The problem to be solved can be solved.

  The failure handling system of the present invention is realized by an OS, application, database, program, etc. built on hardware resources including a computer CPU, memory, storage device, display, input / output device, etc. The information processing procedure for creating a schedule that is scheduled in time for the deadline is specifically realized using the above hardware resources. It falls under the technical idea used and can be used as a system for recovering when the system goes down anywhere in the field where the computer system is used.

It is a block diagram of the failure response system which concerns on one Embodiment of this invention. It is the figure which showed the example of the information preserve | saved in the failure information database. It is the figure which showed the example of the information preserve | saved in the recovery procedure information database. It is the figure which showed the example of the information preserve | saved in the case accumulation | storage database. It is the figure which showed the example of the schedule table which the schedule analysis and production | generation means produced | generated. It is a flowchart showing the processing flow in the failure response system of this invention. It is a flowchart showing the production | generation flow of a schedule. It is a flowchart showing the update flow of the case accumulation database and the recovery procedure information database.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Monitoring object server 11 Monitoring object program 20 Client 30 Network 40 Monitoring server 41 Error information analysis means 42 Schedule analysis / generation means 43 Correspondence history management means 44 Failure information database 45 Recovery procedure information database 46 Case accumulation database

Claims (9)

When a failure occurs in a monitored server, a failure response system that selects a response procedure for recovering the failure and generates a schedule indicating the order in which the response procedure is tried,
A failure information database in which information about failures is stored;
A recovery procedure in which a response procedure, a time required for the response procedure, and information on whether or not a recovery has been performed in the response procedure are stored in association with a failure event that is the type of failure that has occurred in the monitored server An information database;
A case accumulation database that stores fault events that occurred in the past, the response procedures that were taken to recover from the failures, the time required for the response procedures, and whether or not there was recovery in the response procedures;
A failure information receiving means for receiving information on a failure of the monitored server transmitted from the monitoring means built in the monitored server;
Deadline input acceptance means for accepting deadline input that is the deadline for which the failure must be recovered;
Based on the information transmitted from the monitoring means, by searching the failure information database, error information analysis means for determining a failure event that has occurred in the monitored server;
Search the recovery procedure information database based on the failure event, read the response procedure, and generate a schedule table that schedules the response procedure so that failure recovery is in time by the deadline input from the deadline input receiving means Schedule analysis generation means for
Corresponding history monitoring means for updating the contents of the case accumulation database and the contents of the recovery procedure information database each time a failure occurs in the monitored server;
A failure response system comprising: The schedule analysis generation means recovers per unit time from the recovery rate stored in the recovery procedure information database, which is the recovery rate for each corresponding procedure, and the time required for the corresponding procedure having the recovery rate. Calculate the rate,
Select the corresponding procedure in order of the recovery rate per unit time,
The selection is performed by adding the time required for the selected response procedure to the date and time when the response procedure is scheduled to be performed, and until the date and time after the addition reaches the deadline,
The failure response system according to claim 1, wherein the schedule is generated so that the selected response procedure is executed in order of the recovery rate per unit time. The schedule analysis generation means generates the schedule table at least once after the schedule table is generated first and before the failure of the monitored server is recovered,
The schedule is generated again based on the date and time when the schedule is regenerated, the deadline, and the contents of the recovery procedure information database. Failure response system. A failure information database in which information about failures is stored;
Corresponding procedure for recovering the failure in association with a failure event that is the type of failure that has occurred in the monitored server, time required for the handling procedure, and information on whether or not the recovery procedure has recovered Recovery procedure information database where
A case accumulation database that stores fault events that occurred in the past, the response procedures that were taken to recover from the failures, the time required for the response procedures, and whether or not there was recovery in the response procedures;
A failure information receiving means for receiving information on a failure of the monitored server transmitted from the monitoring means built in the monitored server;
Deadline input acceptance means for accepting deadline input that is the deadline for which the failure must be recovered;
Error information analysis means for determining a failure event occurring in the monitored server;
A schedule analysis generation means for generating a schedule that schedules the response procedure so that failure recovery is in time by the deadline;
Corresponding history monitoring means for updating the contents of the case accumulation database and the contents of the recovery procedure information database each time a failure occurs in the monitored server;
A failure handling method in a failure handling system comprising:
A failure information receiving step in which the failure information receiving means receives information related to a failure of the monitored server;
The deadline input receiving means, a deadline input receiving step for receiving a deadline input that is a time limit for repairing a failure;
An error information analysis step in which the error information analysis means searches the failure information database based on the information about the failure received in the failure information receiving step to determine a failure event that has occurred in the monitored server; ,
The schedule analysis generation means reads the response procedure by searching the recovery procedure information database based on the failure event, and the response is made so that the failure recovery is in time by the deadline input by the deadline input reception step. A schedule analysis generation step for generating a schedule with scheduled procedures;
A response history monitoring step in which the response history monitoring means updates the contents of the recovery procedure information database and the case accumulation database each time a failure occurs in the monitoring target server;
A failure handling method in a failure handling system, characterized in that The schedule analysis generation step is based on the recovery rate stored in the recovery procedure information storage step, which is the probability of recovery for each corresponding procedure, and the time required for the corresponding procedure having the recovery rate. Calculating a recovery rate;
Selecting a corresponding procedure in order of high recovery rate per unit time; and
Adding the time required for the selected response procedure to the date and time when the response procedure is scheduled to be performed, and ending immediately before the date and time after the addition reaches the deadline;
Generating a schedule so that the selected response procedure is performed in the order of the high recovery rate per unit time; and
The failure handling method in the failure handling system according to claim 4, further comprising: The schedule analysis generation means step generates the schedule table at least once after the schedule table is generated first and before the failure of the monitored server is recovered.
When the schedule is regenerated, the schedule includes the step of regenerating the schedule based on the date and time at the time of recreation, the deadline, and the contents saved by the recovery procedure information saving step. The failure handling method in the failure handling system according to claim 4 or 5. Information related to the failure event associated with the information related to the failure and information related to the procedure associated with the failure associated with the failure event stored in the storage device of the computer when a failure occurs in the monitored server Based on the information on the time required for the response procedure and the information on the presence / absence of recovery in the response procedure, the computer selects the response procedure for recovering the failure and tries the response procedure. A failure handling program that generates a schedule showing the order,
On the computer,
A failure information receiving step for receiving information on a failure transmitted from the monitoring means built in the monitored server;
A deadline input acceptance step for accepting deadline input, which is a time limit for which a failure must be recovered;
Based on the information received in the failure information receiving step, the information on the failure event associated with the failure information stored in the storage device of the computer is searched, and the error occurred in the monitored server. An error information analysis step to determine the failure event;
Based on the failure event, the deadline input in the deadline input receiving step is read in the information on the response procedure for recovering the failure associated with the failure event stored in the storage device of the computer. A schedule analysis generating step for generating a schedule table in which the corresponding procedure is scheduled so that failure recovery is in time;
Each time a failure occurs in a monitored server, information about the failure that has occurred, a failure event, the response procedure that has been performed, the time required for the response procedure, and information about whether or not the recovery procedure has been restored are stored in the storage device of the computer The response history monitoring step to be stored in
A failure handling program characterized by causing In the schedule analysis generation step, recovery per unit time is calculated from a recovery rate that is stored in the storage device of the computer and is a probability of recovery for each corresponding procedure, and a time required for the corresponding procedure having the recovery rate. Calculating a rate;
Selecting a corresponding procedure in order of high recovery rate per unit time; and
Adding the time required for the selected response procedure to the date and time when the response procedure is scheduled to be performed, and ending immediately before the date and time after the addition reaches the deadline;
Generating a schedule so that the selected response procedure is performed in the order of the high recovery rate per unit time; and
The failure handling program according to claim 7, comprising: The schedule analysis generation means step generates the schedule table at least once after the schedule table is generated first and before the failure of the monitored server is recovered.
When the schedule is regenerated, the schedule includes the step of regenerating the schedule based on the date and time when the schedule is regenerated, the deadline, and the contents stored in the storage device of the computer. The failure handling program according to claim 7 or 8.

Images