JP2012079212A - Information processor and failure recovery method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor and a failure recovery method capable of automatically performing failure recovery according to an occurrence situation.SOLUTION: An information processor for making an information processor to be monitored recover from a failure occurring therein includes: a storage unit for storing, as a history, error information indicating a failure that occurred in the information processor to be monitored in the past in association with a failure recovery procedure; an operation check unit for obtaining operation information on the information processor to be monitored from the information processor to be monitored, and performing an operation check for determining whether the obtained operation information satisfies a predetermined condition; and a recovery unit for, when the operation check unit determines that the operation information does not satisfy the predetermined condition, specifying a recovery procedure corresponding to the operation information among the recovery procedures stored in the storage unit on the basis of the operation information and the error information stored in the storage unit, and making the information processor to be monitored recover from the failure according to the specified recovery procedure.

Description

  The present invention relates to an information processing apparatus that monitors the operating status of hardware, an OS (Operating System), an application, a system, and the like and automatically recovers when a failure occurs, and a failure recovery method.

  Conventionally, when a failure occurs in a hardware, OS, application, system, etc., a monitoring device such as a terminal that monitors the failure has detected a failure at a certain time interval with respect to the monitoring target device in order to recover the failure. Monitor the situation and execute a command to recover from the failure. Alternatively, when the monitoring apparatus detects an abnormality, the error is recovered by referring to the error code on the log output from the monitored device and taking a recovery method corresponding to the error code.

  For example, when starting fault monitoring, a fault monitoring tool such as scheduler software installed on the monitoring device checks the operating status of the monitored device at a predetermined interval, and monitors and monitors the log file output from the monitored device A message issued from the target device is received (for example, Patent Document 1).

JP 2003-150407 A

  However, there is a problem that it is difficult to change whether or not the monitoring target device as described above is to be checked or whether the message can be received or not depending on the operating status of the server. In disaster recovery, it is common to execute a pre-defined recovery procedure for an expected error, but in this case, it is difficult to respond flexibly unlike the case of manual intervention. There is a problem.

  For example, in the technique disclosed in Patent Document 1 described above, even if the failure monitoring can be executed using the scheduler software or the like during normal operation of the system or the like, When the situation changes in a short time, it is difficult to recover the failure in a fixed manner, and there is a problem that the failure must be recovered manually. That is, there is a problem that the failure cannot be automatically recovered according to the occurrence state.

  The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus and a failure recovery method capable of automatically recovering from a failure according to the state of occurrence.

  In order to solve the above-described problems and achieve the object, an information processing apparatus according to the present invention is an information processing apparatus that recovers a failure that has occurred in an information processing apparatus to be monitored, and the information to be monitored A storage unit that stores error information indicating a past failure that has occurred in the processing device and a recovery procedure for the failure in association with each other and a history of the information processing device to be monitored from the information processing device to be monitored An operation check unit that acquires operation information and performs an operation check to determine whether the acquired operation information satisfies a predetermined condition, and the operation check unit determines that the operation information does not satisfy a predetermined condition Accordingly, based on the operation information and the error information stored in the storage unit, a recovery procedure corresponding to the operation information is stored in the storage procedure stored in the storage unit. Identified from, characterized in that and a recovery unit to recover the failure that occurred in the be monitored information processing apparatus according to the specified the recovery procedure.

  The present invention is also a failure recovery method performed by the information processing apparatus.

  According to the present invention, it is possible to provide an information processing apparatus and a failure recovery method capable of automatically recovering from a failure according to an occurrence state.

It is a block diagram which shows the structure of the failure monitoring and automatic recovery system concerning embodiment of this invention. It is a figure which shows the example of the command and process result which execution control DB memorize | stores. It is a figure which shows the example of the execution condition and determination condition which a result determination table memorize | stores. It is a figure which shows the example of the concrete content and the judgment method of the command which a command table memorize | stores. It is a figure which shows the example of the specific content of the keyword which a keyword table memorize | stores. It is a figure which shows the example of the setting information which a server information table memorize | stores. It is a figure which shows the example of the information regarding the notification destination which a notification destination table memorize | stores. It is a figure which shows the example of the information regarding the execution timing of the command which a schedule table memorize | stores. It is a figure which shows the example of the information regarding the performance which performance data DB memorize | stores. It is a figure which shows the example of the extraction conditions which a result extraction table memorize | stores. It is a figure which shows the example which error management DB memorize | stores the information regarding recovery. It is a figure which shows the example of the information regarding the recovery result which a failure management table memorize | stores. It is a flowchart which shows the process sequence of the operation check process which an operation check process part performs. It is a flowchart which shows the process sequence of the performance data collection process which a performance data collection part performs. It is a flowchart which shows the process sequence of the recovery process which a recovery part performs.

  Exemplary embodiments of an information processing apparatus and a failure recovery method according to the present invention will be explained below in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a configuration of a failure monitoring / automatic recovery system 1000 according to an embodiment of the present invention. As shown in FIG. 1, a failure monitoring / automatic recovery system 1000 includes a Windows (registered trademark) server 101, a Unix (registered trademark) server 102, a DB (DataBase) server 103, an application server 104, and a common monitoring terminal. 251 and a network N. The network N is a general communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network). In the following, the failure monitoring / automatic recovery system 1000 targets the above-described various servers, but is not limited to the devices including these OSs and databases.

  The Windows server 101 and the Unix server 102 are servers to be monitored by the failure monitoring / automatic recovery system 1000 (hereinafter simply referred to as monitoring target servers). For example, various applications registered in the application server 104 Is to execute. The DB server 103 stores, for example, data generated by an external application. In addition, the application server 104 stores various applications executed in the above-described Windows server 101 and Unix server 102, for example.

  The common monitoring terminal 251 is a terminal such as a PC (Personal Computer) that monitors each of the monitoring target servers described above and automatically recovers from the failure when a failure occurs in each monitoring target server. As shown in FIG. 1, the common monitoring terminal 251 includes a control unit 201 and a storage unit 350. The control unit 201 includes an arithmetic device such as a CPU (Central Processing Unit), and includes an operation check unit 301, a performance data collection unit 302, a recovery unit 303, and a communication unit 304. Note that the control unit 201 has a timer (not shown) that counts the current date and time (time). The communication unit 304 is a communication device such as a NIC (Network Interface Card).

  The storage unit 350 includes a storage medium such as an HDD (Hard Disk Drive), an execution control DB 351, a result determination table 352, a command table 353, a keyword table 354, a server information table 355, and a notification destination. A table 356, a schedule table 357, a performance DB 358, a result extraction table 359, an error management DB 360, and a failure management table 361 are stored.

  The execution control DB 351 stores a command to be executed on the monitoring target server shown in FIG. 1 and processing results of an operation check process, a performance data collection process, and a recovery process (all described later) executed by the command. is there.

  FIG. 2 is a diagram illustrating an example of commands and processing results stored in the execution control DB 351. As illustrated in FIG. 2, the execution control DB 351 includes a target server for uniquely identifying a monitoring target server, a command for causing the monitoring target server to execute, and an OS (Operating System) type of the monitoring target server. Indicates the server type indicating the date and time when the command was executed, the acquired value indicating the execution result acquired by the executed command, and the type of status of the monitored server when the command was executed Corresponds to the status, the time variation flag that indicates whether the command execution interval is changed according to the status of the monitored server, and the next execution interval that indicates the interval from the command execution to the next execution It is remembered.

  In FIG. 2, for example, for the target server “development server 1” of the server type “OS1”, in the status “A” (described later, the monitored server is operating stably), the command “ UBDF01 ”is executed at the execution date“ 2009/4/10 22:00: 00 ”, and the acquired value“ 80 ”(%) is acquired at that time. Further, the interval until the next execution is “10 minutes later”, indicating that the execution interval is not changed according to the situation (No). Further, it is shown that the command “USYL01” is set for the same target server “development server 1”, and the above-described items are set similarly.

  In the example shown in FIG. 2, the execution control DB 351 stores the execution date and time, the acquired value, and the status from (1) to (15) so that the result and value can be stored each time the command is executed. However, the number is not particularly limited. In addition, each item other than the above-described execution date / time, acquired value, and status is set in advance by, for example, the administrator of the failure monitoring / automatic recovery system 1000 or the like. Next, returning to FIG. 1, the result determination table 352 will be described.

  The result determination table 352 stores execution conditions for commands executed on the monitoring target server and determination conditions for determining the status of the monitoring target server.

  FIG. 3 is a diagram illustrating an example of execution conditions and determination conditions stored in the result determination table 352. As shown in FIG. 3, the result determination table 352 includes an execution command for uniquely specifying a command to be executed, a target server type indicating the type of OS of the monitoring target server that executes the command, and a monitoring target server. Status A indicating the command execution interval in a normal state (for example, no OS error) and stable operation, and status indicating the command execution interval when the monitored server is normal B, status C indicating the command execution interval when the monitored server is not normal (at the time of warning), initial value indicating the command execution interval when the monitored server is in the initial state, and an operation check described later Process and performance data collection process should be determined that the monitored server is stable A threshold (A) for determining that there is a status (status A), and an operating status for determining that the monitored server needs attention (status B) Threshold value (B), threshold value (C) for determining that the monitored server is in an operating state (status C) that should be determined as an error, and execution of commands in statuses A to C The number of status reviews indicating the number of times to review the interval is stored in association with each other.

  In FIG. 3, for example, the execution command “UBDF01” executed for the monitoring target server of the target server type “OS1” is “12” when the monitoring target server is in a normal state and operates stably. Executed every "hour", executed every "6 hours" when the monitored server is normal, and "every 2 hours" when the monitored server is not normal (at the time of warning) Is shown. In addition, the interval at which the execution command “UBDF01” is initially executed is “every 3 hours”. Furthermore, if the acquired value acquired in the operation check process, performance data collection process, and recovery process, which will be described later, is a value equal to or greater than the threshold value (A) “80” (%), the monitored server is in a normal state. If the acquired value is equal to or greater than the threshold value (B) “90” (%), the monitored server needs to be noted. (Status B status) and the acquired value is “98” (%) or more, this indicates that the monitored server is in an error status (status C status), and the command is “ This indicates that the execution intervals of the commands indicated by the statuses A to C are reviewed when the command is executed “10 times”. Each item of the result determination table 352 described above is set in advance by, for example, an administrator of the failure monitoring / automatic recovery system 1000 or the like. Next, returning to FIG. 1, the command table 353 will be described.

  The command table 353 stores specific contents of the command to be executed and a determination method for determining the state of the monitoring target server when the command is executed.

  FIG. 4 is a diagram illustrating an example of specific contents and determination methods of commands stored in the command table 353. As shown in FIG. 4, the command table 353 includes a command ID for uniquely identifying a command to be executed, a determination type indicating a determination method for determining the state of the monitored server when the command is executed, and The target server type described above, the command actually executed, and option information (such as an argument) when executing the command are stored in association with each other.

  In FIG. 4, for example, the execution command “UBDF01” executed for the monitoring target server of the target server type “OS1” is determined by “threshold” as a determination method for determining the state of the monitoring target server. The actually executed command is “dbf”, indicating that no argument is specified. Further, the execution command “CDBC01” executed in common to each monitored server is determined by checking the log output at the time of command execution (“log check”) as a determination method, and the actually executed command is “ "brconnect", indicating that an argument "-u / -cf check" is specified. It should be noted that each item of the command table 353 described above is set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, like the result determination table 352 and the like. Next, returning to FIG. 1, the keyword table 354 will be described.

  The keyword table 354 stores various information (keywords) such as numerical values, character strings, codes, and the like included in a log output when a command is executed as an alarm.

  FIG. 5 is a diagram showing an example of specific contents of keywords stored in the keyword table 354. As shown in FIG. 5, the keyword table 354 is sufficient for the above-mentioned execution command, an error keyword indicating a keyword that should be an error among the keywords included in the log, and a level to be noted among the keywords included in the log. In addition, an ignoring keyword indicating a keyword that is not at an error level is stored in association with each other.

  In FIG. 5, for example, an execution command “CDBC01” is executed, and an error occurs when the output log includes the error keywords “ORA-01555”, “ORA-0660”, and “ORA-16631”. On the other hand, if the output log includes the ignored keywords “ORA-23701”, “ORA-03106”, and “ORA-01428”, it indicates that the processing is continued without causing an error. Yes. It should be noted that each item of the keyword table 354 described above is set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, like the result determination table 352 and the like. Next, returning to FIG. 1, the server information table 355 will be described.

  The server information table 355 stores various setting information related to the monitoring target server.

  FIG. 6 is a diagram illustrating an example of setting information stored in the server information table 355. As shown in FIG. 6, the server information table 355 includes a host name of the monitoring target server, an IP address of the monitoring target server, an OS type of the monitoring target server, a usage type indicating the usage of the monitoring target server, and a monitoring target. Instance name when the usage type of the server is “DB”, administrator account (management user account) and password (management user password) that can access the managed server, and user account that can access the DB ( DB user account) and password (DB user password) are stored in association with each other.

  In FIG. 6, for example, the IP address of the monitoring target server having the host name “development server 1” and the OS type “OS1” is “AAA.AAA.AAA.AAA”, and is set as the DB server of the instance name “BSC”. It has been shown. In addition, the management user account and the management user password of this server are “root” and “AAAAAAA”, respectively, and the DB user account and the DB user password are “bscadm” and “AAAAAAA”, respectively. Each item of the server information table 355 described above is set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, like the result determination table 352 and the like. Next, returning to FIG. 1, the notification destination table 356 will be described.

  The notification destination table 356 stores information related to a notification destination that notifies that when a monitoring target server has an error or failure.

  FIG. 7 is a diagram illustrating an example of information regarding the notification destination stored in the notification destination table 356. As shown in FIG. 7, the notification destination table 356 has a target server that notifies the notification destination that an error or failure has occurred, an error code, a contact means that indicates the notification method, and an error or failure. A contact address for notification to the effect and a contact registrant name indicating the name and name of the address are stored in association with each other.

  In FIG. 7, for example, when an error with the error code “ORA-1555” occurs in the target server “development server 1”, the registrant “AAAA” is notified to “029-XXX-XXXX” by telephone. Show. In FIG. 7, a spare contact address (up to contact address 5) is stored in preparation for a case where the contactee registrant cannot be contacted for reasons such as absence. Each item shown here is assumed to be set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, like the result determination table 352 and the like. Next, returning to FIG. 1, the schedule table 357 will be described.

  The schedule table 357 stores information related to the timing for executing the command.

  FIG. 8 is a diagram illustrating an example of information related to the execution timing of the command stored in the schedule table 357. As shown in FIG. 8, the schedule table 357 stores a command execution date, a command execution time, a target server indicating a monitoring target server that executes the command, and an execution command in association with each other.

  In FIG. 8, for example, the execution command “UBDF01” is executed at “00:00” of “2009/01/01”, indicating that “development server 1” is executed as the target server. . Each item shown here is assumed to be set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, like the result determination table 352 and the like. Next, returning to FIG. 1, the performance data DB 358 will be described.

  The performance data DB 358 stores information related to the operating status (performance) of the monitoring target server.

  FIG. 9 is a diagram illustrating an example of information related to performance stored in the performance data DB 358. As shown in FIG. 9, the performance data DB 358 includes an identification ID for uniquely identifying the type (what kind of check is performed) for checking the performance of the monitored server, and a check for the check identified by the identification ID. The contents, the above-described target server and execution command, and the command execution interval and the execution result are stored in association with each other. In the example shown in FIG. 9, the execution interval and the execution result are registered from 1 to 10, but the number is not limited to this number.

  FIG. 9 shows that, for example, the execution command “UBDF01” identified by the identification ID “HW001” is executed for the server “development server 1” at the memory usage rate every minute. As an execution result, a value such as 35 (%) or 50 (%) is stored. The execution results are set in the performance data collection process described later, and the other items are set in advance by the administrator of the failure monitoring / automatic recovery system 1000 as in the result determination table 352 and the like. To do. In this way, by registering the execution results periodically and repeatedly, for example, the administrator can easily grasp the transition of the memory usage rate. Next, returning to FIG. 1, the result extraction table 359 will be described.

  The result extraction table 359 performs operation check processing and performance data collection processing, which will be described later, and only information necessary for the failure monitoring / automatic recovery system 1000 out of various information included in the log output as a result thereof. The extraction condition is stored for extracting the.

  FIG. 10 is a diagram illustrating an example of extraction conditions stored in the result extraction table 359. As shown in FIG. 10, the result extraction table 359 includes an execution command, an extraction keyword for analyzing and extracting only necessary information from various information included in the log, and information extracted by the extraction keyword. Are stored in association with the totaling conditions for the totalization.

  In FIG. 10, for example, a character string (or its character string) including two character strings “cpu” and “us” from information such as a log output when the execution command “UBDF01” is executed. (A group of paragraphs including) is extracted. In the example shown in FIG. 10, five keywords are set as extraction keywords. Each of these items is set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, as in the result determination table 352 and the like. Next, returning to FIG. 1, the error management DB 360 will be described.

  The error management DB 360 stores information related to recovery for performing recovery according to an error in a recovery process described later. The error management DB 360 is not automatically registered, but is registered in advance by an administrator or the like in case a similar recovery procedure does not exist in the past.

  FIG. 11 is a diagram illustrating an example in which the error management DB 360 stores information related to recovery. As shown in FIG. 11, the error management DB 360 has an error code indicating the content of an error included in a log or the like output from the monitoring target server, the above-described target server, and a correspondence order indicating the priority order for executing the recovery process. A notification content indicating a message notified to a destination registered in the notification destination table 356, a check execution command indicating a command executed in the operation check process, and a recovery procedure indicating a command including a procedure for recovering an error; Are stored in association with each other.

  In FIG. 11, when the execution command “UBDF01” at the time of the operation check process is executed for the target server of “OS1” and the error code “ORA-16631” is included in the log or the like, This indicates that “the table / index has reached MaxExtent (***)” and that the command “R33TCDB13” is executed as a recovery procedure. In addition, when the execution command “UPSC01” at the time of the operation check process is executed for the target server of the same “OS1” and the error code “ORA-01631” is included in the log or the like, the administrator etc. Indicates that “the table / index has reached MaxExtent (***)” and that the command “ORAEXTCH01” is executed as a recovery procedure. When these two errors occur at the same timing, the command “R33TCDB13” whose correspondence order is “1” is executed, and thereafter the command “ORAEXTCH01” whose correspondence order is “2” is executed. It is shown that. Each of these items is set in advance by the administrator of the failure monitoring / automatic recovery system 1000 or the like, as in the result determination table 352 and the like. Next, returning to FIG. 1, the failure management table 361 will be described.

  The failure management table 361 stores information related to the recovery result executed in the recovery process.

  FIG. 12 is a diagram illustrating an example of information related to the recovery result stored in the failure management table 361. As shown in FIG. 12, the failure management table 361 includes the error code, the execution server indicating the monitoring target server on which the recovery process is performed, the execution date and time indicating the date and time when the recovery process is started, and the recovery process. A recovery procedure indicating the performed procedure, a confirmation procedure indicating a check procedure when the recovery process is completed, and a completion date and time indicating the date and time when the recovery process is completed are stored in association with each other.

  In FIG. 12, for example, as a recovery process when an error code “ORA-1631” occurs in the monitoring target server “development server 5”, the completion date and time from “2008/12/31 18:30” This indicates that “Extent extension” has been performed until “2008/12/31 21:30”. Further, it is indicated that the procedure is to confirm the result of the recovery process by “DBCheck”. The recovery procedure and the confirmation procedure are actually stored in the failure management table 361 as codes such as “ABCD” and “1234”. These items are set when the recovery process is executed. Subsequently, returning to FIG. 1, the operation check unit 301 will be described.

  The operation check unit 301 performs a health check for checking the operation state of each server to be monitored and confirms the operation status from various log information output from each server. The operation check unit 301 refers to various tables such as the execution control DB 351, the result determination table 352, the command table 353, the keyword table 354, the server information table 355, the notification destination table 356, and the schedule table 357 described later. Perform the process. The various log information includes, for example, an event log, an access log, a system log, and an error log stored in each server described above, a DB alert log, a server operation information log, an application log, and a job log.

  The performance data collection unit 302 collects the usage status of resources included in each server to be monitored and the operation information of each server, and monitors the collected operation status of each server. The performance data collection unit 302 refers to various tables such as a result determination table 352, a command table 353, a server information table 355, a schedule table 357, a performance DB 358, a result extraction table 359, and performs various processes described later.

  When the operation check unit 301 or the performance data collection unit 302 detects an abnormality of each server to be monitored, the recovery unit 303 analyzes the abnormality state and executes a response corresponding to the abnormality state. . The recovery unit 301 performs various processes described later with reference to various tables such as the command table 353, the notification destination table 356, the error management DB 360, and the failure management table 361. In addition, when the recovery unit 303 detects an abnormality in each server, an abnormality has occurred in a telephone or e-mail addressed to the server administrator 401 or maintenance contractor 402 of the failure monitoring / automatic recovery system 1000 registered in advance. Notification to that effect.

Next, processing performed in the failure monitoring / automatic recovery system 1000 will be described. First, the operation check process performed by the operation check unit 301 will be described.
FIG. 13 is a flowchart illustrating the processing procedure of the operation check process performed by the operation check processing unit 301. In the following description, it is assumed that devices such as servers to be monitored are determined in advance, and the failure monitoring / automatic recovery system 1000 is activated at a predetermined timing.

  As shown in FIG. 13, the operation check processing unit 301 first refers to the schedule table 357 (step S1151), compares the current time with the execution date and time, and the target server should execute the execution command. It is determined whether it is timing (step S1152).

  If the operation check unit 301 determines that it is the execution timing at which the execution command should be executed (step S1151; Yes), it refers to the server information table 1102, identifies the host name of the target server in the schedule table 357, Target server information including login information of information (IP address, host name, various accounts, passwords, etc.) of the same record as the identified host name is read (step S1154).

  On the other hand, when the operation check unit 301 determines that it is not the execution timing for executing the execution command (step S1151; No), the operation check unit 301 waits until the execution timing is reached. As will be described later, if the recovery unit 303 is executing recovery processing at this point (step S1153), steps S1151 to S1152 are not performed (determination of execution timing is not performed), and immediately after step S1154. Perform the process.

  Subsequently, the operation check unit 301 refers to the command table 353 using the execution command of the schedule table 357 and the OS type of the server information table 355 as keys, and the commands and options included in the record that matches the execution command and the OS type. (That is, a command or option that is actually executed. Hereinafter, it is referred to as command detailed information.) (Step S1155).

  Then, the operation check unit 301 transmits the acquired command detailed information to the server indicated in the target server information acquired in step S1154, and executes the command transmitted to the server (step S1158), and the execution result of the command Is acquired (step S1159).

  Here, since the execution result is simply the result of executing the command, unnecessary information is also included in the operation check of the server. Therefore, the operation check unit 301 refers to the result extraction table 359 using the executed command as a key, specifies a record including an execution command that matches the executed command, and extracts only information related to the extracted keyword included in the specified record. To process (step S1160).

  Since information indicating normal termination / abnormal termination / simple operation included in various log information is included, the operation check unit 301 extracts only the information related to the extracted keyword as described above. By performing the operation check, a more accurate result can be obtained.

  When the processing in step S1160 ends, the operation check unit 301 refers to the command table 353 using the executed command as a key, identifies a record including a command ID that matches the command, and determines the determination type included in the identified record. It is determined whether or not it is a threshold value (that is, whether or not the execution result is confirmed by the threshold value) (step S1161).

  When it is determined that the execution result is confirmed by the threshold value (step S1161; Yes), the operation check unit 301 reads the threshold values (A) to (C) of the result determination table 352 (step S1163), and the execution result is After determining whether the read threshold value (A) to (C) falls within the range, it is determined whether the threshold value (C) is exceeded. (Step S1164).

  When the operation check unit 301 determines that the execution result is within the range exceeding the threshold value (C) (step S1164; Yes), the operation check unit 301 determines that the operating state of the server is abnormal, The process proceeds to step S1171. Note that if an abnormality is detected in the performance data collection process described later, the process proceeds to step S1171 as in step S1164 (step S1162).

  On the other hand, when the operation check unit 301 determines that the execution result is not within the range exceeding the threshold value (C) (step S1164; No), the operation check unit 301 determines that the server is not in an abnormal operating state (server And the execution result is stored in the acquired value and status of the execution control DB 351 (step S1165).

  After that, the operation check unit 301 refers to the execution control DB 351 and the result determination table 352 to set the next execution time of the operation check, determines the timing for executing the command next time, and sets the result in the schedule table 357 and The data is stored in the execution control DB 351 (step S1166).

  For example, if the execution result of the execution command “UBDF01” is “80%” in step S1164, the status of the server is “A”, so refer to the status A of the result determination table 352 shown in FIG. The timing is set so that the command “UBDF01” is executed “every 12 hours”.

  Similarly, when the execution result of the execution command “UBDF01” is “90%”, the status of the server is “B”, so the status “B” in the result determination table 352 is referred to, and the command “ When the timing is set so that “UBDF01” is executed “every 6 hours” and the execution result of the execution command “UBDF01” is “98%”, the status of the server is “C”. Therefore, the status C of the result determination table 352 is referred to, and the timing is set so that the command “UBDF01” is executed “every 3 hours”. As described above, as the server operating state approaches an abnormal state (dangerous state), the interval at which the command is executed can be shortened to quickly grasp the server operating state.

  Thereafter, the operation check unit 301 determines whether or not the operation check process is performed in a state where the recovery process is activated and executed (that is, whether the operation check is executed alone or as part of the recovery process). (Step S1167). If it is determined that the operation check has not been executed as part of the recovery process (Step S1167; No), the process is terminated.

  On the other hand, when the operation check unit 301 determines that the operation check is being executed as part of the recovery process (step S1167; Yes), the operation check unit 301 proceeds to the recovery process illustrated in FIG. 15 (step S1168).

  The operation check unit 301 refers to the keyword table 1112 when it is determined in step S1161 that the execution result is not confirmed by the threshold (that is, it is determined that the execution result is confirmed by various log information) (step S1161; No). The execution command that matches the command executed in step S1157 is read from the various log information and specified (step S1169), and there is an error in the execution result of the command, and the error is registered as an error keyword in the keyword table 354. It is determined whether or not (i.e., whether an error exists in the execution result and the error cannot be ignored) (step S1170).

  Then, the operation check unit 301 does not have an error in the execution result, or the error cannot be ignored (for example, even if an error exists in the execution result, the error is an ignorable keyword). When it determines (step S1170; No), it progresses to step S1165 mentioned above.

  On the other hand, when the operation check unit 301 determines that an error exists in the execution result or the error cannot be ignored (for example, an error exists in the execution result and the error is an error keyword). (Step S1170; Yes), with reference to the notification destination table 356, the target server that matches the host name included in the target server information acquired in Step S1154 is specified (Step S1171), and registered as the contact information of the specified target server The notified destination is notified (step S1172).

  Thereafter, as in the case of steps S1165 and S1166, the operation check unit 301 stores the execution result in the execution control DB 351 (step S1173), determines the timing for executing the command next time, and stores the result in the schedule table 357 and Stored in the execution control DB 351 (step S1174). When the process of step S1174 or the process of step S1167 ends, all the process of the operation check process shown in FIG. 13 ends. Next, performance data collection processing performed by the performance data collection unit 302 will be described.

  FIG. 14 is a flowchart illustrating a processing procedure of performance data collection processing performed by the performance data collection unit 302.

  As shown in FIG. 14, the performance data collection unit 302 reads the schedule table 357 (step S1251), compares the execution date of the schedule table 357 with the current time, and determines whether it is the execution timing of the performance data collection process. Is determined (step S1252).

  If the performance data collection unit 302 determines that it is the execution timing of the performance data collection process (step S1252; Yes), it further reads the server information table 355 and sets the same host name as the target server of the schedule table 357. The target server information including the OS type is acquired (step S1253).

  Thereafter, the performance data collection unit 302 refers to the command table 353, and identifies the execution command included in the schedule table 357 read in step S1251 and the command included in the record that matches the OS type acquired in step S1253 ( In step S1254, an execution file (execution command) for executing the command is generated (step S1255).

  Specifically, the performance data collection unit 302 reads the target server (for example, the development server 1) and the execution command (for example, UBDF01) of the schedule table 357 in step S1251, and further uses the read target server as a key. In S1253, the OS type (for example, OS1) in the server information table 355 is read. Thereafter, in step S1254, the performance data collection unit 302 further acquires a command (for example, dbf) in the command table 353 using the execution command and the OS type as a key, and generates a file for executing the acquired command. To do.

  The performance data collection unit 302 transmits the file generated in step S1255 to the target server to execute the process (steps S1256 and S1257), acquires the execution result, and executes the target server and the execution command (for example, the development server). 1 (UBDF01) as a key, the execution result (for example, 35 (%)) is registered in the performance data DB 358 (step S1258).

  Here, since the execution result is simply the result of executing the command, information unnecessary for collecting server performance data is also included. Therefore, the performance data collection unit 302 refers to the result extraction table 359 using the executed command as a key, and specifies a record including an executed command that matches the executed command, as in step S1160 shown in FIG. Only the information related to the extracted keyword included in the specified record is extracted and processed (step S1259).

  Furthermore, the performance data collection unit 302 reads the result determination table 352 using the execution command (for example, UBDF01) and the OS type (for example, OS1) as keys because the acquired execution result may be abnormal. The record including the execution command and the target server type that matches the execution command and the OS type is specified, and the threshold value of the record (for example, 80 (% or more), 90 (% or more), 98 (% or more)) is acquired. In step S1260, the execution result acquired in step S1258 is registered in the execution control DB 351 using the target server, command, and target server type (for example, development server 1, UBDF01, OS1) as keys (step S1261).

  Thereafter, the performance data collection unit 302 determines whether or not the execution result acquired in step S1258 satisfies the threshold acquired in step S1260 (whether or not the threshold is exceeded) (step S1262). When it is determined that the execution result does not exceed the threshold value (step S1262; No), it is determined which threshold value the execution result satisfies, and the result (acquired value, status) is executed. Register in the control DB 351 (step S1265).

  On the other hand, if the performance data collection unit 302 determines that the execution result exceeds the threshold (step S1262; Yes), the performance data collection unit 302 determines that the server operating status is abnormal, and the result (acquired value, status). Is registered in the execution control DB 351 (step S1263), and the operation check process shown in FIG. 13 is performed to check the operation of the server (step S1264). When the processing of step S1264 or step S1265 is completed, all the performance data collection processing shown in FIG. 14 is completed. Next, the recovery process performed by the recovery unit 303 will be described.

  FIG. 15 is a flowchart showing a processing procedure of recovery processing performed by the recovery unit 303. Note that the recovery process shown below does not start on its own, there is an error that cannot be ignored in the execution result in the operation check process shown in FIG. 13, and the abnormality is registered in the destination registered as a contact. When notified (when the processing from step S1171 to 1174 shown in FIG. 13 is executed), it is automatically started.

  As illustrated in FIG. 15, the recovery unit 303 reads information in the notification destination table 356 at the timing when the process transitions from the operation check process to the recovery process (step S1351), and contacts the notification destination table 356 referred to in step S1171. The destination registered as the destination is notified that the recovery process is to be started (step S1352).

  Then, the recovery unit 303 acquires a target server that matches the host name included in the target server information acquired in step S1154 of the operation check process illustrated in FIG. 13 (step S1353). Thereafter, the recovery unit 303 reads a record in which the acquired target server and the execution server of the failure management table 361 match (step S1354), and stores the past in the failure management table 361 using the error code included in the read record as a key. It is checked whether an error has occurred with the same command (step S1355), and it is determined whether the same failure history is stored in the failure management table 361 (step S1356).

  When it is determined that the same failure history is stored in the failure management table 361 (step S1356; Yes), the recovery unit 303 selects a recovery procedure corresponding to the stored failure history (error code), The past procedure is adopted (step S1357).

  For example, when the target server information acquired in step S1154 of the operation check process illustrated in FIG. 13 is “development server 5”, the recovery unit 303 performs the implementation server “development server” of the failure management table 361 that matches the target server. A record including “5” is read. Then, the recovery unit 303 refers to the error code “ORA-16311” included in the record, and adopts “Extent extension (ABCD)” as the recovery procedure.

  On the other hand, when the recovery unit 303 determines that the same failure history is not stored in the failure management table 361 (step S1356; No), the recovery unit 303 refers to the server information table 355 and has the same OS type among other servers. The same name as the host name of the server is identified from the failure management table 361, and the recovery procedure of the identified other server (implementing server) is confirmed (step S1366).

  If the recovery unit 303 determines that there is a recovery procedure for another server (step S1366; Yes), the process proceeds to step S1357, and the recovery procedure for the other server is performed in the same manner as when the past recovery procedure is adopted. adopt.

  On the other hand, if the recovery unit 303 determines that there is no recovery procedure for another server (step S1366; No), the recovery unit 303 refers to the error code in the error management DB 360 (step S1367) and stores the same failure history in the error management DB 360. It is determined whether it has been performed (step S1368).

  When the recovery unit 303 determines that the same failure history is stored in the error management DB 360 (step S1368; Yes), the recovery unit 303 adopts the recovery procedure stored in the error management DB 360 (step S1369). The process proceeds to S1358.

  On the other hand, if the recovery unit 303 determines that the same failure history is not stored in the error management DB 360 (step S1368; No), the automatic recovery process is difficult, and thus the recovery process of step S1172 shown in FIG. As in the case, the relevant person is notified again (step S1370), and the administrator takes action (step S1371).

  When step S1357 or step S1369 ends, the recovery procedure command adopted in step S1357 or step S1369 (for example, the command when the recovery procedure “Extent extension” in the failure management table 361 is executed, or the recovery procedure of the error management DB 360) The command “UBDF01” corresponding to the file “R3RTCDB13” and the error code “ORA-1631” is acquired from the command table 353 (step S1358), and the acquired command is the same as in step S1255 shown in FIG. A file for executing is generated (step S1359).

  Then, the recovery unit 303 transmits the file generated in step S1359 to the target server to execute the recovery process (steps S1360 and S1361), and acquires the execution result (for example, various log information of the server). (Step S1362).

  When the process of step S1362 ends, the operation check unit 301 executes the operation check process from steps S1153 (A) to S1168 (D) shown in FIG. 13 in order to confirm the validity of the recovery procedure. When these operation check processes are completed, the recovery unit 303 determines whether or not the error has been resolved by the operation check process (for example, it is not in a range exceeding the threshold value (C)). (Step S1365).

  When the recovery unit 303 determines that the error has not been eliminated by the operation check process (step S1365; No), the recovery unit 303 returns to step S1369 and adopts the recovery procedure stored in the error management DB 360 ( The process proceeds to step S1369) and step S1358.

  On the other hand, when the recovery unit 303 determines that the error has been eliminated by the operation check process (step S1365; Yes), the recovery procedure is registered in the failure management table 361 (step S1372), and the error code is used as a key. In order to preferentially execute the recovery procedure when the error code is generated, the correspondence order of the error management DB 360 is changed (step S1373).

  Specifically, in the example illustrated in FIG. 11, when the error code “ORA-01631” occurs in the server of the target server “OS1”, as a recovery procedure, first, the file “R3RTCDB13” is executed. However, for example, when the “ORAEXTCH01” file is executed, the recovery process can be executed in a short processing time when a larger capacity can be secured. If the executed recovery process is valid as shown above, the order of executing the “ORAEXTCH01” file is executed first when a failure occurs in the future by switching the correspondence order of the recovery procedures. As such, the priority is set high.

  Then, when the processing up to step S1373 is completed, the recovery unit 303 resets the items by clearing each item of the execution control DB 351 (step S1374), and notifies that the recovery processing is completed, as shown in FIG. The related parties are notified in the same manner as in S1172 (steps S1375 and S1376). When the process of step S1376 or step S1371 is finished, all the recovery processes shown in FIG. 15 are finished.

  As described above, in the common monitoring terminal 251 that recovers a failure that has occurred in the server to be monitored, the storage unit 350 corresponds to an error code indicating a past failure that has occurred in the server to be monitored and a recovery procedure for the failure. The operation check unit 301 acquires the operation information of the monitoring target server from the monitoring target server, and whether the acquired operation information satisfies a predetermined condition (threshold value, keyword) In response to the operation check for determining whether or not the operation check unit 301 determines that the operation information does not satisfy the predetermined condition (threshold value, keyword), the operation information and the storage are stored. Based on the error code stored in the unit 350, a recovery procedure corresponding to the operation information is identified from the recovery procedures stored in the storage unit 350, and Since to recover the failure that occurred in the server to be monitored according to the recovery procedure, it is possible to automatically recover the fault in accordance with the occurrence.

For example, the conditions for checking server operation are automatically changed according to the operating status of the server, and when a failure occurs in the server, not only a predefined scenario but also a recovery procedure with a past record It is possible to recover automatically.
Also, when the server is operating normally, its operation check or resource usage status and performance are automatically executed, and if an abnormality is detected at that time, an error is automatically detected from the report to the administrator. Since a series of work up to the recovery of the failure that caused the problem is performed, the operation and maintenance of the server and system will be performed automatically, and it will be possible to reduce the burden on the operations staff and administrator.
Furthermore, since a series of operations up to the recovery of the failure is automatically performed, the failure can be recovered quickly, and as a result, the system downtime can be suppressed, so that a stable service can be provided to the user. Can do.

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1000 Fault monitoring / automatic recovery system,
101 Windows server 102 Unix server 103 DB (DataBase) server 104 Application server 251 Common monitoring terminal 201 Control unit 301 Operation check unit 302 Performance data collection unit 303 Recovery unit 304 Communication unit 350 Storage unit 351 Execution control DB
352 Result determination table 353 Command table 354 Keyword table 355 Server information table 356 Notification destination table 357 Schedule table 358 Performance DB
359 Result extraction table 360 Error management DB
361 Fault management table N Network.

Claims (9)

An information processing apparatus that recovers a failure that occurred in an information processing apparatus to be monitored,
A storage unit that stores, in a history, error information indicating a past failure that has occurred in the information processing apparatus to be monitored and a recovery procedure for the failure;
An operation check unit that acquires operation information of the information processing apparatus to be monitored from the information processing apparatus to be monitored and performs an operation check to determine whether the acquired operation information satisfies a predetermined condition; ,
In response to determining that the operation information does not satisfy a predetermined condition, the operation check unit responds to the operation information based on the operation information and the error information stored in the storage unit. A recovery unit that identifies a recovery procedure from the recovery procedure stored in the storage unit, and recovers a failure that has occurred in the information processing apparatus to be monitored according to the specified recovery procedure;
An information processing apparatus comprising: When the operation check unit determines that the operation information does not satisfy a predetermined condition, the operation check unit notifies a predetermined administrator that a failure has occurred in the information processing apparatus to be monitored,
The recovery unit is generated in the information processing device to be monitored in response to the operation check unit informing a predetermined administrator that a failure has occurred in the information processing device to be monitored. To recover from a failure
The information processing apparatus according to claim 1. The operation check unit determines whether the operation information is within a predetermined condition by determining whether the operation information is within a predetermined threshold range or whether the operation information includes a predetermined keyword. To determine whether or not
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The information processing apparatus monitors the operating status of a plurality of information processing apparatuses to be monitored,
The storage unit associates and stores device identification information for identifying the information processing device to be monitored, the error information, and the recovery procedure in a history,
The operation check unit acquires operation information and the device identification information from a plurality of information processing devices to be monitored, determines whether the acquired operation information satisfies a predetermined condition,
The recovery unit, when the operation check unit determines that the operation information does not satisfy a predetermined condition, with the device identification information as a key, the operation information and the storage unit stored in the storage unit Based on the error information, recovering a failure that occurred in the information processing apparatus to be monitored according to the recovery procedure corresponding to the same apparatus identification information as the apparatus identification information,
The information processing apparatus according to any one of claims 1 to 3. When the recovery procedure corresponding to the same device identification information as the device identification information does not exist, the recovery unit occurs in the information processing apparatus to be monitored according to the recovery procedure corresponding to the other device identification information. To recover from a failure
The information processing apparatus according to claim 4. The operation check unit acquires operation information from the information processing device to be monitored in a state where the recovery unit has recovered a failure that has occurred in the information processing device to be monitored, and the acquired operation information Determine whether a given condition is met,
The recovery unit raises the priority of the recovery procedure when the operation check unit determines that the operation information satisfies a predetermined condition in a state where the failure is recovered,
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. When the operation check unit determines that the operation information satisfies a predetermined condition, the operation check unit performs the operation check at a shorter interval as the operation information approaches a state that does not satisfy the predetermined condition.
The information processing apparatus according to any one of claims 1 to 6. Further comprising a performance unit that monitors an operation status of the information processing apparatus to be monitored at predetermined intervals and determines whether the operation information satisfies a predetermined condition;
When the performance unit determines that the operation information does not satisfy a predetermined condition, the operation check unit notifies a predetermined administrator that a failure has occurred in the information processing apparatus to be monitored. ,
The recovery unit is generated in the information processing device to be monitored in response to the operation check unit informing a predetermined administrator that a failure has occurred in the information processing device to be monitored. To recover from a failure
The information processing apparatus according to any one of claims 1 to 7. A fault monitoring / automatic recovery method performed by an information processing apparatus that recovers a fault that occurred in an information processing apparatus to be monitored,
An operation check step of obtaining operation information of the information processing apparatus to be monitored from the information processing apparatus to be monitored and performing an operation check for determining whether or not the acquired operation information satisfies a predetermined condition; ,
In the operation check step, in response to determining that the operation information does not satisfy a predetermined condition, the operation information and the monitoring target stored in the history in association with the recovery procedure for the failure in the storage unit A recovery procedure corresponding to the operation information is identified from the recovery procedures stored in the storage unit based on error information indicating a past failure that has occurred in the information processing apparatus, and the identified recovery procedure Therefore, a recovery step for recovering a failure that has occurred in the information processing apparatus to be monitored;
A failure recovery method comprising:

Images