JP5651381B2 - Failure cause determination rule verification device and program - Google Patents

Description

  The present invention relates to a device for verifying a failure cause analysis rule used in a failure cause analysis system based on a past event log, and a program for realizing the device in software.

  Delays in recovery work in the event of a system failure have a major impact on corporate performance and social infrastructure. To quickly recover from a system failure, it is important to identify the cause of the failure at an early stage and determine the recovery procedure.

  In view of this, a failure cause analysis system that supports rapid recovery of failures has been proposed (Patent Document 1). This system has a failure cause determination rule that correlates an event that occurs at the time of a specific failure with a recovery procedure, and provides an appropriate recovery procedure to a recovery person by analyzing the failure event using the rule .

  However, it is difficult to manually create a failure cause determination rule. Therefore, a method for automatically generating a failure cause determination rule from an event log has been proposed (Patent Documents 2 and 3). Patent Document 2 describes a method of using the occurrence frequency of a specific event. Patent Document 3 describes a method of using an event occurrence pattern.

International Publication No. 2004/061681 JP 2008-41041 A JP 2006-4346 A

Fisher, Douglas H. “Knowledge acquisition via incremental clustering”, Machine Learning 2, 139-172, 1987

However, even if the failure cause determination rule is once created and registered, the registration content needs to be updated for the following reason.
(1) Addition of new IT service / Abolition of existing IT service When the operation of a new IT service is started, a system failure related to the service is newly generated. In this case, it is necessary to create a failure cause determination rule corresponding to a new system failure and add it to the existing failure cause determination rule. On the other hand, when an existing IT service is abolished, a system failure related to the service will not occur thereafter. In this case, it is necessary to delete the failure cause determination rule corresponding to the failure that will not occur in the future from the existing failure cause determination rule.
(2) Change of IT infrastructure configuration In the operation process of the system, the IT infrastructure may be changed even if there is no change in the provided IT service itself. For example, hardware replacement or network configuration change may occur. When the system configuration is changed in this way, even if the system failure is caused by the same cause, the attribute value of the event that occurs and the appearance of the event are affected. That is, a change to the failure cause determination rule is required.
(3) Change in recognition of system failure Naturally, when creating a failure cause determination rule, a failure cause determination rule is created based on information available at that time. However, an erroneous failure cause determination rule may be generated due to a lack of information regarding system failure. For example, the system failure A and the system failure B that have been determined to be caused by the same cause may be subsequently found to be caused by different causes. On the other hand, what was initially determined as a failure due to a different cause may be later found to be a failure due to the same cause.

  However, the conventional method does not consider the subsequent maintenance even if the failure cause determination rule is automatically generated. In other words, no consideration is given to constantly verifying the effectiveness of a failure cause determination rule after creation and updating the rule as necessary.

  Therefore, the inventor provides a mechanism for automatically updating the failure cause determination rule according to the operation status. Specifically, a process that automatically sets a time window that gives a time interval for reviewing the failure cause determination rule, a process that creates a temporary failure classification tree based on events that occurred within the most recent time window, Comparing the created temporary failure classification tree with a failure classification tree (registered failure classification tree) corresponding to the failure cause determination rule in operation, and updating the failure cause determination rule used for operation based on the comparison result Provide a mechanism to have.

  According to the present invention, a failure cause determination rule used for operation can be automatically optimized according to a change in operation status.

The figure which shows the system configuration example of a failure cause analysis system. The figure explaining the specific example of the event table which log DB hold | maintains. The figure explaining the failure classification tree and failure cause determination rule table which failure cause determination rule DB holds. The figure explaining the failure node table which failure cause determination rule DB hold | maintains. The figure which shows the system configuration example of a failure cause determination rule verification computer. The flowchart which shows the outline | summary of a failure cause analysis process. The figure which shows the example of a feature table of an event block. The flowchart which shows the example of an execution procedure of the scheduling process in a failure cause determination rule. The flowchart which shows the example of an execution procedure of the verification / update process in a failure cause determination rule. The figure explaining the concept of the update process of a failure cause determination rule. The figure explaining the specific example of an update process. The figure explaining the concept of the addition process of a failure cause determination rule. The figure explaining the concept of the deletion process of a failure cause determination rule. The figure explaining the concept of the failure cause determination rule integration process. The figure explaining the specific example of an integration process. The figure explaining the concept of the division | segmentation process of a failure cause determination rule. The figure explaining the specific example of a division | segmentation process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the contents of the apparatus configuration and processing operation described later are examples for explaining the invention. The present invention also includes combinations of device configurations described later, combinations of device configurations described below and known technologies, and combinations of a part of device configurations described below and known technologies.

(Overall configuration of failure cause analysis system)
FIG. 1 shows a configuration example of a failure cause analysis system in which the failure cause determination rule verification computer 107 is installed. The failure cause analysis system shown in FIG. 1 includes a monitoring target server group 101, a monitoring server 102, a log database (DB) 103, a failure cause determination rule generation computer 104, a failure cause analysis computer 105, and a failure cause determination rule. It has a DB 106, a failure cause determination rule verification computer 107, a recovery procedure manual database (DB) 108, and a recovery procedure manual browsing computer 109.

  Among these, the monitoring server 102 provides a function of monitoring the state (life and death) of the monitoring target server group 101 and generating an event corresponding to the state. Events generated by the monitoring server 102 are stored in a log database (DB) 103. The failure cause determination rule generation computer 104 provides a function of reading an event log from the log DB 103 and generating a failure cause determination rule. The failure cause determination rule generated by the failure cause determination rule generation computer 104 is stored in the failure cause determination rule DB 106. The failure cause analysis computer 105 provides a function of analyzing an event based on the failure cause determination rule stored in the failure cause determination rule DB 106 and specifying a recovery procedure manual for the failure. The failure cause determination rule verification computer 107 analyzes the event generated by the monitoring server 102 and verifies the validity of the failure cause determination rule stored in the failure cause determination rule DB 106. The recovery procedure database (DB) 108 stores documents relating to a recovery procedure at the time of failure. In this document, not only manuals (whether hardware or software) that describe troubleshooting in the event of a failure, but also a record of the maintenance staff's response to past failures, reports, and other failures are recovered. Documentation on the procedure for this is also included. The recovery procedure manual browsing computer 109 provides a function of displaying the recovery procedure manual identified by the failure cause analysis computer 105 on the screen.

(Specific example of event table)
FIG. 2 shows a specific example of the event table 200 stored in the log DB 103. The event table 200 includes an identifier (ID) 201 that uniquely identifies an event, an occurrence date and time 202 that specifies the date and time when the event occurred, and an event 203 that is a set of attribute values of individual events. In the case of this embodiment, the attributes of the event 203 are defined by <type>, <source>, <event number>, <user>, and <computer>. Of these, <Type> indicates the importance of the event. <Source> indicates the source of the process or application that generated the event. <Event number> indicates a number for identifying the content of the event. <User> indicates the user who is executing the process or application that is the source of the event. <Computer> indicates a server in the monitoring target server group 101 that is an event generation source.

(Specific example of failure cause determination rule DB)
FIGS. 3A and 3B show a configuration example of the failure cause determination rule DB 106. FIG. The failure cause determination rule DB 106 is a DB that stores a failure cause determination rule registered in the failure cause analysis computer 105 and information related thereto. The failure cause determination rule DB 106 includes a registered failure classification tree 300, a failure cause determination rule table 310, and a failure node table 320.

  The registered failure classification tree 300 is created when a failure cause determination rule registered in the failure cause analysis computer 105 is generated. In the registered fault classification tree 300, faults are classified based on features that are commonly shared by a set of one or more events (hereinafter referred to as “event blocks”) that occurred at the time of the fault, and are represented as a classification tree. A node of the registered failure classification tree 300 is called a failure node. Faults classified into the same fault node are considered to be faults caused by the same fault cause because the generated events and the manner of occurrence are similar.

  The failure cause determination rule table 310 is a data table that stores failure cause determination rules registered in the failure cause analysis computer 105. The failure cause determination rule table 310 includes a failure node 311 of the registered failure classification tree 300 and a failure cause determination rule 312 that is applied to a failure classified as a target failure node. The failure cause determination rule 312 includes one or more determination events 313, a determination time 314, and a recovery procedure manual 315. The determination event 313 is a set of event attributes that characterize the target failure node. The determination time 314 is a time interval at which an event that satisfies the determination event 313 occurs. The recovery procedure manual 315 is a document displayed on the recovery procedure manual browsing computer 109 when an event that satisfies the determination event 313 occurs within the determination time 314.

  When a plurality of determination events 313 are specified for one failure node 311, the determination events 313 appear in the order described in the failure cause determination rule table 310. In the case of (2) in FIG. 3A, as the failure cause determination rule 312 for the “failure node 1-1”, the attribute values of (“alert”, “process71”, “80”, “user2”, “server9”) If an event with an attribute value of ("*", "process39", "*", "user4", "server8") occurs within the determination time "2 minutes 9 seconds" after the occurrence of an event with A rule of displaying “procedure A.doc” on the recovery procedure browsing computer 109 is set. Here, the attribute value “*” means that the value is indefinite and indicates that an arbitrary value can be taken.

  The failure node table 320 stores event blocks used as training data when the registered failure classification tree 300 is constructed. The failure node table 320 includes a failure node 311, an event block 321 classified as the failure node, and an event 203 included in the event block 321.

(Configuration example of failure cause determination rule verification computer)
FIG. 4 shows a configuration example of the failure cause determination rule verification computer 107. The failure cause determination rule verification computer 107 includes a computer main body 400, an input device 430, a display device 431, and a communication device 432. The communication device 432 communicates with the monitoring server 102, the log DB 103, and the failure cause determination rule DB 106.

  The computer main body 400 includes a CPU 401 that performs data calculation, a ROM 402, a RAM 410, a hard disk drive 420, a CPU bus 407 that realizes data transfer between these devices, and interfaces 403 to 406 that couple these devices to the CPU bus 407. The

  The RAM 410 has at least an execution area for the failure cause determination rule verification program 411 for causing the CPU 401 to perform arithmetic processing and a work area 412 for storing data to be temporarily generated at the time of verification. The storage area of the hard disk drive 420 temporarily stores a program storage unit 421 as a storage area for the failure cause determination rule verification program and data acquired from the monitoring server 102 and the failure cause determination rule DB 106. At least the data storage unit 422 is secured.

(Failure cause analysis operation)
FIG. 5 shows an outline of the failure cause analysis process of the entire failure cause analysis system.
(Step 501)
The failure cause determination rule generation computer 104 acquires an event log from the log DB 103, generates a failure cause determination rule, and stores it in the failure cause determination rule DB. Here, the failure cause determination rule is created in the order of (1) creation of a failure classification tree, (2) discovery of frequent event patterns, and (3) recovery procedure manual search.

(1) Creation of Failure Classification Tree The failure cause determination rule generation computer 104 classifies events acquired from the log DB 103 by failure. Events that are classified by failure are called event blocks. Next, the failure cause determination rule generation computer 104 extracts features from each event block, performs unsupervised clustering based on the extracted features, and constructs a classification tree. This classification tree corresponds to the registered failure classification tree 300 of the failure cause determination rule DB 106. As a clustering method in this case, there is a concept clustering COBWEB described in Non-Patent Document 1.

(2) Discovery of frequent event patterns The failure cause determination rule generation computer 104 finds one or a plurality of frequent events in a plurality of event blocks classified as a failure node of a designated classification tree. Furthermore, when there are a plurality of frequent events, the order in which the frequent events appear and the time interval are obtained. These correspond to the determination event 313 and the determination time 314 in the failure cause determination rule 312 of the failure cause determination rule table 310 of the failure cause determination rule DB 106.

  FIG. 6 shows a configuration example of a feature table 600 for event blocks created by extracting features of each event block. The feature table 600 includes an event block ID 601 that identifies an event block, and an attribute list 602 that is a feature for each event block. The attribute list 602 is configured with an attribute value that appears most frequently in the event block and an attribute value that appears next frequently for each attribute constituting the event 203. For this reason, two attribute values are assigned to each attribute of “type”, “source”, “event”, “user”, and “computer”.

(3) Recovery Procedure Manual Search The failure cause determination rule generation computer 104 generates a search key based on the attribute value of the frequent event obtained in (2). For example, a combination of attribute values that appears most frequently for each of five attributes, that is, “type”, “source”, “event”, “user”, and “computer” is set as a search key. Next, the failure cause determination rule generation computer 104 searches the recovery procedure manual DB 108 using the generated search key, and acquires an appropriate recovery procedure manual. The recovery procedure document here corresponds to the recovery procedure document 315 in the failure cause determination rule 312 of the failure cause determination rule table 310 (FIG. 3A).

(Step 502)
When the failure cause analysis computer 105 detects that the failure cause determination rule DB 106 has been updated by the failure cause determination rule generation computer 104, the failure cause determination rule DB 312 of the failure cause determination rule table 310 (FIG. 3) -1) is acquired and registered.

(Step 503)
The monitoring server 102 monitors the monitoring target server group 101. When the monitoring server 102 finds an abnormality caused by a failure in the servers in the monitoring target server group 101, the monitoring server 102 generates an event corresponding to the state of the corresponding server. The monitoring server 102 stores the generated event in the log DB 103 and transmits the event to the failure cause analysis computer 105 and the failure cause determination rule verification computer 107.

(Step 504)
The failure cause analysis computer 105 executes matching processing between the received event and the failure cause determination rule 312. If any of the failure nodes registered in the failure cause determination rule 312 matches the received event, the failure cause analysis computer 105 uses the recovery procedure manual 315 registered for the injured node whose match is confirmed. Obtained from the DB 108 and transmitted to the recovery procedure manual browsing computer 109.

(Step 505)
The recovery procedure manual browsing computer 109 displays the recovery procedure manual 315 received from the failure cause analysis computer 105 on the display device.

(Step 506)
Upon receiving an event from the monitoring server 102, the failure cause determination rule verification computer 107 creates a failure cause determination rule from the set of events within the set time window, and updates the failure cause determination rule DB 106. Details of this processing will be described later.

(Step 507)
When the failure cause analysis computer 105 detects that the failure cause determination rule verification computer 107 has updated the failure cause determination rule DB 106, the failure cause analysis computer 105 acquires the failure cause determination rule 312 from the failure cause determination rule DB 107 and currently uses it. Replace with failure cause determination rule.

(Failure cause determination rule verification operation)
7A and 7B show an overview of the failure cause determination rule verification / update process executed through the failure cause determination rule verification program 411. FIG. First, FIG. 7-1 shows the contents of the scheduling process of the verification process.

(Detailed operation of scheduling)
(Step 700)
Execution of the failure cause determination rule verification program 411 is started when the failure cause determination rule generation computer 104 detects an update of the failure cause determination rule DB 106 via the communication device 432.

(Step 701)
The failure cause determination rule verification computer 107 calculates the boundary time tb of the failure occurrence time interval. The boundary time tb is a time interval that is considered not to occur if a failure due to the same failure does not occur within the boundary time tb of the time interval after the last failure. The occurrence time of the failure is the occurrence date and time of the first event of the corresponding event block. This boundary time tb is determined by the following procedures (1) to (3).

(1) Acquisition of registered failure classification tree The failure cause determination rule verification computer 107 acquires the registered failure classification tree 300, the failure cause determination rule table 310, and the failure node table 320 from the failure cause determination rule DB 106 via the communication device 432. . Based on these pieces of information, the failure cause determination rule verification computer 107 calculates the time difference between the occurrence date and time t0 of the first event and the occurrence date and time of the last event used as training data when the registered failure classification tree 300 is constructed. To do. The time difference is set as a classification tree construction time Δ that gives the time range of the event when the failure cause determination rule verification computer 107 creates the failure classification tree.

(2) Failure occurrence time interval of verification failure node The failure cause determination rule verification computer 107 sets the failure node 311 in the failure cause determination rule table 310 in which the failure cause determination rule 312 is set as a verification target. The failure node as the verification target is hereinafter referred to as a “verification failure node”. Thereafter, the failure cause determination rule verification computer 107 accesses the failure node table 320 and calculates a failure occurrence time interval for the event block 321 corresponding to the verification failure node.

(3) Calculation of the boundary time tb The failure cause determination rule verification computer 107 calculates the failure that occurred after tb from the occurrence of the failure classified as a certain failure node from the distribution of the occurrence time intervals obtained in (2). A hypothesis test is used to determine a boundary time tb that is a boundary value such that “is classified as a failure node” is rejected with a probability of 1% significance level.

(Step 702)
The failure cause determination rule verification computer 107 acquires events from the date and time (t0 + tb) to the date and time (t0 + Δ) from the log DB 103 via the communication device 432.

(Step 703)
The failure cause determination rule verification computer 107 creates a temporary failure classification tree using the event acquired in step 702 as training data. Here, the temporary failure classification tree is a failure classification tree temporarily created by the failure cause determination rule verification computer 107, and is created by the same method as the registered failure classification tree 300. Also in the case of this temporary failure classification tree, a failure cause determination rule table and a failure node table are created at the same time as in the failure cause determination rule DB 106. The created temporary failure classification tree, corresponding failure cause determination rule table, and failure node table are stored in the work area 410.

(Step 704)
The failure cause determination rule verification computer 107 calculates the verification start date and time tvs. The verification start date and time tvs is t0 + Δ + tb.

(Step 705)
The failure cause determination rule verification computer 107 receives an event transmitted from the monitoring server 102 that has detected a failure in the monitoring target server group 101 via the communication device 432.

(Step 706)
The failure cause determination rule verification computer 107 compares the received event occurrence date te with the verification start date tvs. If the event occurrence date and time te is equal to or less than the verification start date and time tvs, the failure cause determination rule verification computer 107 executes step 710. If the event occurrence date and time te is greater than the verification start date and time tvs, the failure cause determination rule verification computer 107 executes step 707.

(Step 707)
The failure cause determination rule verification computer 107 verifies the validity of the failure cause determination rule 312 set in the failure node to be verified in the failure cause determination rule table 310 of the failure cause determination rule DB 106, and if necessary, The contents of the failure cause determination rule DB 106 are updated. Details of this processing will be described later.

(Step 708)
The failure cause determination rule verification computer 107 sets a new verification start date and time tvs. The new verification start date and time tvs is obtained as follows. First, for the failure node of the temporary failure classification tree created in step 709, the failure cause determination rule verification computer 107 uses the same method as in steps 701 (2) and (3) to determine the boundary time of the failure occurrence time interval. tb 'is calculated. Next, the failure cause determination rule verification computer 107 calculates tvs + tb ′ and sets this as a new verification start date and time tvs.

(Step 709)
The failure cause determination rule verification computer 107 acquires events that occurred in the time range (tvs−Δ to te) from the failure node table of the current temporary failure classification tree stored in the work area 412. Next, the failure cause determination rule verification computer 107 creates a new temporary failure classification tree, a failure cause determination rule table, and a failure node table using the acquired events as training data, and stores them in the work area 412. Thereafter, the failure cause determination rule verification computer 107 deletes the current temporary failure classification tree, the corresponding failure cause determination rule table, and the failure node table.

(Step 710)
The failure cause determination rule verification computer 107 creates or updates an event block from the received event, and updates the temporary failure classification tree. At the same time, the failure cause determination rule verification computer 107 also updates the failure cause determination rule table and the failure node table corresponding to this temporary failure classification tree.

(Detailed operation of the verification process)
Next, the detailed operation of the verification / update process shown in FIG.
(Step 750)
The failure cause determination rule verification computer 107 compares the temporary failure classification tree with the registered failure classification tree 300 of the failure cause determination rule DB 106. That is, the temporarily generated classification tree is compared with the classification tree in operation. If the configurations of the two classification trees match, the failure cause determination rule verification computer 107 executes step 760. On the other hand, if they do not match, the failure cause determination rule verification computer 107 executes step 770. Correspondence between the registered fault classification tree and the temporary fault classification tree is performed as follows. The failure event blocks used as training data when creating the registered failure classification tree are classified by both the registered classification tree and the temporary failure classification tree. The failure node corresponding to the failure node in the registered classification tree and the temporary failure classification tree in which the same event block is classified is set as the corresponding failure node. As a result, it is possible to determine which fault node in the temporary fault classification tree corresponds to the fault node in the registered fault classification tree.

(Step 760)
The failure cause determination rule verification computer 107 determines whether or not a new failure that does not exist in the corresponding failure node of the registered failure classification tree is classified into the verification failure node of the temporary failure classification tree. If there is no failure that exists only in the temporary failure classification tree, the failure cause determination rule verification computer 107 executes step 761. On the other hand, if there is a fault that exists only in the temporary fault classification tree, the fault cause determination rule verification computer 107 executes step 762.

(Step 761)
The failure cause determination rule verification computer 107 updates the failure cause determination rule DB 106. That is, the failure cause determination rule verification computer 107 deletes the failure cause determination rule corresponding to the verification failure node from the failure cause determination rule table 310. Note that the temporary failure classification tree here is created after the boundary time tb of the time interval after the registered failure classification tree is created. Therefore, if a failure classified into the same failure node does not occur, it can be determined that a failure classified as this failure node will not occur in the future. For this reason, there is no problem even if the failure cause determination rule set in the verification failure node is deleted.

(Step 762)
The failure cause determination rule verification computer 107 generates a failure cause determination rule for the verification failure node of the temporary failure classification tree by the method described in steps 501 (2) and (3). The failure cause determination rule verification computer 107 compares the generated failure cause verification rule of the temporary failure classification tree with the failure cause determination rule of the failure cause determination rule table 310. If they do not match, step 763 is performed. Run.

(Step 763)
The failure cause determination rule verification computer 107 updates the failure cause determination rule DB 106. That is, the failure cause determination rule verification computer 107 replaces the failure cause determination rule set in the verification failure node with the failure cause determination rule created in step 762.

  FIGS. 8A and 8B are diagrams illustrating the failure cause determination rule update process. (1) in FIG. 8A is a registered failure classification tree 800 created from events that occurred from January to March 2009. For the “failure node 1-1” 801 to be verified, FIG. 2 (A-1) failure cause determination rule 820 is set. (2) in FIG. 8A is a temporary classification tree 810 created from an event that occurred from October to December 2009. This “failure node 1-1 ′” 811 is shown in FIG. The failure cause determination rule 830 of B-1) is generated. At this time, the configurations of the registered classification tree 800 and the temporary classification 810 are the same. Further, all event blocks created from events generated from January to March 2009 classified as “failure node 1-1” are classified into temporary failure classification tree 810, “failure node 1-1” 811 ′ and “ It is assumed that the failure node 1-1 ′ ”811 corresponds.

  At this time, when the failure cause determination rule 820 of the “failure node 1-1” 801 and the failure cause determination rule 830 of the “failure node 1-1 ′” 811 are compared, the attribute “computer” of the second determination event The attribute value is “server8” 821 for “failure node 1-1”, whereas “server25” is different for “failure node 1-1 ′”. In the above case, the failure cause determination rule 820 of “failure node 1-1” 801 registered in the failure cause determination rule table 310 is replaced with the failure cause determination rule 830 of “failure node 1-1 ′” 811. In association with this, the failure node table 320 is also updated.

(Step 770)
The failure cause determination rule verification computer 107 obtains a difference between the temporary failure classification tree and the registered failure classification tree. If there is a failure node that is not in the registered failure classification tree 300 in the temporary failure classification tree, the failure cause determination rule verification computer 107 executes step 771. If the failure node that existed in the registered failure classification tree 300 does not exist in the temporary failure classification tree, the failure cause determination rule verification computer 107 executes Step 772. When a plurality of failure nodes existing in the registered failure classification tree 300 are grouped into one failure node in the temporary failure classification tree, the failure cause determination rule verification computer 107 executes step 773. When one failure node in the registered failure classification tree 300 is divided into a plurality of failure nodes in the temporary failure classification tree, the failure cause determination rule verification computer 107 executes step 774.

(Step 771)
The failure cause determination rule verification computer 107 updates the failure cause determination rule DB 106. That is, the failure cause determination rule verification computer 107 creates a failure cause determination rule for the new failure node added to the temporary failure classification tree according to the steps (2) and (3) of step 501. The failure cause determination rule is added to the failure cause determination rule table 310.

  FIG. 9 is a diagram for explaining failure cause determination rule addition processing. (1) in FIG. 9 is a registered failure classification tree 900 created from events that occurred from January to March 2009, and (2) in FIG. 9 was created from events that occurred from October to December 2009. This is a temporary failure classification tree 910. Comparing the registered failure classification tree 900 and the temporary failure classification tree 910, it can be seen that the registered failure classification tree 900 does not exist, but the temporary failure classification tree 910 includes the “failure node 3 '” 911.

  At this time, the failure cause determination rule verification computer 107 creates a failure cause determination rule for the “failure node 3 ′” 911 and registers it in the failure cause determination rule table 310. In connection with this registration, the failure cause determination rule verification computer 107 also updates the registered failure classification tree 300 and the failure node table 320.

(Step 772)
The failure cause determination rule verification computer 107 updates the failure cause determination rule DB 106. That is, the failure cause determination rule verification computer 107 does not exist in the temporary failure classification tree, but the failure cause determination rule 312 of the failure cause determination rule table 310 set in the verification failure node 311 of the registered failure classification tree 300 is displayed. delete.

  FIG. 10 is a diagram for explaining a failure cause determination rule deletion process. (1) in FIG. 10 is a registered failure classification tree 1000 created from events that occurred from January to March 2009, and (2) in FIG. 10 was created from events that occurred from October to December 2009. This is a temporary failure classification tree 1010. Comparing the registered fault classification tree 1000 and the temporary fault classification tree 1010, it can be seen that the fault node 1001 existing in the registered fault classification tree 1000 is not the temporary fault classification tree 1010.

  At this time, the failure cause determination rule verification computer 107 deletes the failure cause determination rule set in the “failure node 2” 1001 from the failure cause determination rule table 310. In connection with this deletion, the failure cause determination rule verification computer 107 also updates the registered failure classification tree 300 and the failure node table 320.

(Step 773)
The failure cause determination rule verification computer 107 updates the failure cause determination rule DB 106. That is, the failure cause determination rule verification computer 107 deletes the failure cause determination rule 312 set in the plurality of failure nodes of the registered failure classification tree 300 to be integrated from the failure cause determination rule table 310. Further, the failure cause determination rule verification computer 107 creates a failure cause determination rule for the failure node integrated with the temporary failure classification tree by the steps (2) and (3) in step 501, and the failure cause determination rule table Add to 310. However, the recovery procedure manual for the failure cause determination rule of the failure node of the created temporary failure classification tree utilizes the recovery procedure 315 of the failure cause determination rule 312 set in the failure node 311 of the registered failure classification tree 300 with a proven record. To do.

  FIGS. 11A and 11B are diagrams illustrating the failure cause determination rule integration processing. (1) in FIG. 11-1 is a registered failure classification tree 1100 created from an event that occurred from January to March 2009, and (2) in FIG. 11-1 occurred from October to December 2009. A temporary failure classification tree 1110 created from an event. The event blocks created from the events that occurred from January to March 2009 are classified into “failed node 1-1” 1101 and five “failed node 1-2” 1102. When these 15 event blocks are classified by the temporary failure classification tree 1101, all 15 event blocks are classified as “failure node 1 ′” 1101. That is, “failure node 1-1” and “failure node 1-2” are integrated into “failure node 1 ′”. The failure cause determination rules set in the “failure node 1-1” 1101 and the “failure node 1-2” 1102 are respectively 1120 in FIG. 11-2 (A-1) and (A- in FIG. 11-2). This corresponds to 1021 of 2).

  Further, the failure cause determination rule verification computer 107 creates a failure cause determination rule 1130 of “failure node 1 ′” 1111 from events that occurred from October to December 2009. However, in the recovery procedure manual 1031 of the failure cause determination rule 1030, more event blocks are classified and the recovery procedure manual 1021 of the failure cause determination rule 1020 corresponding to the “failed node 1-1” that has been proven. Is adopted.

  At this time, the failure cause determination rules 1120 and 1130 corresponding to the “failure node 1-1” 1101 and “failure node 1-2” 1102 are deleted from the failure cause determination rule table 310, and the newly created “failure node” 1 ′ ”failure cause determination rule 1030 is added. In association with this, the registered failure classification tree 300 and the event block table 320 are also updated.

(Step 774)
The failure cause determination rule verification computer 107 updates the failure cause determination rule DB 106. That is, the failure cause determination rule verification computer 107 deletes the failure cause determination rule 312 set in the failure node of the registered failure classification tree 300 to be divided from the failure cause determination rule table 310. Further, the failure cause determination rule verification computer 107 creates a failure cause determination rule for the plurality of failure nodes divided in the temporary failure cause classification tree by the steps (2) and (3) in step 501, and the failure cause It adds to the judgment rule table 310. However, the recovery procedure manual of the failure cause determination rule of the failure node of the created temporary failure classification tree utilizes the recovery procedure manual 315 of the failure cause determination rule 312 set for the failure node 311 of the registered classification tree 300 with a proven record. To do.

  12A and 12B are diagrams for explaining the failure cause determination rule division processing. (1) in FIG. 12-1 is a registered failure classification tree 1200 created from an event that occurred from January to March 2009, and (2) in FIG. 12-1 occurred from October to December 2009. A temporary failure classification tree 1210 created from an event. Of the event blocks created from events that occurred from January to March 2009, 10 are classified as “failed node 1-1” 1201. In the temporary failure classification tree 1210, 6 of these 10 event blocks are classified as “failure node 1-1-1 ′” 1211 and 4 are classified as “failure node 1-1-2 ′” 1212. . The failure cause determination rule set in the “failure node 1-1” 1201 is 1220 of (A-1) in FIG. 12-2, and “failure generated from an event that occurred from October to December 2009”. The failure cause determination rules of the node 1-1-1 ′ ”1211 and the“ failure node 1-1-2 ′ ”1212 are respectively 1221 in FIG. 12-2 (B-1) and 12B in FIG. 2) of 1231. However, in the attribute “recovery procedure” of the failure cause determination rule 1221 of “failure node 1-1-1 ′” 1211 in which more event blocks classified as “failure node 1-1” 1201 are classified, “failure node” 1-1 "1201 failure cause determination rule 1120 is assigned a proven failure recovery procedure document specified by the attribute" recovery procedure document ".

  At this time, the failure cause determination rule verification computer 107 deletes the failure cause determination rule 1220 corresponding to the “failure node 1-1” 1201 from the failure cause determination rule table 310, and “failure node 1-1-1 ′” 1211. And failure cause determination rules 1221 and 1231 corresponding to “failure node 1-1-2 ′” 1212 are added. In association with this, the registered failure classification tree 300 and the event block table 320 are also updated.

101 ... Monitoring target server group 102 ... Monitoring server 103 ... Log database (DB)
104 ... Failure cause determination rule generation computer 105 ... Failure cause analysis computer 106 ... Failure cause determination rule DB
107 ... Failure cause determination rule verification computer 108 ... Recovery procedure manual database (DB)
109 ... Recovery procedure manual browsing computer

Claims (10)

A first processing unit that acquires an event generated by a monitoring server based on a status of a server group to be monitored when a system failure occurs;
A second processing unit that classifies events that occur within a preset time window by failure and generates a temporary failure classification tree;
A third processing unit that compares the registered failure classification tree corresponding to the failure cause determination rule in operation with the temporary failure classification tree;
A failure cause determination rule verification device comprising: a fourth processing unit that updates the failure cause determination rule in operation based on a difference between the registered failure classification tree and the temporary failure classification tree. The second processing unit is
A time window for creating a temporary fault classification tree,
Based on the statistics of the appearance interval time of failures classified as failure nodes to be verified in the registered failure classification tree, if no failures classified into the same failure node occur within that time, similar failures will occur in the future. The failure cause determination rule verification device according to claim 1, wherein the failure cause determination rule verification device is calculated as a boundary time of a time interval in which it can be determined that no failure occurs. The third processing unit includes:
Failure node and temporary failure corresponding to each classification destination of the same event block when the event block used as training data for the registered failure classification tree is classified for each of the registered failure classification tree and the temporary failure classification tree The failure cause determination rule verification device according to claim 1, wherein the comparison between the registered failure classification tree and the temporary failure classification tree is performed by comparison with a failure node of the classification tree. The fourth processing unit includes:
Based on the difference between the failure cause determination rule generated from the failure node of the temporary failure classification tree and the failure cause determination rule in operation, further has a function of updating the failure cause determination rule in operation,
When a failure cause determination rule is created for a failure node in the temporary failure classification tree corresponding to the failure node in the registered failure classification tree used to set the failure cause determination rule in operation. When a difference is detected,
The failure cause determination rule verification apparatus according to claim 1, wherein the failure cause determination rule in operation is replaced with a failure cause determination rule created for a failure node in a temporary failure classification tree. The fourth processing unit includes:
If there is a failure node in the temporary failure classification tree that does not exist in the registered failure classification tree corresponding to the failure cause determination rule in operation,
The failure cause determination rule verification apparatus according to claim 1, wherein a failure cause determination rule for a failure node existing only in the temporary failure classification tree is created and additionally registered in a failure cause determination rule in operation. The fourth processing unit includes:
Based on the statistics of the appearance interval time of failures classified as failure nodes to be verified in the registered failure classification tree, if no failures classified into the same failure node occur within that time, similar failures will occur in the future. If a failure that is classified as a failed node to be verified does not occur even after the boundary time of the time interval that can be determined not to have passed,
The failure cause determination rule verification device according to claim 1, wherein the failure cause determination rule set for the failure node is deleted from the failure cause determination rule in operation. The fourth processing unit includes:
If the failure node that existed in the registered failure classification tree corresponding to the failure cause determination rule in operation does not exist in the temporary failure classification tree,
The failure cause determination rule verification according to claim 1, wherein the failure cause determination rule set for the failure node existing only in the registered failure classification tree is deleted from the failure cause determination rule in operation. apparatus. The fourth processing unit includes:
If the registered failure classification tree corresponding to the failure cause determination rule in operation is classified into a plurality of failure nodes, but is temporarily classified into one failure node in the temporary failure classification tree,
The failure cause determination rule set for multiple failure nodes in the registered failure classification tree is deleted from the failure cause determination rule in operation, and the failure cause determination rule created for the failure node in the temporary failure classification tree is in operation The failure cause determination rule verification device according to claim 1, wherein the failure cause determination rule is additionally registered. The fourth processing unit includes:
If the registered failure classification tree corresponding to the failure cause determination rule in operation is divided into one failure node, but temporarily divided into multiple failure nodes in the temporary failure classification tree,
The failure cause determination rule set for the failure node in the registered failure classification tree is deleted from the operating failure cause determination rule, and each failure cause determination created for each of the corresponding failure nodes in the temporary failure classification tree The failure cause determination rule verification device according to claim 1, wherein the rule is additionally registered in a failure cause determination rule. In the computer that functions as the failure cause determination rule verification device,
A first process for acquiring an event generated by a monitoring server based on a status of a server group to be monitored when a system failure occurs;
A second process for classifying events that occurred within a preset time window by fault and generating a temporary fault classification tree;
A third process for comparing the registered failure classification tree corresponding to the failure cause determination rule in operation and the temporary failure classification tree;
A computer program for executing a fourth process of updating the failure cause determination rule in operation based on a difference between the registered failure classification tree and the temporary failure classification tree.

Images