JP5514643B2 - Failure cause determination rule change detection device and program - Google Patents

Description

  The present invention relates to a device that can detect a change in a failure cause analysis rule used in a failure cause analysis system, for example, based on a past event log, and a program that implements the device in software.

  There are many system failures that have a major impact on corporate management and society. This is due to the complexity of cooperation with external IT services. As a result, the propagation reach of system failures has been widened, and one system failure has a major impact on society as a whole. Prompt and appropriate initial response is important to prevent the spread of damage due to such system failures.

  In view of this, a failure cause analysis system that supports initial response by presenting failure detection and an appropriate failure recovery procedure has been proposed (Patent Document 1). This system registers in advance a failure cause determination rule that associates an event pattern generated by a monitoring system when a specific system failure occurs with a recovery procedure for the failure, and the failure cause determination rule and the monitoring system generate By matching with event patterns (streams), it is possible to provide appropriate recovery procedures corresponding to failure detection.

  However, it is difficult for the user to describe the failure cause determination rule. Therefore, a method for automatically generating a failure cause determination rule from an event log generated by a monitoring system has been proposed (Patent Documents 2 and 3). These methods are based on generating a rule by identifying a characteristic event from events that occur when a system failure occurs and analyzing its behavior. Note that Patent Document 2 discloses a method of using the occurrence frequency of a specific event. Patent Document 3 describes a method of using an event occurrence pattern.

International Publication No. 2004/061681 JP 2008-41041 A JP 2006-4346 A

Fisher, Douglas H. “Knowledge acquisition via incremental clustering”, Machine Learning 2, 139-172, 1987

  However, during the operation of the IT service, there may be a change in which the fault cause determination rule once created becomes invalid. When this type of change occurs, it is required to correct the failure cause determination rule before a related failure occurs as much as possible. Such changes include the following cases.

(1) Deletion of IT service When an unnecessary IT service is deleted, a system failure related to the IT service does not occur thereafter. In this case, the failure cause determination rule related to this system failure is invalid.

(2) IT infrastructure configuration change When a network configuration change, hardware change, or other IT infrastructure configuration change is made, the failure cause determination rule having an attribute value of an event related to the change becomes invalid.

  Such changes are reflected in repeated failures. Therefore, a change can be detected by learning the event at that time. However, the time required to detect this change varies depending on the content of the failure. Failures that occur frequently can be detected in a short time, whereas failures that occur less frequently require time to be detected.

  If changes detected due to frequent failures also affect failure cause determination rules for failures that occur less frequently, ideally it will not only correct the failure cause determination rules for frequent failures, but also affect It is required to correct the failure cause determination rule before a failure having a low appearance frequency occurs.

  However, the conventional technology for automatically generating a failure cause determination rule does not provide a method for detecting the two changes described above and correcting all related failure cause determination rules.

  Therefore, the present inventor provides a mechanism for updating the failure cause determination rule according to the operation status. Specifically, when a system failure occurs, the event generated by the monitoring server based on the status of the monitored server group is acquired to create an event block, and the created event log is used as training data to create a temporary failure classification tree. A process for updating a set of objects, a process for selecting a temporary fault classification tree object having the highest weight in the set, and comparing the selected temporary fault classification tree object with a registered fault classification tree object, If they do not match, the change is predicted from the difference between the two, a temporary failure classification tree object based on the prediction is created, and the created temporary failure classification tree object is added to the temporary failure classification tree object set and selected. Providing a mechanism for replacing a registered fault classification tree object with a temporary fault classification tree object That.

  According to the present invention, it is possible to detect a change related to a failure cause determination rule. Further, according to the present invention, it is possible to predictively update a failure cause determination rule corresponding to a failure that has not been observed as a related failure occurs.

The figure which shows the system configuration example of a failure cause analysis system. The figure explaining the specific example of the event log which log DB hold | maintains. The figure which shows the specific example of the registration failure classification | category tree object which failure cause determination rule DB hold | maintains. The figure which shows the specific example of the failure classification tree and failure cause determination rule table which comprise the registration failure classification tree object. The figure which shows the specific example of the failure node table which comprises a registration failure classification | category tree object. The figure which shows the system structural example of a failure cause determination rule change detection computer. The figure which shows the example of a screen of a failure cause determination rule change detection program. The flowchart which shows the outline | summary of a failure cause analysis process. The flowchart which shows the outline | summary of a failure cause determination rule production | generation process. The figure explaining the specific example of an event block. The figure which shows the example of a feature table of an event block. The flowchart which shows the execution procedure of a failure cause determination rule change detection process. The figure explaining the temporary failure classification | category tree object set which a failure cause determination rule change detection apparatus produces | generates. The figure explaining the loss | disappearance detection of the failure node of a failure classification tree. The figure explaining the failure cause determination rule corresponding to the lost failure node. The figure explaining the change detection of the attribute value of a failure cause determination rule. The figure explaining the change of the attribute value in the failure cause determination rule in which the change of the attribute value was detected.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the contents of the apparatus configuration and processing operation described later are examples for explaining the invention. The present invention provides an arbitrary combination of device configurations and processing operations described later, a combination of adding a known technology to the device configuration and processing operations described later, and replacing a part of the device configuration and processing operations described later with known technologies. Combinations to include are also included.

(System configuration of failure cause analysis system)
FIG. 1 shows a configuration example of a failure cause analysis system in which the failure cause determination rule change detection computer 107 is installed. The failure cause analysis system shown in FIG. 1 includes a monitoring target server group 101, a monitoring server 102, a log database (DB) 103, a failure cause determination rule generation computer 104, a failure cause analysis computer 105, and a failure cause determination rule. It has a DB 106, a failure cause determination rule change detection computer 107, a recovery procedure manual database (DB) 108, and a recovery procedure manual browsing computer 109.

  Among these, the monitoring server 102 provides a function of monitoring the state (life and death) of the monitoring target server group 101 and generating an event corresponding to the state. Events generated by the monitoring server 102 are stored in a log database (DB) 103. The failure cause determination rule generation computer 104 provides a function of reading an event log from the log DB 103 and generating a failure cause determination rule. The failure cause determination rule generated by the failure cause determination rule generation computer 104 is stored in the failure cause determination rule DB 106. The failure cause analysis computer 105 provides a function of analyzing an event based on the failure cause determination rule stored in the failure cause determination rule DB 106 and specifying a recovery procedure manual for the failure.

  The failure cause determination rule change detection computer 107 provides a function of analyzing an event generated by the monitoring server 102 and detecting a change related to the failure cause determination rule stored in the failure cause determination rule DB 106. The detection here includes predictive detection.

  The recovery procedure database (DB) 108 stores documents relating to a recovery procedure at the time of failure. In this document, not only manuals (whether hardware or software) that describe troubleshooting in the event of a failure, but also a record of the maintenance staff's response to past failures, reports, and other failures are recovered. Documentation on the procedure for this is also included. The recovery procedure manual browsing computer 109 provides a function of displaying the recovery procedure manual identified by the failure cause analysis computer 105 on the screen.

(Specific example of event table)
FIG. 2 shows a specific example of the event table 200 stored in the log DB 103. The event table 200 includes an identifier (ID) 201 that uniquely identifies an event, an occurrence date and time 202 that specifies the date and time when the event occurred, and an event 203 that is a set of attribute values of individual events. In the case of this embodiment, the attributes of the event 203 are defined by <type>, <source>, <event number>, <user>, and <computer>. Of these, <Type> indicates the importance of the event. <Source> indicates the source of the process or application that generated the event. <Event number> indicates a number for identifying the content of the event. <User> indicates the user who is executing the process or application that is the source of the event. <Computer> indicates a server in the monitoring target server group 101 that is an event generation source.

(Specific example of failure cause determination rule DB)
3-1 to 3-3 show configuration examples of the failure cause determination rule DB 106. FIG. The failure cause determination rule DB 106 is a DB that stores a failure cause determination rule registered in the failure cause analysis computer 105 and information related thereto as a registered failure classification tree object 300. The registered failure classification tree object 300 includes a failure classification tree 310, a failure cause determination rule table 320, and a failure node table 330, as shown in FIG.

  The failure classification tree 310 is created when a failure cause determination rule registered in the failure cause analysis computer 105 is generated. In the fault classification tree 310, faults are classified based on features that are commonly shared by a set of one or more events (hereinafter referred to as “event blocks”) that occurred at the time of the fault, and are represented as a classification tree. A node in the failure classification tree 310 is called a failure node. Faults classified into the same fault node are considered to be faults caused by the same fault cause because the generated events and the manner of occurrence are similar. FIG. 3-2 (1) shows an example of the structure of the failure classification tree 310.

  The failure cause determination rule table 320 is a table that stores failure cause determination rules registered in the failure cause analysis computer 105. The failure cause determination rule table 320 includes a failure node 321 of the failure classification tree 310 and a failure cause determination rule 322 applied to the failure classified as the target failure node, as shown in (2) of FIG. 3-2. Is done. The failure cause determination rule 322 includes one or a plurality of determination events 323, a determination time 324, and a recovery procedure manual 325. The determination event 323 is a set of event attributes that characterize the target failure node. The determination time 324 is a time interval at which all events specified by the determination event 323 occur. The recovery procedure manual 325 is a document displayed on the recovery procedure manual browsing computer 109 when the determination event 322 occurs within the determination time 324.

  When a plurality of determination events 323 are specified for one failure node 321, it is assumed that determination events appear in the order specified in the failure cause determination rule table 320. In the case of (2) in FIG. 3B, as the failure cause determination rule 322 for “failure node 1-1”, the attribute values of (“alert”, “process71”, “80”, “user2”, “server9”) If a determination event having an attribute value (“*”, “process39”, “*”, “user4”, “server8”) occurs within the determination time “2 minutes 9 seconds” after the generation of the determination event having A rule for displaying “recovery procedure A.doc” on the recovery procedure browsing computer 109 is set. Here, the attribute value “*” means that the value is indefinite and indicates that an arbitrary value can be taken.

  The failure node table 330 stores event blocks used as training data when the failure classification tree 310 is constructed. An example of the failure node table 330 is shown in FIG. The failure node table 330 includes a failure node 321, an event block 331 classified as the failure node, and an event 203 included in the event block 321.

(Configuration example of failure cause determination rule change detection computer)
FIG. 4 shows a configuration example of the failure cause determination rule change detection computer 107. The failure cause determination rule change detection computer 107 includes a computer main body 400, an input device 430, a display device 431, and a communication device 432. The communication device 432 communicates with the monitoring server 102, the log DB 103, and the failure cause determination rule DB 106.

  The computer main body 400 includes a CPU 401 that performs data calculation, a ROM 402, a RAM 410, a hard disk drive 420, a CPU bus 407 that realizes data transfer between these devices, and interfaces 403 to 406 that couple these devices to the CPU bus 407. The

  The RAM 410 has at least an execution area for the failure cause determination rule change detection program 411 for causing the CPU 401 to perform arithmetic processing and a work area 412 for storing data to be temporarily generated at the time of calculation. The storage area of the hard disk drive 420 temporarily stores the program storage unit 421 as a storage area for the failure cause determination rule change detection program, and data acquired from the monitoring server 102 and the failure cause determination rule DB 106. The data storage unit 422 to be stored is secured at least.

  FIG. 5 shows an example of a GUI screen displayed on the display device 431 connected to the failure cause determination rule change detection computer 107. The failure cause determination rule change detection program screen 500 displays the minimum event block support rate input unit 501 for inputting the minimum event block support rate that should be satisfied by the failure cause determination rule 322 (FIG. 3-2), and the determination of the failure cause determination rule 322. The minimum effective attribute number input unit 502 for inputting the minimum number of valid attributes to be satisfied by the event 323, and a time that is a time range of events used as training data when a fault classification tree is temporarily created to detect a change The window width input unit 503 and a start button 505 for starting the failure cause determination rule detection process are configured.

  Here, the minimum event block support rate is applied to each failure node to be detected, and the number of event blocks to which the generated failure cause determination rule can be applied among all event blocks classified as the target failure node Means the percentage of In the case of the example of FIG. 5, this means that it must be applied to at least 60% of event blocks in order to be detected as the “failure cause determination rule”. In addition, an effective attribute means an attribute having an attribute value other than the undefined “*” attribute value. In the case of FIG. 5, the minimum value of the number of valid attributes is “5”.

(Failure cause analysis operation)
FIG. 6 shows an outline of the failure cause analysis process of the entire failure cause analysis system.

(Step 601)
The failure cause determination rule generation computer 104 acquires an event log from the log DB 103, generates a failure cause determination rule 322, and stores it in the failure cause determination rule DB. Details of this processing will be described later.

(Step 602)
When the failure cause analysis computer 105 detects an update of the failure cause determination rule DB 106 by the failure cause determination rule generation computer 104, the failure cause analysis rule acquisition computer 104 acquires the updated failure cause determination rule 322 from the failure cause determination rule table 320. . At this time, the acquired failure cause determination rule 322 is registered in the data storage unit 422.

(Step 603)
The monitoring server 102 constantly monitors the monitoring target server group 101. When the monitoring server 102 finds an abnormality caused by a failure in a certain server in the monitoring target server group 101, the monitoring server 102 generates an event from the state of the server. The monitoring server 102 stores the generated event in the log DB 103 and transmits the event to the failure cause analysis computer 105 and the failure cause determination rule change detection computer 107.

(Step 604)
The failure cause analysis computer 105 matches the received event with the failure cause determination rule 322. When the received event matches the failure cause determination rule 322, the failure cause analysis computer 105 acquires the recovery procedure manual 325 of the failure cause determination rule 322 from the recovery procedure manual DB 108 and transmits it to the recovery procedure manual browsing computer 109. To do. On the other hand, when the received event does not match the failure cause determination rule 322, the failure cause analysis computer 105 does nothing.

(Step 605)
The recovery procedure manual browsing computer 109 displays the recovery procedure manual 315 received from the failure cause analysis computer 105 on the display device.

(Step 606)
When the failure cause determination rule change detection computer 107 receives an event from the monitoring server 102, a temporary failure classification tree (hereinafter referred to as “temporary failure classification tree”) is generated from the event set that appears within the set time window. It is created and compared with the failure classification tree stored in the failure cause determination rule DB 106. The comparison here includes not only comparison of tree structures but also comparison of failure nodes to which each event is assigned. When a change is detected, the failure cause determination rule change detection computer 107 determines that a change related to the failure cause determination rule has occurred, and updates the registered failure classification tree object 300 in the failure cause determination rule DB 106. Details of this processing will be described later.

(Step 607)
When the failure cause determination rule verification computer 107 detects an update of the failure cause determination rule DB 106, the failure cause analysis computer 105 acquires the failure cause determination rule 312 from the failure cause determination rule DB 107, and currently uses the failure cause determination rule. Replace with By this replacement process, the failure cause analysis computer 105 can always analyze the event according to the latest failure cause determination rule.

(Error cause determination rule generation operation)
FIG. 7 shows an outline of a failure cause determination rule generation process performed by the failure cause determination rule generation computer 104.

(Step 701)
The failure cause determination rule generation computer 104 acquires an event from the event table 203 of the log DB 103.

(Step 702)
The failure cause determination rule generation computer 104 creates an event block table 800 (FIG. 8) that is a table of event blocks in which the events acquired in step 701 are grouped for each failure. In creating an event block, for example, the following rules are applied. When the time difference between the occurrence date 202 of an event and the occurrence date 202 of the previous event is within a given threshold, the event is classified into the previous event block. On the other hand, if the time difference is greater than or equal to the threshold, the event is classified into a new event block.

  FIG. 8 shows a specific example of the event block table 800 created based on the event table 200 of FIG. The event block table 800 includes an event block ID 801 that uniquely identifies an event block, an event ID 201 that identifies an event included in each event block, an occurrence date and time 202 that specifies the date and time when each event occurred, and an attribute that configures the event that occurred The event 203 is a set of values.

(Step 703)
The failure cause determination rule generation computer 104 extracts the feature of each event block based on the event block table 800 created in step 702, and creates the event block feature table 900 (FIG. 9). The feature here means an attribute that frequently appears in the event set classified into each event block.

  FIG. 9 shows a specific example of the event block feature table 900 created from the event block table 800 of FIG. The event block feature table 900 includes an event block ID 801 that identifies an event block, and an attribute list 901 that is a feature for each event block. In this example, the attribute list 901 includes attribute values that appear most frequently in the event block for each attribute of the event 203 and attribute values that appear next frequently. For example, in the case of FIG. 9, in the event set belonging to the event block with the event block ID “1”, it is understood that “error” is the most common and “fatal” is the second most common for the attribute “type”.

(Step 704)
The failure cause determination rule generation computer 104 clusters the event block feature table 900 created in step 703. In clustering, a failure classification tree 310 of the registered failure classification tree object 300 (FIG. 3-1) is created, and a failure node that classifies event blocks in the event block table 800 for each failure node of the created failure classification tree 310. Creation of the table 330 (FIG. 3-3) is performed.

  The event blocks classified into the same failure node in the failure classification tree 310 are similar in the manner of appearance of an event at the time of failure. Therefore, the event block can be regarded as a failure caused by the same failure cause. As the clustering algorithm, for example, conceptual clustering COBWEB described in Non-Patent Document 1 is used.

(Step 705)
The failure cause determination rule generation computer 104 finds an event that frequently appears in each failure node 321 of the failure node table 330 (FIG. 3C) created in step 704. In this frequent event, not all attribute values need to match, and some attribute values may be indefinite “*”. However, the minimum value to be satisfied of the number of attribute values other than the indefinite value “*” among the attributes constituting the event is given in advance as the minimum number of valid attributes (FIG. 5).

  The frequency of frequent events is determined by the number of event blocks that include the frequent events. The minimum number of event blocks to be satisfied by a frequent event is given in advance as the minimum support event block rate with respect to the total number of event blocks in the failed node (FIG. 5).

(Step 706)
The failure cause determination rule generation computer 104 discovers what pattern (the order of appearance) the frequent event obtained in step 705. The failure cause determination rule generation computer 104 stores the found frequent events in the determination event 323 of the failure cause determination rule table 320 in the order of appearance. When a plurality of determination events 323 are found, the failure cause determination rule generation computer 104 stores the maximum value of the time interval at which the plurality of patterns appear in the determination time 324 of the failure cause determination rule table 320.

(Step 707)
The failure cause determination rule generation computer 104 uses a part of the attribute value of the determination event 323 of the failure cause determination rule 322 obtained in step 706 as a search key, and generates a failure recovery procedure manual corresponding to the failure classified as the failure node. Obtained from the recovery procedure manual DB. The failure cause determination rule generation computer 104 stores the file name of the acquired failure recovery procedure manual in the recovery procedure manual 325 of the failure cause determination rule table 320 of FIG. In this step, the failure cause determination rule table 320 is completed. With the above steps, the registered failure classification tree object 300 is completed.

(Step 708)
The failure cause determination rule generation computer 104 registers the registered failure classification tree object 300 created up to step 707 in the failure cause determination rule DB 106.

(Details of operation for detecting changes in failure cause determination rules)
FIG. 10 shows an overview of a change detection process for a failure cause determination rule executed through the failure cause determination rule change detection program 411.

(Step 1000)
This program includes a support rate in the minimum event block support rate input unit 501 of the failure cause determination rule change detection program screen 500 (FIG. 5) displayed on the display device 431, a number of valid attributes in the minimum valid attribute number input unit 502, and time. After the time is input to the window width input unit 503, the operation is started when a click operation on the start button 505 is detected. The click operation here is input through a user operation on the input device 430 constituting the failure cause determination rule change detection computer 107. The failure cause determination rule change detection program 411 is executed by the failure cause determination rule change detection computer 107.

(Step 1001)
When the failure cause determination rule change detection program 411 detects a corresponding operation input, the failure cause determination rule change detection program 411 reads numerical values input to the minimum event block support rate input unit 501, the minimum valid attribute number input unit 502, and the time window width input unit 503, and the RAM 410 Are stored in the work area 412. Further, the failure cause determination rule change detection program 411 acquires the registered failure classification tree object 300 from the failure cause determination rule DB 106 via the communication device 432 and temporarily stores it in the data storage unit 422 of the hard disk drive 420. And stored in the work area 412 of the RAM 410.

(Step 1002)
The failure cause determination rule change detection program 411 creates a temporary failure classification tree object set 1100 (FIG. 11). The temporary failure classification tree object set 1100 is a set of objects temporarily created in order to detect a change in the failure cause determination rule being used for failure analysis.

  The failure cause determination rule change detection program 411 converts the registered failure classification tree object 300 acquired in step 1001 into a temporary failure classification tree object 1110 (FIG. 11) and stores it in the temporary failure classification tree object set 1100.

  The conversion from the registered failure classification tree object 300 to the temporary failure classification tree object 1110 is performed by converting each of the failure classification tree 310, the failure cause determination rule table 320, and the failure node table 330 constituting the registered failure classification tree object 300 into a temporary failure classification. This is realized by a process of substituting the fault classification tree 1120, fault cause determination rule table 1140, and fault node table 1150 of the tree object 1110, and a process of setting a value “1” in the weight 1130 of the temporary fault classification tree object 1110.

(Step 1003)
When the failure cause determination rule change detection program 411 receives an event from the monitoring server 102, it creates an event block and extracts its characteristics. The contents of the event block creation process and the feature extraction process are the same as the contents of step 702 and step 703 shown in FIG. Note that the event from the monitoring server 102 is acquired through the communication device 432.

(Step 1004)
The failure cause determination rule change detection program 411 updates all temporary failure classification tree objects 1110 in the temporary failure classification tree object set 1100 using the characteristics of the event block extracted in step 1003 as training data. Hereinafter, a method for updating the temporary failure classification tree object 1110 will be described in detail.

(Update of fault classification tree, fault cause determination rule table, fault node table)
First, the failure cause determination rule change detection program 411 acquires the time window width input to the time window width input unit 503 of the failure cause determination rule change detection program screen 500 of FIG. Next, the program acquires events included in the time window width from the failure node table 1150 of the temporary failure cause object 1110 starting from the current date and time. A failure classification tree 1120, a failure cause determination rule table 1140, and a failure node table 1150 are created from the acquired events in accordance with the procedure from step 702 to step 707 in FIG. At this time, the minimum event block support rate and the minimum number of valid attributes input on the failure cause determination rule change detection program screen 500 of FIG. 5 are considered.

(Weight update)
Next, the failure cause determination rule change detection program 411 calculates a temporary failure classification tree object selection weight 1130 using the following two indices. At this time, the temporary failure classification tree object 1110 having a weight less than or equal to a threshold given in advance is deleted.

(1) Category Utility Concept Clustering COBWEB creates a concept tree that is a classification tree so that the category utility defined in “Equation 3-3” on page 147 of Non-Patent Document 1 is maximized. . The failure cause determination rule change detection program 411 applies the numerical calculation to the failure classification tree 1120 of the temporary failure classification tree object 1110 and sets the calculated value as a weight.

(2) Degree of conformity to fault node However, the category utility is an index for evaluating the entire fault classification tree 1120 in the temporary fault classification tree object 1110, and may be the latest that may be accompanied by the new change created in step 1003. The effect of the event block is difficult to be reflected. For this reason, detection of changes is delayed with an index of only category units.

  Therefore, the degree of conformity between the latest event block and the failure node into which the event block is classified is calculated. In the latest event block, if the attribute value that appears most frequently among the i-th attributes is a1, and the attribute value that appears next is a2, the fitness can be calculated by equation (1).

Here, Nev is the total number of event blocks classified in the failed node. Natt is the number of event attributes. In the case of the example of this specification, as shown in FIG. 2, there are five types (type, source, event number, user, computer). ^{_{Also, P a (i, j)}} (a 1, a 2) , in the event the block that fall into disorder node, the attribute value of the most frequent among the j-th attribute of the event is at a _1, then frequent This means the probability that an event block whose attribute value is a ₂ will appear.

は上記のよう与えられる。 For example, there are 10 event blocks in the failure node, and for the attribute “type”, the attribute value pair that appears most frequently in the event block and the attribute value pair that appears next (“error”, “fatal”) If there are 5 (“error”, “warning”) and 2 (“fatal”, “warning”), then P _a ^{(i, j)} (a ₁ , a ₂ ) is

Is given above.

(Step 1005)
The failure cause determination rule change detection program 411 selects the temporary failure classification tree object 1110 having the heaviest weight 1130 from the temporary failure classification tree object set 1100 and compares it with the registered failure classification tree object 300. If the objects match, the failure cause determination rule change detection program 411 does nothing and returns to step 1003. On the other hand, if the objects do not match, the failure cause determination rule change detection program 411 executes Step 1006.

  The failure cause determination rule change detection program 411 first compares the failure classification trees with each other and determines whether or not there is a correspondence between the failure nodes. Next, the failure cause determination rule change detection program 411 compares the failure cause determination rules set in the corresponding failure nodes.

  At this time, prior to the comparison process, the failure cause determination rule change detection program 411 acquires event blocks for each failure node from the failure node table 330 of the registered failure classification tree object 300 and stores them in the temporary failure classification tree object 1110. Classification is performed according to the fault classification tree 1120. The temporary failure classification tree object is created based on the latest event block, but by comparing the event block to be classified with the registered failure classification tree object 300, whether or not the correspondence between the failure nodes has changed is compared. Is possible.

(Step 1006)
This step is executed when the registered failure classification tree object 300 and the temporary failure classification tree object 1110 do not match in step 1005.

  The failure cause determination rule change detection program 411 detects a difference in cause of a change that occurs between the temporary failure classification tree object 1110 selected in step 1005 and the registered classification tree object 300. When the cause of the change is predicted to be the disappearance of the failure node, the failure cause determination rule change detection program 411 executes Step 1008. On the other hand, when the cause of the change is predicted to be a change in the failure cause determination rule, the failure cause determination rule change detection program 411 executes Step 1009. If the cause of the change is none of these, the failure cause determination rule change detection program 411 does nothing and proceeds to step 1009.

  Here, the failure cause determination rule change detection program 411 detects (1) disappearance of the failure node and (2) change of the failure cause determination rule as follows.

(1) Failure Node Loss Detection FIG. 12-1 is a diagram illustrating a specific example of failure node loss detection.
(1) in FIG. 12A is a failure classification tree 1200 of the registered failure classification tree object 300 created from events from January to March 2009. As shown in (A-1) of FIG. 12-2, a failure cause determination rule 1220 is set in the “failure node 2” 1201.

  (2) in FIG. 12A is a failure classification tree 1210 of the temporary failure classification tree object 1110 created from events of October to December 2009. This temporary failure classification tree object 1110 is selected from the temporary failure classification tree object set 1100 in step 1005 for comparison with the registered classification tree object 300.

  First, the failure cause determination rule change detection program 411 compares the tree structure of the failure classification tree 1200 and the tree structure of the failure classification tree 1210. In the case of FIG. 12A, the “failure node 2” 1201 present in the failure classification tree 1200 does not exist in the failure classification tree 1210. In this case, the failure cause determination rule change detection program 411 predicts (determines) that the failure node has disappeared. This means that the event object belonging to “failure node 2” 1201 is not classified as any failure node in the failure classification tree 1210 of the temporary failure classification tree object 1110.

(2) Change Detection of Failure Cause Determination Rule FIG. 13-1 is a diagram for explaining a specific example of change detection of the failure cause determination rule.
(1) in FIG. 13A is a failure classification tree 1300 of the registered failure classification tree object 300 created from events from January to March 2009. In the “failure node 1-1” 1301, a failure cause determination rule 1320 is set as shown in (A-1) of FIG.

  (2) in FIG. 13A is a failure classification tree 1310 of the temporary failure classification tree object 1110 created from events from October to December 2009. As shown in (B-1) of FIG. 13-2, a failure cause determination rule 1330 is set in the “failure node 1-1 ′” 1311. This temporary failure classification tree object 1110 is selected from the temporary failure classification tree object set 1100 in step 1005 for comparison with the registered classification tree object 300.

  First, the failure cause determination rule change detection program 411 compares the failure classification tree 1300 with the failure classification tree 1310. In this example, the tree structure of the failure classification tree 1300 and the tree structure of the failure classification tree 1310 are the same. Therefore, the failure cause determination rule change detection program 411 classifies event blocks belonging to each failure node according to the characteristics of the failure classification tree 1310 of the temporary failure classification tree object, and determines which failure node belongs. When a failure node to which all event blocks belonging to a certain failure node are assigned is classified as a failure node in the failure classification tree 1310 of the temporary failure classification tree object, the failure cause determination rule change detection program 411 detects the failure of the classification source. It is determined that the node and the failure node of the classification destination are in a correspondence relationship.

  In the case of FIG. 13A, “failure node 1-1” 1301 stored in the failure node table 330 of the registered failure classification tree object 300, “failure node 1-1 ′” 1311 of the temporary failure classification tree object 110, and Is determined to be in a correspondence relationship.

  Next, the failure cause determination rule change detection program 411 compares the failure cause determination rule 1320 of the “failure node 1-1” 1301 with the failure cause determination rule 1330 of the “failure node 1-1 ′” 1311. When a difference is detected by this comparison, the failure cause determination rule change detection program 411 determines that the failure cause determination rule has changed.

  In the case of FIG. 13B, the attribute value of the attribute “computer” of the fifth event of the determination event constituting the failure cause determination rule 1320 is “server8” 1321. On the other hand, the attribute value of the attribute “computer” of the fifth event of the determination events constituting the failure cause determination rule 1330 is “server25” 1331. That is, the attribute value has changed. In this case, the failure cause determination rule change detection program 411 determines that the failure cause determination rule has changed.

(Step 1007)
This step is executed when it is determined in step 1006 that “the failure node has disappeared”.

  The failure cause determination rule change detection program 411 predicts the cause of “disappearance of the failure node”, and generates a temporary failure classification tree object 1110 based on the prediction.

  A method of generating the temporary fault classification tree object 1110 based on the prediction will be described based on the specific examples of FIGS. 12-1 and 12-2. In the example of FIG. 12A, “failure node 2” 1201 belonging to the failure classification tree 1200 of the registered failure classification tree object 300 has disappeared in the failure classification tree 1210 of the temporary failure classification tree object 1101. This can be predicted because the IT service related to “failure node 2” 1201 has been deleted. That is, it is considered that a failure due to the attribute values “process71” and “process39” of the attribute “source” of the failure cause determination rule 1220 set in the “failure node 2” 1201 does not occur in the future.

  In this case, the failure cause determination rule change detection program 411 generates an event whose attribute value of the attribute “source” is other than “process71” and “process39” from the failure node table 1150 of the temporary failure classification tree object 1110 selected in step 1005. get. Next, a failure classification tree, a failure cause determination rule, and a failure node table are created for these acquired events in accordance with the procedure from step 702 to step 707 in FIG. At this time, the minimum event block support rate and the minimum number of valid attributes input on the failure cause determination rule change detection program screen 500 of FIG. 5 are considered.

  Finally, a new temporary failure classification tree object 1110 is created, and the created failure classification tree, failure cause determination rule, and failure node table are stored in the failure classification tree 1120, the failure cause determination rule table 1140, and the failure node table 1150, respectively. Store. At this time, “1” is set to the weight 1130. The created temporary failure classification tree object 1110 is added to the temporary failure classification tree object set 1100.

(Step 1008)
This step is executed when it is determined in step 1006 that “change in failure cause determination rule”.

  The failure cause determination rule change detection program 411 predicts the cause of the change in the failure cause determination rule, and generates a temporary failure classification tree object 1110 based on the prediction.

  A method of generating the temporary fault classification tree object 1110 based on the prediction will be described based on the specific examples of FIGS. 13-1 and 13-2. In the example of FIG. 13A, “failure node 1-1” 1301 constituting the failure classification tree 1300 of the registered failure classification tree object 300 and “failure node 1” constituting the failure classification tree 1310 of the temporary failure classification tree object 1100. -1 ′ ”1311 corresponds. Further, comparing the failure cause determination rules 1320 and 1330 for the respective failure nodes, the attribute value of the attribute “computer” of the fifth determination event is changed from “server8” 1321 to “server25” 1331. This can be predicted because the computer of “server8” was replaced with the computer of “server25” due to a hardware failure or the like.

  In this case, the failure cause determination rule change detection program 411 acquires all events from the failure node table 1150 of the temporary failure classification tree object 1110 selected in step 1005, and the attribute value of the attribute “computer” is “server8”. Set the attribute value “server25” in the attribute “computer” of all events.

  Next, for these corrected events, a failure classification tree, a failure cause determination rule, and a failure node table are created according to the procedure from step 702 to step 707 in FIG. At this time, the minimum event block support rate and the minimum number of valid attributes input on the failure cause determination rule change detection program screen 500 of FIG. 5 are considered.

  Finally, a new temporary failure classification tree object 1110 is created, and the created failure classification tree, failure cause determination rule, and failure node table are stored in the failure classification tree 1120, failure cause determination rule 1140, and failure node table 1150, respectively. To do. At this time, “1” is set to the weight 1130. The created temporary failure classification tree object 1110 is added to the temporary failure classification tree object set 1100.

(Step 1009)
The failure cause determination rule change detection program 411 replaces the registered failure classification tree object 300 in the failure cause determination rule DB 106 with the temporary failure classification tree object 1110 selected in step 1005 via the communication device 432. For the replacement method, the failure classification tree 1120 of the temporary failure classification tree object 1110, the failure cause determination rule table 1140, and the failure note table 1150 are replaced with the failure classification tree 310 of the registered failure classification tree object 300, the failure cause determination rule 320, the failure, respectively. This is done by substituting into the node table 330.

  The temporary failure classification tree object 1110 selected in step 1005 includes the temporary failure classification tree object created in step 1007 and the temporary failure classification tree object created in step 1008.

(Summary)
The operation of this embodiment proceeds in the following order.
(1) The event generated by the monitoring server that monitors the system failure is sequentially acquired, and an event block in which the event is summarized for each failure is created (step 1003).
(2) An event block acquired within a predetermined time window width from the current processing time point is acquired as training data, and a set of temporary failure classification tree objects is updated based on the event block (step 1004).
(3) Select a temporary failure classification tree object having the heaviest selection weight from the set of updated temporary failure classification tree objects, and relate to the selected temporary failure classification tree object and the current failure cause determination rule. A change between the objects is detected by comparison with the registered failure classification tree object to be performed (step 1005).
(4) If it is detected in step 1005 that the two objects do not match, the cause of the change is predicted from the difference between the two objects, a temporary failure classification tree object based on the prediction is created, and a set of temporary failure classification tree objects (Steps 1006 to 1008).
(5) If it is detected in step 1005 that the two objects do not match, the registered fault classification tree object is determined by the temporary fault classification tree object selected in step 1005 or the temporary fault classification tree object created in step 1007 or 1008. Replace (step 1009).

  Thus, in the case of the present embodiment, the failure classification tree in use and the temporarily created failure classification tree are compared with each other, and the failure cause determination rule in use and the failure cause determination rule temporarily created Based on the comparison between them, the cause of the change is predicted, and when the cause can be predicted, specific attribute values of the determination event are rewritten all at once according to the content of the cause. If the cause cannot be predicted, the contents of the registered classification tree object are rewritten to the contents of the current temporary failure classification tree object.

  As a result, the content of the registered classification tree object can be changed at least when a change in the failure classification tree object is detected. In addition, when the cause of the change of the failure classification tree object can be predicted, the attribute values of all the determination events related to the predicted cause can be rewritten simultaneously. This means that when a failure with a high appearance frequency is related to a failure with a low appearance frequency, the failure cause determination rule corresponding to the failure with a low appearance frequency can be changed in advance in the occurrence cycle with a high appearance frequency.

101 ... Monitoring target server group 102 ... Monitoring server 103 ... Log database (DB)
104 ... Failure cause determination rule generation computer 105 ... Failure cause analysis computer 106 ... Failure cause determination rule DB
107 ... Failure cause determination rule change detection computer 108 ... Recovery procedure database (DB)
109 ... Recovery procedure manual browsing computer

Claims (5)

A first processing unit that sequentially acquires events generated by a monitoring server that monitors system failures and creates an event block that summarizes events for each failure;
Get event blocks acquired from the current processing time within a predetermined time window width as the training data, in the second processing unit for updating the set of the temporary fault classification tree objects that are temporarily generated based on the event block The temporary fault classification tree object includes (1) a fault classification tree in which faults are classified by characteristics, (2) a fault cause determination rule table storing fault cause determination rules, and (3) construction of the fault classification tree A second node that is configured with a failure node table that stores event blocks used as training data in (4) and an object selection weight ;
A temporary failure classification tree object having the highest selection weight is selected from the set of updated temporary failure classification tree objects, and the registered failure related to the selected temporary failure classification tree object and the current failure cause determination rule. A third processing unit that detects a change between objects by comparison with a classification tree object, and the registered failure classification tree object includes (1) a failure classification tree in which failures are classified according to characteristics; and (2) failure cause determination. A third processing unit composed of a failure cause determination rule table storing rules, and (3) a failure node table storing event blocks used as training data in the construction of the failure classification tree ;
If both objects do not match in the third processing unit, the cause of the difference between the two objects is predicted, a new temporary failure classification tree object is created based on the prediction, and the temporary failure classification tree object and a fourth processing unit to be added to the collection of,
If both objects do not match in the third processing unit, the registration is performed by the temporary fault classification tree object selected by the third processing unit or the temporary fault classification tree object created by the fourth processing unit. A fifth processing unit for replacing the fault classification tree object;
A failure cause determination rule change detection device having The second processing unit is configured to temporarily evaluate the classification tree based on the evaluation of the entire classification tree based on the category utility and the matching degree between the newly acquired event block and the failure node into which the event block is classified. The failure cause determination rule change detection device according to claim 1, wherein a weight for selecting an object is determined. When the failure node that existed in the failure classification tree that constitutes the registered failure classification tree object does not exist at the corresponding position of the failure classification tree that constitutes the temporary failure classification tree object, the fourth processing unit Is created, and a new temporary failure classification tree object is created based on an event set in which events related to the service are deleted from events that are training data of the temporary failure classification tree object. The failure cause determination rule change detection device according to claim 1. In the fourth processing unit, the content of the failure cause determination rule set for the failure node of the failure classification tree constituting the registered failure classification tree object is the content of the failure cause determination rule of the corresponding temporary failure classification tree object. If it is different from that, change the attribute of the event of the training data to the content after the change by predicting that the configuration of the system infrastructure has changed, and a new temporary failure based on the event block including the event after the change of the attribute The failure cause determination rule change detection device according to claim 1, wherein a classification tree object is created. On the computer,
A first process for sequentially acquiring events generated by a monitoring server that monitors system failures and creating an event block that summarizes events for each failure;
This is a second process for acquiring an event block acquired within a predetermined time window width from the current processing time point as training data and updating a set of temporary failure classification tree objects that are temporarily generated based on the event block . The temporary fault classification tree object includes: (1) a fault classification tree that classifies faults according to features; (2) a fault cause determination rule table that stores fault cause determination rules; and (3) construction of the fault classification tree. A second process consisting of a failure node table storing event blocks used as training data, and (4) weights for object selection ;
A temporary failure classification tree object having the highest selection weight is selected from the set of updated temporary failure classification tree objects, and the registered failure related to the selected temporary failure classification tree object and the current failure cause determination rule. A third process of detecting a change between objects by comparison with a classification tree object , wherein the registered fault classification tree object includes (1) a fault classification tree in which faults are classified by characteristics, and (2) fault cause determination rules. A third process consisting of a failure cause determination rule table storing (3) and a failure node table storing event blocks used as training data in the construction of the failure classification tree ;
If the third Oite in processing both objects do not match, to predict the cause of the differences between the two objects is generated, the one o'clock fault classification to create a new temporary fault classification tree object based on a prediction A fourth process to add to the set of tree objects ;
If the third Oite in processing both objects do not match, by the third processing with the selected one o'clock fault classification tree object or the fourth temporary fault classification tree objects created by processing A fifth process for replacing the registered failure classification tree object ;
A program that executes

Images