JP2011209908A - Rule creation device in failure cause analysis system and its program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a failure cause analysis system that creates a complete failure cause determination rule including an action phrase from an event log, which can be commonly applied to system failures having the same cause.SOLUTION: The rule creation device in a failure cause analysis system is configured to extract features from a plurality of events which have occurred in a system failure, to cluster the failures based on the features, to create a conditional phrase of a failure cause determination rule from the events when the failures occurred from the same cause included in the same failure cluster, to extract a retrieval keyword from the created conditional phrase, to retrieve a recovery procedure corresponding to the failure cluster, and to combine an action phrase defined as the retrieved recovery procedure with the created conditional phrase to create a complete failure cause determination rule.

Description

  The present invention relates to a device for generating a failure cause determination rule used in a failure cause analysis system based on a past event log and a program thereof.

  Delays in recovery work in the event of a system failure have a major impact on corporate performance and social infrastructure. In order to quickly recover from a system failure, it is important to respond to the initial action, that is, identify the cause of the failure early and determine the recovery procedure.

  At present, a failure cause analysis system has been proposed for the purpose of supporting quick recovery when a failure occurs (Patent Document 1). This analysis system uses a failure cause determination rule created and registered in advance by the user, mainly manually. When a failure occurs, this analysis system automatically analyzes the event generated by the monitoring server that monitors the IT infrastructure such as the server based on the failure cause determination rule created in advance, and responds to the analysis result. Appropriate recovery procedures are automatically presented to recovery personnel. The failure cause determination rule has a data structure in which an occurrence pattern of an event that occurs at the time of a specific failure is associated with a recovery procedure. Of the failure cause determination rules, a phrase that defines an occurrence pattern of an event that identifies a failure is called a “condition”, and a phrase that defines a recovery procedure for the failure is called an “action”.

  On the other hand, it is difficult to manually create a failure cause determination rule. For this reason, a method for automatically generating a condition phrase of a failure cause determination rule from an event log has been proposed (Patent Documents 2 to 4). Each patent document discloses a method for analyzing a behavior of an individual event when a failure occurs and obtaining a condition phrase of a failure cause determination rule. For example, Patent Literature 2 describes a method of using the occurrence frequency of a specific event. Further, Patent Document 3 describes a method of using an event occurrence pattern. Patent Document 4 describes a method of defining an event that characterizes a failure that has occurred by using the degree of abnormality of an attribute value compared with a normal value, and reflecting the result in a determination rule.

International Publication No. 2004/061681 JP 2008-41041 A JP 2006-4346 A JP 2009-217382 A

Fisher, Douglas H. “Knowledge acquisition via incremental clustering”, Machine Learning 2, 139-172, 1987

  In order to shorten the recovery time, it is necessary to determine the same appropriate response procedure for failures with the same cause of failure, and to review and improve the once determined response procedure. For this reason, the failure cause determination rule generated by the automatic generation technique can be commonly applied to failures with the same failure cause, and it is required to present an appropriate response procedure for the failure.

However, the above-described automatic generation technique of the failure cause determination rule has the following two problems.
(1) Guarantee of application to faults with the same cause of failure In the automatic technique for generating conditional phrases according to Patent Documents 2 to 4, when a fault occurs, a conditional phrase for the fault is created based on the individual events that occurred. However, these methods do not guarantee that the same response procedure can be applied to failures with the same cause of failure. In fact, even if the cause of the failure is the same, an event unrelated to the failure may occur for some reason in each failure. For this reason, a conditional phrase derived from an event that occurs for each failure may not be applicable to other failures.
(2) Generation of incomplete failure cause determination rule The automatic generation technology according to Patent Documents 2 to 4 analyzes the behavior of an event when a failure occurs and automatically generates only the condition phrase of the failure cause determination rule. No action phrase is generated. For this reason, in the conventional system, an action phrase needs to be created by the user himself / herself. However, the content of the created action phrase depends on the skill of the user, and an appropriate recovery procedure may not be selected.

  Therefore, the present inventor (1) clustered one or a plurality of events (event blocks) that occur due to a failure based on the characteristics (in this case, the failure and the event pattern of the failure belonging to the same cluster are similar). (2) Generate a condition clause for the failure cause determination rule based on events that frequently appear in event blocks belonging to the same cluster, and (3) Generated conditions A search key is generated based on the event attribute value specified in the phrase, and (4) the recovery procedure document searched using the generated search key is used as the action phrase of the failure cause determination rule. An invention characterized in that a failure cause determination rule is automatically generated.

  According to the present invention, a failure cause determination rule that can be applied in common to failures with the same cause of failure and that includes both a conditional phrase and an action phrase of the failure cause determination rule can be automatically generated. .

The figure which shows the system configuration example of a failure cause analysis system. The table figure explaining the specific example of the event log which log DB hold | maintains. The figure which shows the system structural example of a failure cause determination rule production | generation computer. The flowchart which shows the outline | summary of a failure cause analysis process. The figure explaining the specific example of a failure cause determination rule. The flowchart which shows the example of an execution procedure of a failure cause determination rule production | generation process. The figure which shows the example of GUI of the initial state in a failure cause determination rule production | generation computer. The figure which shows the example of GUI which displayed the failure cluster in the failure cause determination rule production | generation computer. The figure which shows the GUI example which displayed the frequent event in the failure cause determination rule production | generation computer. The figure which shows the example of GUI of failure cause determination rule preparation in the failure cause determination rule production | generation computer. The flowchart which shows the example of an execution procedure of a failure clustering process. The figure explaining the specific example of an event block. The figure explaining the specific example of the feature of an event block. The flowchart which shows the example of an execution procedure of the search process of a recovery procedure manual.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the contents of the device configuration and processing operation described below are examples for explaining the invention, and the present invention relates to an invention in which a known technology is combined with the device configuration and processing operation described later, and the device configuration and processing operation described later. It also includes inventions that partially replace known techniques.

(System configuration of failure cause analysis system)
FIG. 1 shows a configuration example of a failure cause analysis system that implements a failure cause determination rule generation device. The failure cause analysis system monitors the monitoring target server group 101 and the status (life and death etc.) of these servers and generates an event according to the status, and a log database that stores events generated by the monitoring server 102 (DB) 103, a failure cause analysis computer 104 that analyzes an event generated by the monitoring server 102 based on a failure cause determination rule, a manual such as hardware and software that describes troubleshooting when a failure occurs, and a maintenance person A recovery procedure database (DB) 105 in which documents related to recovery procedures in the event of a failure such as reports on recovery operations in the past are stored, and a failure cause analysis computer 104 analyzes events based on failure cause determination rules, Specify the recovery procedure manual, and the recovery procedure manual DB 105 The recovery procedure manual browsing computer 106 displays the acquired recovery procedure manual for browsing by the person in charge of fault recovery, and the failure cause determination rule generation computer 107 that acquires an event log from the log DB 103 and generates a failure cause determination rule. The

  FIG. 2 shows an example table of events stored in the log DB 103. The table includes an identifier (ID) 201 that uniquely identifies an event, an occurrence date and time 202 that identifies the time when the event occurred, and an event 203 that is a set of attribute values of individual events. In this embodiment, the attributes of the event 203 are defined as follows.

(<Type>, <Source>, <Event number>, <User>, <Computer>)
<Type> means the urgency of the event. <Source> means the source of the process, application, etc. that generated the event. <Event number> means a number that identifies the content of the event. <User> means a user who is executing the process or application that is the source of the event. <Computer> means a server in the monitoring target server group 101 that is an event generation source.

  FIG. 5 shows an example of the failure cause determination rule. In this example, an event with an attribute value tuple (*, process39, *, user4, server8) is followed by an event with an attribute value tuple (alert, process71, 80, user2, server9) within the judgment time of 2 minutes and 9 seconds. If it occurs, this means that the restoration procedure “XX System restoration procedure. Doc” is presented to the person in charge of restoration. Here, the attribute value “*” indicates that the value is indefinite. Further, in this failure cause determination rule, a combination of the event pattern 501 and the determination time 502 items is a rule condition phrase, and an item of the recovery procedure manual 503 corresponds to a rule action phrase.

  FIG. 3 shows a configuration example of the failure cause determination rule generation computer 107. The failure cause determination rule generation computer 107 includes a computer main body 300, an input device 330, a display device 331, and a communication device 332. Here, the communication device 332 communicates with the log DB 103, the recovery procedure manual DB 105, and the failure cause analysis computer 104. The computer main body 300 includes a CPU 301 that executes data operations, a ROM 302, a RAM 310, a hard disk drive 320, a CPU bus 307 that realizes data transfer between these devices, and interfaces 303 to 306 that couple the aforementioned devices and the CPU bus 307. Composed.

  The RAM 310 has at least an execution area for the failure cause determination rule generation program 311 for causing the CPU 301 to execute arithmetic processing and a work area 312 for storing data to be temporarily generated at the time of verification. Hereinafter, the “failure cause determination rule generation program 311” is also simply referred to as “program 311”. In addition, the storage area of the hard disk drive 320 stores a program storage unit 321 as a storage area for the failure cause determination rule generation program, and data storage for temporarily storing data acquired from the log DB 103 and the recovery procedure manual DB 105. The part 322 is at least secured.

  FIG. 7 and FIG. 10 show examples of GUI screens for condition input and calculation result output of the failure cause determination rule generation program displayed on the display device 331 connected to the failure cause determination rule generation computer 107. The failure cause determination rule generation program screen 700 includes a failure clustering tab 701 that displays the execution and results of failure clustering, and a rule creation tab 1001 that displays failure cause determination rules. These screens are displayed on the screen of the display device 331 through the program 311.

  The failure clustering tab 701 includes a region (failure clustering unit 710) that collectively displays information elements related to failure clustering processing, and a region (failure cluster detail unit 720) that collectively displays information elements related to individual failure clusters. ).

  The fault clustering unit 710 is obtained as a result of clustering faults, a time window unit input unit 711 that provides event block separation criteria, a clustering start button 712 that is clicked through a pointer on the screen when clustering event blocks, and the like. A fault classification tree display section 713 which is a fault classification tree. Here, the event block means a set of one or a plurality of events that occur at the time of one failure. In the case of the example in FIG. 7, the program 311 determines that an event caused by a failure is caused by the same failure as the previous event if the difference between the occurrence date and time and the occurrence date and time of the previous event is within 10 minutes. We do not see the event that occurred and classify it into the same event block. On the other hand, when the difference in occurrence date and time is 10 minutes or more, the program 311 regards the generated event as an event generated by a failure different from the previous event, and classifies it as a new event block.

  The failure cluster detail unit 720 displays frequent events included in a plurality of failures (hereinafter referred to as “failure clusters”) classified into nodes selected from the failure classification tree displayed in the failure classification tree display unit 713. A minimum valid attribute number input unit 721 for inputting a condition for calculation, a minimum support event block number input unit 722, a frequent event discovery button 723 that is clicked through a pointer on the screen when calculation of a frequent event is started, The event block table 730 displays information about event blocks included in the failure cluster selected by the failure classification tree display unit 713, and the frequent event table 740 displays information about frequent events. Here, the number of valid attributes is the number of attribute values other than “*” in the frequent event, and the minimum number of valid attributes is the minimum number of valid attributes that the frequent event should satisfy. Further, the minimum number of supported event blocks is the minimum number of event blocks that should contain frequent events. As described above, “*” indicates that the attribute value is indefinite.

  The event block table 730 displays a failure event block 731 for displaying the event block name in the designated failure cluster, an event 732 for displaying the attribute value of the event in the event block, and a label for the frequent event corresponding to each event. It is composed of frequent event labels 733. The frequent event table 740 includes a frequent event label 741 that displays a label that uniquely identifies a frequent event, an event 742 that displays an attribute value of the frequent event, and an event block that includes a frequent event among the event blocks in the event block table 730. This is composed of a number 743 of support event blocks for displaying the number.

  The rule creation tab 1001 includes a final failure cause determination rule unit 1010 that summarizes elements related to a failure cause determination rule that is finally registered in the failure cause analysis computer 104, and a determination of a condition phrase of the failure cause determination rule. The event pattern candidate unit 1030 that collects the elements to be performed and the recovery procedure manual search unit 1050 that collects the elements related to the determination of the action phrase of the failure cause determination rule.

  After determining the final failure cause determination rule, the failure cause determination rule unit 1010 displays a determination button 1011 that is clicked through a pointer on the screen when registering in the failure cause analysis computer 104, and the content of the failure cause determination rule. The failure cause determination rule table 1020 is displayed. The failure cause determination rule table 1020 includes an event pattern 1021 that displays a pattern in which an event appears, a determination time 1022 that is a time interval in which the pattern appears, and a recovery procedure when the event pattern appears within the determination time 1022 The recovery procedure manual 1023 for displaying the name of the recovery procedure document to be displayed on the document browsing computer 106.

  The event pattern candidate unit 1030 displays an event pattern table 1040 for displaying candidates for the event pattern 1021 and the determination time 1022 of the condition phrase of the failure cause determination rule, and selects a candidate selected from the event pattern table 1040 as the failure cause determination rule. When the event pattern 1021 and the determination time 1022 are determined, the determination button 1031 is clicked through a pointer on the screen. The event pattern table 1040 includes a candidate 1041 for displaying a number for uniquely identifying an event pattern candidate, a frequent event label 1042 for displaying a label of a corresponding frequent event, and an event pattern 1043 for displaying an event attribute value of the event pattern. And a determination time 1044 for displaying the determination time, and an event block support number 1045 for displaying the number of event blocks including the event pattern.

  The recovery procedure manual candidate unit 1050 includes a search keyword input unit 1052 that inputs a search keyword, a search button 1053 that is clicked through a pointer on the screen when searching the recovery procedure manual in the recovery procedure manual DB 105, Clicking through the pointer on the screen when determining the recovery procedure manual table 1060 for displaying the recovery procedure manual list and the recovery procedure manual selected from the recovery procedure manual table 1060 as the recovery procedure manual 1063 of the failure cause determination rule The determination button 1051 is configured. The recovery procedure manual table 1060 includes a recovery order document 1061 that displays the searched recovery procedure manual name and a score 1062 that displays the score of the searched recovery procedure manual.

(Failure cause analysis operation)
FIG. 4 shows an outline of a failure cause analysis process executed in the entire failure cause analysis system. First, the failure cause determination rule generation computer 107 acquires an event log from the log DB 103, generates a failure cause determination rule, and registers it in the failure cause analysis computer 104 (step 401). FIG. 5 shows an example of the failure cause determination rule generated by the failure cause determination rule generation computer 107.

  The monitoring server 102 monitors the monitoring target server group 101 and generates an event corresponding to the state (step 402). Thereafter, the monitoring server 102 transmits the generated event to the failure cause analysis computer 104, and simultaneously stores the generated event in the event log DB 103. Here, the generated event is a set of attribute values constituting the event 203 of the event log table shown in FIG.

  The failure cause analysis computer 104 receives the event generated by the monitoring server 102 (step 403), and executes a matching process between the registered condition phrase pattern of the failure cause determination rule and the appearance pattern of the received event (Step 404).

  If the event occurrence pattern matches the condition phrase of the failure cause determination rule, the failure cause analysis computer 104 acquires the recovery procedure manual for the failure cause determination rule from the recovery procedure manual DB 105 (step 406), and the recovery procedure. It is transmitted to the book browsing computer 106 (step 407). Thereafter, the recovery procedure browsing computer 106 displays the received recovery procedure manual (step 408).

  If the event appearance pattern does not match the condition phrase of the failure cause determination rule, the failure cause analysis computer 104 returns to step 402.

(Error cause determination rule generation operation)
(A) Outline of operation (step 600)
FIG. 6 shows an outline of the failure cause determination rule generation process executed through the failure cause determination rule generation program 311. The program 311 is a click operation on the clustering start button 712 after a unit time is input to the time window unit input unit 711 of the failure cause determination rule generation program screen 700 (FIG. 7) in the initial state displayed on the display device 331. It is started when is detected. The click operation here is input through a user operation on the input device 330 constituting the failure cause determination rule generation computer 107. The program 311 is executed by the failure cause determination rule generation computer 107.

(Step 601)
When the failure cause determination rule generation program 311 detects a corresponding operation input, the failure cause determination rule generation program 311 reads the numerical value or content input to the time window unit input unit 711 and stores it in the work area 312 of the RAM 310. Further, the failure cause determination rule generation program 311 executes the failure clustering process according to the processing procedure of the failure clustering process described in the failure cause determination rule generation program 311. Details of this processing will be described later.

(Step 602)
The failure cause determination rule generation program 311 displays the execution result of the failure clustering on the failure cause determination rule generation program screen 700 through the display device 331. FIG. 8 shows a specific example of a state in which the processing result of the failure clustering is displayed on the failure cause determination rule generation program screen 700. In the failure cluster display unit 713, the execution result of the failure clustering is displayed as a concept tree, and information on event blocks included in the selected failure cluster 714 that is a node of the concept tree is displayed in the event block table 730. In the case of the display example of FIG. 8, the failure cluster 714 (failure cluster 3) selected by the user is composed of failures that occurred on 2008/10/11, 2008/10/25, and 2008/12/02. It can be seen that three, two, and three events occurred at each failure.

(Step 603)
Next, the failure cause determination rule generation program 311 executes frequent event discovery processing. In this frequent event discovery process, a specific failure cluster 714 is selected by the user in the failure cluster display unit 731 constituting the failure cause determination rule generation program screen 700 (FIG. 8) displayed on the display device 331, and the selection is performed. Clustering is performed after numerical values are input to the input units (minimum valid attribute number input unit 721 and minimum supported event block number input unit 722) in a state where event block information regarding the failed cluster 714 is displayed in the event block table 730. The operation is started when a click operation on the start button 712 is detected. The selection, numerical input, and click operation here are input through a user operation on the input device 330 constituting the failure cause determination rule generation computer 107.

(Step 604)
When the failure cause determination rule generation program 311 detects the operation input, the failure cause determination rule generation program 311 inputs to the input units (the minimum valid attribute number input unit 721 and the minimum support event block number input unit 722) in the failure cause determination rule generation program screen 700 of FIG. The input numerical value and the content displayed in the event block table 730 are read and stored in the work area 312 of the RAM 310. Furthermore, the failure cause determination rule generation program 311 finds a frequent event from events in the event block table stored in the work area 312 according to the frequent event discovery process described in the failure cause determination rule generation program 311.

  Here, a frequent event is an event that appears at a frequency that satisfies the number of event blocks input to the minimum supported event block number input unit 722 among events included in the event block table 730. However, the frequent event does not need to completely match the attribute value of the event included in the event block table 730, and some attributes may be indefinite (*). At this time, the number of effective attributes of frequent events (the number of attribute values other than *) needs to satisfy the minimum effective attribute number of the minimum effective attribute number input unit 721.

(Step 605)
The failure cause determination rule generation program 311 displays the execution result of the frequent event discovery process on the failure cause determination rule generation program screen 700 through the display device 331. FIG. 9 shows a specific example in which the execution result of the frequent event discovery process is displayed in the event block table 730 and the frequent event table 740 constituting the failure cause determination rule generation program screen 700.

  In the frequent event table 740, the attribute values constituting the frequent event are displayed with the color, pattern, label 741 and the like uniquely identifying each frequent event with respect to the event 742. In the case of FIG. 9, the frequent events in the frequent event table 740 are displayed in numerical order of the number of support event blocks 743 corresponding to each frequent event (FIG. 9 is in descending order). Further, the event 732 of the event block table 730 is assigned the same color and pattern as those assigned to the frequent events in the frequent event table 740 and the corresponding frequent event label 733 is assigned. Thus, the user can intuitively and easily understand how the frequent events are distributed in the event block from the screen display. If the distribution method is unnatural, the user may change the values of the condition input parts (minimum valid attribute number input part 721 and minimum support event block number input part 722) to re-determine frequent events. .

(Step 606)
In the frequent event pattern discovery process in the failure cause determination rule generation program 311, the rule creation tab 1001 of the failure cause determination rule generation program screen 700 shown in FIG. 9 displayed on the display device 331 is clicked through a user operation on the input device 330. To start.

(Step 607)
When the failure cause determination rule generation program 311 detects the operation input, the failure cause determination rule generation program 311 reads the contents displayed in the event block table 730 and the frequent event table 740 constituting the failure cause determination rule generation program screen 700 shown in FIG. Store in the work area 312. Further, the failure cause determination rule generation program 311 finds a frequent event pattern from the frequent event label 733 of the event block table stored in the work area 312 according to the frequent event pattern discovery processing procedure. Here, the failure cause determination rule generation program 311 arranges the frequent event labels assigned to the frequent events constituting each failure displayed in the event block table 730 in the order of the attribute “time” of the event 732. Discover frequent event patterns by creating columns. That is, the frequent event pattern is specified as a pattern in which frequent event labels are arranged in order of appearance time. In the example of FIG. 9, the frequent event sequence is “CAB”, “AB”, “ACB”. Next, the failure cause determination rule generation program 311 obtains an appearance pattern in which event labels satisfying the number of event blocks specified by the minimum supported event block number input unit 722 appear frequently. In the example of FIG. 9, “AB” and “CB” are obtained. The event label pattern replaced with the frequent event 742 in the frequent event table 740 becomes the frequent event pattern.

(Step 608)
The failure cause determination rule generation program 311 displays the execution result of the frequent event pattern discovery process on the failure cause determination rule generation program screen 700 through the display device 331. FIG. 10 shows a specific example in which the execution result of the frequent event pattern discovery process is displayed in the event pattern candidate table 1040 constituting the failure cause determination rule generation program screen 700.

(Step 609)
The failure cause determination rule generation computer 107 selects event pattern candidates in the event pattern table 1040 constituting the failure cause determination rule generation program screen 700 of FIG. 10 displayed on the display device 331 through the operation of the input device 330 by the user. When the detected operation is detected, the contents of the selected event pattern candidate 1046 are read and stored in the work area 312 of the RAM 310. Further, the failure cause determination rule generation program 311 follows the processing procedure of the recovery procedure document search process described in the failure cause determination rule generation program 311 and based on the contents of the event pattern candidates stored in the work area 312. The retrieval process of the document is executed, and the retrieved recovery procedure document is stored in the work area 312. Details of this processing will be described later.

(Step 610)
The failure cause determination rule generation program 311 displays the execution result of the recovery procedure manual search stored in the work area 312 on the failure cause determination rule generation program screen 700 through the display device 331. FIG. 10 shows a specific example in which the search result of the recovery procedure manual is displayed on the recovery procedure manual table 1060 constituting the failure cause determination rule generation program screen 700.

(Step 611)
The failure cause determination rule creation program 311 of the failure cause determination rule generation program 311 performs the following on the failure cause determination rule generation program screen 700 (FIG. 10) displayed on the display device 331 through a user operation input to the input device 330. It is executed through operation input.

  First, after an event pattern candidate is selected from the event pattern candidate table 1040 constituting the event pattern candidate unit 1030, the determination button 1031 is clicked. By this operation, the contents of the selected event pattern candidate 1046 are displayed in the event pattern 1021 and the determination time 1022 of the failure cause determination table 1020.

  Next, after a recovery procedure manual to be associated is selected from the recovery procedure manual table 1060 constituting the recovery procedure manual candidate unit 1050, the determination button 1051 is clicked. By this operation, the content of the selected recovery procedure manual 1063 is displayed in the recovery procedure manual 1023 of the failure cause determination table 1020.

  Finally, when the determination button 1011 in the final failure cause determination rule unit 1010 is clicked, the failure cause determination rule generation program 311 starts registration processing of the created failure cause determination rule. Note that selection and click operations on the screen are input through a user operation on the input device 330 constituting the failure cause determination rule generation computer 107.

  In this embodiment, the user selects a conditional phrase and selects an action phrase to be associated with the selected conditional phrase, but is displayed in the recovery procedure document table 1060 of the recovery procedure document candidate unit 1050. By automatically associating the individual recovery procedure place with the highest score 1062 with each conditional phrase, the failure cause determination rule creation process can be automated. The score 1062 represents the statistical degree of coincidence between the search keyword 1052 and the document in the recovery procedure manual 1061, and the higher the score, the more search keywords appear in the document of the recovery procedure manual 1061.

(Step 612)
When the failure cause determination rule generation program 311 detects the above-described series of operation inputs, the failure cause determination rule generation program 311 reads the content displayed in the failure cause determination rule table in the failure cause determination rule generation program screen 700 of FIG. To store. Further, the failure cause determination rule generation program 311 transmits the contents of the failure cause determination rule table stored in the work area 312 to the failure cause analysis computer 104 through the communication device 332 according to the processing procedure of the failure cause determination rule registration process. Then, a failure cause determination rule (a rule in which a condition phrase and an action phrase are associated) is registered in the failure cause analysis computer 104.

(b) Detailed Operation (1) Fault Clustering Here, the detailed operation of the fault clustering process executed in step 601 will be described. FIG. 11 shows an outline of a processing procedure executed in the failure clustering process.

(Step 1101)
After the failure clustering process is started, the failure cause determination rule generation program 311 accesses the log DB 103 via the communication device 332, and an event related to the failure (the attribute “type” of the event 203 is “fatal” or “emergency”). And events that take a value such as “error” and do not include an event that takes a value not related to a failure such as “information”). Thereafter, the failure cause determination rule generation program 311 temporarily stores the acquired event in the data storage unit 322 of the hard disk drive 320 and then stores it in the work area 312 of the RAM 310. Here, the event includes an event ID 201, an event occurrence date / time 202, and an event attribute value 203 displayed in FIG.

(Step 1102)
The failure cause determination rule generation program 311 indicates that the difference between the recently acquired event occurrence date and time 202 stored in the work area 312 and the previous event occurrence date and time 202 is in the failure cause determination rule generation program screen 700. It is determined whether or not it is within the time specified by the time window unit input unit 711. If it is within the time, the failure cause determination rule generation program 311 executes Step 1104. If not within the time, the failure cause determination rule generation program 311 executes Step 1103.

(Step 1103)
In step 1103, the failure cause determination rule generation program 311 creates a new event block, and classifies recently acquired events into newly created event blocks. For example, in the case of the event log of FIG. 2, the event occurrence date 202 “2009/05/23 12:23:35” with the event ID 201 “5” and the event occurrence date 202 “2” immediately before the event ID 201 “ The time difference from “2009/05/23 10:30:34” is equal to or greater than the time “10 minutes” specified by the time window unit input unit 711. Therefore, the failure cause determination rule generation program 311 regards events with event IDs 201 “5” and “4” as events caused by different failures, and classifies events with event ID 201 “5” as new event blocks. .

(Step 1104)
In step 1104, the failure cause determination rule generation program 311 adds the recently acquired event to the same event block as the previous event. For example, in the event log of FIG. 2, the event occurrence date 202 “2009/05/23 10:30:34” with the event ID 201 “4” and the event occurrence date 202 “3” immediately before the event ID 201 “ The time difference from “2009/05/23 10:27:26” is within the time “10 minutes” specified by the time window unit input unit 711. Therefore, the failure cause determination rule generation program 311 regards events having event IDs “4” and “3” as events that have occurred continuously due to one failure, and classifies them into the same event block.

(Step 1105)
The failure cause determination rule generation program 311 confirms whether an unacquired event remains in the event log DB 103. If it remains, the failure cause determination rule generation program 311 executes Step 1101, and if not, executes Step 1106.

  FIG. 12 is an event block table 1200 after the events of FIG. 2 (event ID 201 is 1 to 17) are classified for each event block according to the procedure from step 1101 to step 1105. The event block table 1200 includes an event block ID 1201 that uniquely identifies an event block, an event ID 201, an event occurrence date 202, and an event attribute 203.

(Step 1106)
The failure cause determination rule generation program 311 extracts features for each event block in the created event block table 1200. FIG. 13 is an example of an event block feature table 1300 created from the event block table 1200 shown in FIG. The event block feature table 1300 includes an event block ID 1201, an attribute list 1301 that is a feature of the attribute value of the event in the event block, the number of events 1302 included in the event block, the last event in the event block, and the occurrence of the first event It consists of a time width 1303 that is the difference between the date and time 202. The attribute list 1301 includes attribute values that appear most frequently and attribute values that appear second most frequently for each attribute 203 of the event included in the event block.

(Step 1107)
The failure cause determination rule generation program 311 clusters event blocks based on the event block feature table 1300 obtained in step 1106. As the clustering method, for example, conceptual clustering COBWEB described in Non-Patent Document 1 may be used. When clustering is executed, a concept tree of failures is generated.

(2) Recovery Procedure Manual Search Here, the detailed operation of the recovery procedure manual search process executed in step 609 will be described. FIG. 14 shows an outline of the processing procedure executed in the recovery procedure manual search processing.

(Step 1401)
After starting the recovery procedure manual search process, the failure cause determination rule generation program 311 extracts the feature of the attribute value from the event pattern candidate 1046 stored in the work area 312 of the RAM 310. The method for extracting features is determined from the attribute value of the event with the higher importance of the attribute “type”, the attribute value of the first event of the event pattern candidate, the attribute value that appears frequently, etc. . In the event pattern candidate 1046, the attribute value of the attribute “type” is fixed for the attributes “source” and “event number” that are valid for the search, and the attribute value of the event of the frequent event label A that appears first. Therefore, (process71, 80) is a feature.

(Step 1402)
The failure cause determination rule generation program 311 generates a search keyword “source process71 event number 80” that combines the attribute value feature extracted in step 1401 and its attribute name. At this time, the attribute name to be combined can be “process name prcess71 error code 80” by using a name often used in the recovery procedure manual. In addition, “process name process71 error 80” can be set in consideration of the language of the recovery procedure manual.

(Step 1403)
The failure cause determination rule generation program 311 accesses the recovery procedure manual DB 105 via the communication device 332 and searches the recovery procedure manual DB 105 using the search keyword generated in step 1402. Further, the failure cause determination rule generation program 311 acquires the recovery procedure manual hit by the search process executed via the communication device 332 from the recovery procedure DB 105 and temporarily stores it in the data storage unit 332 of the hard disk drive 320. After saving, it is stored in the work area 312 of the RAM 310.

(effect)
As described above, in the failure cause analysis system that implements the failure cause determination rule generation device (program) according to the present embodiment, the following processing operations are executed.
(1) The event generated by the monitoring server 102 at the time of failure is acquired from the log DB 103.
(2) The acquired event is divided into event blocks for each fault, and faults are clustered based on the characteristics of the event block to generate a fault cluster. The event block is composed of one or a plurality of events that are highly related to a failure. A failure cluster is a set of event blocks that are likely to have occurred from the same failure.
(3) Find frequent events that frequently appear in event blocks in the failure cluster.
(4) A frequent event pattern is found from a frequent event sequence in which the frequent events found are arranged in time series.
(5) A search keyword is created based on the attribute value of the found frequent event pattern, and the corresponding recovery procedure manual is searched from the recovery procedure database.
(6) A failure cause determination rule is generated by combining the obtained frequent event pattern and the recovery procedure manual.

  That is, by finding a failure determination rule (frequent event) based on a failure block that is likely to have the same failure cause, a failure cause determination rule that is highly likely to be related to a failure can be created as a conditional phrase. Further, a failure cause determination rule suitable for failure recovery can be automatically generated by combining a statistically high action phrase with the created conditional phrase.

Claims (8)

A first processing unit for automatically acquiring an event generated by the monitoring server at the time of failure from the event log database;
A second processing unit that automatically divides an automatically acquired event into event blocks for each failure, automatically clusters failures based on the characteristics of the event block, and generates a failure cluster;
A third processing unit that automatically finds frequent events that frequently appear in event blocks in the failure cluster;
A fourth processing unit for automatically finding a frequent event pattern from a frequent event sequence in which the frequent events found are arranged in time series;
A fifth processing unit that automatically creates a search keyword based on the attribute value of the frequently-occurring event pattern found, and automatically searches for a corresponding recovery procedure manual from a database of the recovery procedure manual;
A rule generation device for a failure cause analysis system, comprising: a sixth processing unit that automatically generates a failure cause determination rule by combining the obtained frequent event pattern and a recovery procedure manual. The second processing unit categorizes an event whose difference from an occurrence time of the previous event is within a time window unit into an event block to which the previous event belongs. The rule generation device for the failure cause analysis system described. The second processing unit includes an attribute value that frequently appears in an event included in the event block, a time width of the event block (difference between occurrence times of the first event and the last event in the event block), and the number of events in the event block The rule generation device for a failure cause system according to claim 1 or 2, wherein any one or a plurality of the event block is characterized by an event block. The said 3rd process part discovers the frequent event which satisfy | fills the minimum number of effective attributes which do not contain an indefinite attribute value, and the minimum number of support event blocks. Rule generation device for failure cause analysis system described in 1. The third processing unit represents a plurality of events in an event block corresponding to the same frequent event found with the same color or the same symbol, and visualizes the distribution of the frequent events in the failure cluster. The rule generation device for a failure cause analysis system according to any one of claims 1 to 4. 6. The rule generation device for a failure cause analysis system according to claim 1, wherein the fourth processing unit finds a frequent event pattern that satisfies a minimum number of support event blocks. 7. The fifth processing unit includes one or more attribute values of an attribute value of an event having a high importance included in a frequent event pattern, an attribute value that frequently appears, and an attribute value of the first event, and a corresponding attribute The rule generation device for a failure cause analysis system according to any one of claims 1 to 6, wherein a search keyword is generated in combination with a name. On the computer,
A first process for acquiring an event generated by the monitoring server at the time of failure from the event log database;
A second process of dividing the acquired event into event blocks for each failure, clustering failures based on the characteristics of the event block, and generating a failure cluster;
A third process for finding a frequent event that frequently appears in the event block in the failure cluster;
A fourth process for finding a frequent event pattern from a frequent event sequence in which the frequent events found are arranged in time series;
A fifth process of creating a search keyword based on the attribute value of the found frequent event pattern and searching the corresponding recovery procedure manual from the recovery procedure database;
A program for executing a sixth process of generating a failure cause determination rule by combining the obtained frequent event pattern and the recovery procedure manual.

Images