JP5215895B2 - Failure cause analysis system and program - Google Patents

Description

  The present invention relates to a system and a program for automatically analyzing the cause of a failure when a system failure occurs on an IT (Information Technology) system. More specifically, the root of a system failure is determined through a matching process between an event sequence sent from the IT system monitoring system and a predefined determination rule (combination pattern of one or more failure cause determination conditions). The present invention relates to a system and program for identifying the cause.

  In business fields such as manufacturing, communications, finance, etc., where system failures have a major impact on business and society, there is a strong demand for shortening the time to analyze the cause of failure and shortening the recovery time when a system failure occurs. For this reason, the rapid identification of the root cause of system failure is particularly important.

  By the way, the present analysis work is advanced by manual work based on materials such as a failure handling procedure manual. For this reason, there is a limit to shortening the analysis work time. In addition, when a document such as a failure handling procedure document created in the past becomes obsolete due to modification of the IT system and does not match the actual situation, it is difficult to analyze the root cause of the system failure itself.

As described above, as long as the analysis work of the root cause of the system failure is performed manually, the ability of the analysis work becomes a person and delays the recovery of the system failure.
In order to solve this problem, the failure cause analysis system systematizes the analysis of the root cause of the system failure.

  FIG. 1 shows a conventional example of a failure cause analysis system. The failure cause analysis system 101 shown in FIG. 1 includes a connection adapter 104, a complete match determination rule 105, a failure cause complete match determination mechanism 106, a failure cause determination result dashboard generation mechanism 107, and a display monitor 108. ing.

  The connection adapter 104 is an interface unit that receives an event from the monitoring system 103 and interprets the content of the event. The exact match determination rule 105 is a set of determination rules prepared for each root cause of a system failure. In each determination rule, failure cause determination conditions specific to the corresponding root cause are described in the order of occurrence. The failure cause determination condition here corresponds to each event. The contents of each determination rule are generated based on the failure cause match determination rule definition template 109.

  The failure cause complete match determination mechanism 106 compares the notified event with each determination rule, and determines whether there is a failure cause that satisfies all the failure cause determination conditions constituting the determination rule. The failure cause determination result dashboard generation mechanism 107 generates a dashboard screen that displays the cause of the failure specified based on the determination result. The generated dashboard screen is presented to the user (for example, a maintenance manager) through the display screen of the display monitor 108.

  FIG. 1 also shows a monitoring target system 102 to be monitored by the failure cause analysis system 101 and a monitoring system 103 for monitoring the output log. In general, the data structure of the output log differs for each monitoring target system 102. For this reason, the monitoring system 103 executes a process of converting the data structure of the output log into a data structure of a unified format. In this specification, an output log that has been converted to a unified format is referred to as an “event”.

  Next, an outline of an analysis operation executed by the failure cause analysis system 101 will be described. In this conventional system, the monitoring system 103 constantly monitors the operating state of the monitoring target system 102. When the operating state of the monitoring target system 102 satisfies a preset condition, the monitoring system 103 sends a corresponding event to the failure monitoring analysis system 101. The failure cause analysis system 101 compares the determination rule stored in the exact match determination rule 105 with the sequence of events received from the monitoring system 103, and whether or not all failure cause determination conditions are satisfied for each determination rule. Determine whether. When any one of the failure cause determination conditions constituting the determination rule is satisfied, the failure cause analysis system 101 determines that the cause of the failure has been specified, and displays the specified failure cause on the dashboard of the display monitor 108. Display on the screen.

  FIG. 2 shows a description example of the determination rule 201 for each failure cause created based on the failure cause match determination rule definition template 109. This determination rule 201 is prepared for each cause of failure. The determination rule 201 is described in a predetermined rule language. As shown in FIG. 2, the determination rule 201 describes one or a plurality of failure cause determination conditions and actions to be executed when all of them are determined to be true (established). In FIG. 2, this action is referred to as a failure cause determination result action.

JP 2000-183881 A

  As described above, in the conventional system shown in FIG. 1, the cause of the system failure is specifically displayed only when there is a judgment rule that completely matches the event sequence received from the monitoring system 103 when the system failure occurs. can do. This means that the cause of the system failure cannot be identified when there is even one event for which the failure cause determination condition is not determined to be true (established).

  Therefore, in the case of the conventional system, it is necessary to prepare the failure cause coincidence determination rule definition template 109 that enumerates all event string patterns that completely match the cause of failure that may occur.

  However, in the case of a large-scale system, this pattern exists in an almost infinite number. That is, it is virtually impossible to enumerate all the patterns in advance. In addition, when the system configuration is altered, it is necessary to rewrite the failure cause match determination rule definition template 109 in response to the system failure. Such rewriting is virtually impossible.

  Therefore, even if the determination condition is satisfied only for a part of the failure cause determination conditions constituting the determination rule prepared in advance, the applicant can determine the system failure from among the failure causes in which the failure cause determination condition is partially satisfied. A fault cause analysis system and program for identifying fault cause candidates that are likely to be causes and displaying a list on a screen are proposed.

  In addition, a failure cause analysis system and program capable of displaying the number of corresponding failure cause determination conditions and the establishment status of each failure cause determination condition as a set for each failure cause candidate at the time of display are proposed.

  According to the applicant's proposed invention, when a system failure occurs, all of the failure cause identification conditions constituting the determination rule and the received event sequence do not completely match because the available determination rule is incomplete. Even in this case, failure cause candidates that are likely to be the root cause of the system failure can be displayed in a list on the screen. That is, a highly objective failure cause candidate can be displayed on the screen. As a result, the user can execute specific work in order from the most likely failure cause candidates. That is, it is possible to avoid the failure analysis and to shorten the analysis time.

  Further, when displaying the cause of failure candidates, the number of failure cause determination conditions constituting each failure cause candidate and the establishment status of each failure cause determination condition are explicitly displayed. With this function, the user can proceed with the analysis work in consideration of not only the mere establishment ratio of the failure cause determination condition but also the failure cause determination condition that is satisfied or not satisfied.

It is a figure which shows the prior art example of a failure cause analysis system. It is a figure which shows the example of a description of the determination pattern used by the failure cause complete match determination rule. It is a figure which shows the structural example of the failure cause analysis system which concerns on embodiment. It is a figure which shows the example of a data structure of a failure cause determination rule definition template. It is a figure which shows the example of a list display screen of the failure cause analysis result which concerns on embodiment. It is a figure which shows the example of a determination pattern used with the failure cause partial matching determination rule which concerns on embodiment. It is a figure which shows the data structure example of the decision table which concerns on embodiment. It is a figure which shows an example of the flowchart used for preparation of the determination rule of a failure cause. It is a figure which shows an example of the flowchart used for preparation of a decision table.

  Hereinafter, an example of a failure cause analysis system according to the present invention will be described. Needless to say, the present invention is not limited to the embodiments and description to be described later, and can be applied to other modes combined with known techniques.

[First example]
(A) Outline of Processing Function FIG. 3 shows a schematic configuration example of the failure cause analysis system 301. As will be described later, the failure cause analysis system 301 simultaneously generates not only a complete match determination rule but also a partial match determination rule based on the failure cause determination rule definition template 314. By applying these two types of determination rules, the failure cause analysis system 301 selects failure cause candidates that are highly likely to be the cause of the system failure even when failure causes that satisfy all failure cause determination conditions are not identified. Display a list on the screen. At this time, if the one with a large proportion of the determination condition is established is preferentially displayed on the screen, the user can analyze the cause of the failure from candidates that are highly likely to cause the system failure.

  When displaying a list of failure cause candidates, it is desirable to prioritize not only the rate at which the failure cause determination condition is satisfied (that is, the match rate) but also the number of failure cause determination conditions that constitute the failure cause. This is because even if the coincidence rate is the same, the larger the number of failure cause determination conditions, the higher the difficulty of establishment, and the higher the possibility of a failure cause.

  In addition, when displaying a list of failure cause candidates, it is desirable to display the number of failure cause determination conditions corresponding to each failure cause candidate and their establishment status as a set. This is because the ease of establishment differs depending on the failure cause determination condition. By displaying these pieces of information on the screen, it is possible to improve the work efficiency of failure analysis as compared with the case where the priority of work is determined only by the match rate.

  In this specification, “establishment of failure cause determination condition” means that an event corresponding to the failure cause determination condition exists. Further, in this specification, “establishment of a failure cause determination condition” is also referred to as a failure cause determination condition and an event matching. Therefore, the fact that all the failure cause determination conditions corresponding to the cause of the failure are satisfied is also said that the failure cause determination condition and the event sequence completely coincide. Further, the fact that only some failure cause determination conditions of the failure cause determination condition corresponding to the cause of the failure are satisfied also means that the failure cause determination condition and the event partially match.

(B) Schematic Configuration of System The failure cause analysis system 301 includes a connection adapter 304, a complete match determination rule 305, a failure cause complete match determination mechanism 306, a partial match determination rule 307, and a failure cause partial match determination mechanism 308. , A failure cause determination rule generation mechanism 309, a failure cause determination result dashboard generation mechanism 310, a decision table 311, a log database 312, and a display monitor 313.

  The connection adapter 304 is an interface that executes processing for receiving an event from the monitoring system 303 and interpreting the content of the event. Here, the monitoring system 303 monitors the operation state of the monitoring target system 302 through its output log. As described above, the monitoring target system 302 is assumed to be applied to a relatively large-scale system having a large influence of failure occurrence, such as a salary system, a purchasing system, a settlement system, and the like. The monitoring target system 302 is also a system in which failure cause analysis and analysis result accumulation are continuously executed.

  The complete match determination rule 305 is a set of determination rules for complete match determination prepared for each root cause of a system failure. The complete match determination rule 305 is realized as a collection of determination rules described in FIG. The complete match determination rule 305 is realized as a part or all of the area secured in a storage area (not shown).

  The failure cause complete match determination mechanism 306 includes a comparison process between the notified event sequence and each determination rule for complete match determination, and a determination process whether or not all failure cause determination conditions constituting the determination rule are satisfied. When a determination rule that satisfies all failure cause determination conditions is detected, the signal processing unit executes an action described in the determination rule. The comparison process between the determination rule and the event string is executed within the range of the number of failure cause determination conditions defined by each determination rule.

  This failure cause complete coincidence determination mechanism 306 corresponds to a “first determination unit” in the claims. Further, the failure cause complete match determination mechanism 306 can be realized as hardware or software. When realized as software, it is realized as a partial function of a program executed by a computer.

  The partial match determination rule 307 is a set of determination rules for partial match determination prepared for each root cause of a system failure. A determination rule for partial match determination is a set of determination rules that describe a set of actions to be executed when each failure cause determination condition is satisfied for each individual failure cause determination condition that constitutes a complete match determination determination rule. Consists of.

  For example, when there are three failure cause determination conditions that constitute a determination rule for complete match determination, the determination rule for partial match determination includes three determination rules that are a set of individual determination operations and actions for determination results. Described. In the case of this embodiment, the action for the determination result is defined only when the condition is satisfied in the determination operation. For example, it is described that the failure cause determination condition confirmed to be established is recorded in the log database 312 as the content of the action. The partial match determination rule 307 is also realized as a part or all of the area secured in a storage area (not shown).

  The failure cause partial match determination mechanism 308 performs a comparison process with an event for each failure cause determination condition constituting a determination rule for complete match determination based on the notified event sequence and each determination rule for partial match determination And a signal processing unit that executes an action described for each comparison result. As described above, the failure cause partial match determination mechanism 308 executes the recording process of the failure cause determination condition in the log database 312 in the determination unit of each failure cause determination condition.

  Also in the case of the failure cause partial match determination mechanism 308, the comparison process between the determination rule and the event sequence is executed within the range of the number of failure cause determination conditions defined by each determination rule. For example, if the number of failure cause determination conditions corresponding to the determination rule is 6, comparison processing is executed for 6 events. For example, if the number of failure cause determination conditions corresponding to the determination rule is three, the comparison process is executed for three events.

  The failure cause partial match determination mechanism 308 corresponds to a “second determination unit” in the claims. Further, the failure cause partial match determination mechanism 308 can be realized as hardware or software. When realized as software, it is realized as a partial function of a program executed by a computer.

  The failure cause determination rule generation mechanism 309 is a signal processing unit that creates a determination rule for complete match determination and a determination rule for partial match determination when a failure cause match determination rule definition template 314 is input. An input interface (not shown) is used to input the failure cause match determination rule definition template 314.

  The failure cause determination rule generation mechanism 309 corresponds to a “determination rule generation unit” in the claims. Further, the failure cause determination rule generation mechanism 309 can be realized as hardware or software. When realized as software, it is realized as a partial function of a program executed by a computer.

  The failure cause determination result dashboard generation mechanism 310 is a signal processing unit that generates a dashboard screen for presenting the analysis processing result to the user based on the determination processing result using the complete match determination rule or the partial match determination rule. . The failure cause determination result dashboard generation mechanism 310 corresponds to a “display screen generation unit” in the claims.

  The failure cause determination result dashboard generation mechanism 310 can be realized as hardware or software. When realized as software, it is realized as a partial function of a program executed by a computer.

  Note that the failure cause determination result dashboard generation mechanism 310 generates a dashboard screen in accordance with prior settings or user instructions. For example, when the dashboard screen is generated, the matching rate between the failure cause determination condition and the event and the number of failure cause determination conditions constituting the determination rule are referred to. For example, the match rate is used as the first sort condition, and the number of failure cause determination conditions is used as the second sort condition between failure cause candidates having the same match rate. Alternatively, the number of failure cause determination conditions is used as the first sort condition, and the match rate is used as the second sort condition between failure cause candidates having the same number of failure cause determination conditions. In any case, it is advantageous for shortening the work time to devise a failure cause that is more likely to be the root cause of the system failure so that it is displayed higher. However, there is a possibility that the cause of the failure cannot be reached in a short time only with these sort displays. This is because the weight information between the failure cause determination conditions is not reflected in these sorts. Therefore, the failure cause determination result dashboard generation mechanism 310 according to the embodiment employs a mechanism that helps the user to evaluate each failure cause candidate. Details will be described later. The generated dashboard screen is displayed on the display monitor 313.

  In addition, the failure cause determination result dashboard generation mechanism 310 also has a function of managing the progress of user actions or system actions with respect to failure cause candidates displayed on the dashboard screen.

  Further, when the failure cause determination result dashboard generation mechanism 310 is definitely identified as the current cause of failure through the subsequent confirmation work on the failure cause candidate, as shown by the arrow in FIG. It also has a function to automatically create and output rule definition templates. The content of the failure cause determination rule definition template output here is realized, for example, by selectively extracting a failure cause determination condition that matches the event sequence. The output of the failure cause determination rule definition template includes output to an external storage medium, output through a communication line, and output to the failure cause determination rule generation mechanism 309. In any case, the installation of this output function makes it possible to increase the possibility of a perfect match when the same cause of failure occurs.

  The decision table 311 is used as an area for storing individual determination results for the failure cause determination condition when the determination rule for partial match determination is applied, and collecting and storing for each determination rule. This decision table 311 is also realized as a part or all of the area secured in a storage area (not shown). The contents stored in the decision table 311 are referred to by the failure cause determination result dashboard generation mechanism 310 when the dashboard screen is generated.

  The log database 312 is used as a recording area for determination results of failure cause determination condition units executed using the determination rule for partial match determination. In the case of this embodiment, the log database 312 is used only for recording the failure cause determination condition for which the determination condition is satisfied.

(C) Schematic Operation An outline of an analysis operation executed in the failure cause analysis system 301 according to the embodiment will be described.
In the failure cause analysis system 301, the monitoring system 303 constantly monitors the operating state of the monitoring target system 302. When the operation state of the monitoring target system 302 satisfies a preset condition, the monitoring system 303 sends a corresponding event to the failure monitoring analysis system 301. The failure cause analysis system 301 compares the determination rule stored in the exact match determination rule 305 with the sequence of event sequences received from the monitoring system 303, and whether or not all the failure cause determination conditions are satisfied for each determination rule. Determine whether. When all of the failure cause determination conditions constituting the determination rule are satisfied, the failure cause analysis system 301 determines that the failure cause has been specified, visualizes the determination result on the dashboard, and displays the result on the display monitor 313. .

  However, when the failure cause cannot be specified by the determination rule based on the complete match determination rule 305, the failure cause analysis system 301 determines the determination rule stored in the partial match determination rule 307 and the event sequence received from the monitoring system 303. Compare the sequence with. That is, it is determined whether or not the determination rule for complete match determination and the pattern of the sequence of event sequences partially match. When a partial match is confirmed, the failure cause analysis system 301 generates a dashboard screen based on the confirmed failure cause candidates and displays the dashboard screen on the display monitor 313.

(D) Data Structure Example of Failure Cause Determination Rule Definition Template FIG. 4 shows a detailed structure example of the above-described failure cause determination rule definition template. As shown in FIG. 4, the failure cause determination rule definition template 401 includes a failure cause determination rule name 402, a failure cause determination condition list 403, and a failure cause determination result action 404.

  The failure cause determination condition list 403 further includes an array of failure cause determination conditions 405. The contents and number of failure cause determination conditions 405 arranged in each failure cause determination condition 403 are different for each failure cause to be monitored. That is, the number of arrays of failure cause determination conditions 405 is one or more, and the number of determination conditions necessary for specifying the cause of failure is arranged. The failure cause determination result action 404 describes an action to be executed when all the failure cause determination conditions described in the failure cause determination condition list 403 are satisfied. Of course, the contents of the description differ depending on the cause of the failure. The content of this action is taken over by the failure cause determination result action of the failure cause exact match determination rule shown in FIG.

(E) Screen Example Displayed when Partial Match Determination Operation is Performed FIG. 5 shows a display screen example of the failure cause analysis result list display screen 501. The failure cause analysis result list display screen 501 is generated by the failure cause determination result dashboard generation mechanism 310. The failure cause analysis result list display screen 501 is configured in a two-dimensional table format as shown in FIG.

  In each row, information about individual failure cause candidates is displayed. Note that the display order follows a predetermined sorting rule selected or set. For example, a rule that sorts failure cause candidates that have a high possibility of a failure cause to the top is used.

At this time, it is desirable that the sorting rule consider not only the high matching rate but also the number of failure cause determination conditions. For example, a rule that sorts the high match rate as the first sort condition and sorts the corresponding failure cause determination condition number as the second sort condition for the failure cause candidates having the same match rate is used. This is because, if the coincidence rate is the same, the higher the number of failure cause determination conditions, the higher the possibility of the failure cause. On the contrary, the rule that sorts the number of failure cause determination conditions as the first sort condition and sorts the match rate as the second sort condition for the failure cause candidates having the same number of failure cause determination conditions may be used. good.
The display example of FIG. 5 is a case where the number of failure cause determination conditions is 6 for any failure cause candidate. For this reason, the display order basically matches the match rate.

  On the other hand, in each column, items necessary for identifying the cause of the failure and confirming the determination result are displayed in order. In the case of FIG. 5, the ranking 502, the matching rate 503, the failure cause determination rule name 504, and the failure cause determination condition establishment status 505 are configured.

  The rank 502 is a display item for representing the rank in which the lines are sorted and arranged. The coincidence rate 503 represents a rate at which the failure cause determination conditions partially match in the failure cause determination condition list 403 defined in the failure cause determination rule definition template 401. For example, when there are six failure cause determination conditions and there are five failure cause determination conditions that satisfy the condition, a rounded numerical value (80%) is displayed.

  The failure cause determination rule name 504 is the name of the determination rule described in the failure cause determination rule definition template 401 used in the match determination.

  The failure cause determination condition establishment status 505 is a display item that individually indicates the match status of each failure cause determination condition 405 described in the failure cause determination condition list 403 in the failure cause determination rule definition template 401 that is the basis of the match determination. It is. In the case of this form example, two types of display forms are prepared for displaying the failure cause determination condition establishment status 505. One is a normal display screen and the other is an enlarged display screen.

  The normal display screen is a display for confirming the overall establishment status of the failure cause determination condition. In the case of the normal display screen, blocks corresponding to the number of failure cause determination conditions necessary for specifying the failure cause corresponding to each row are displayed in the display column of the failure cause determination condition establishment status 505 corresponding to each row.

  In the display example of FIG. 5, six blocks are displayed for each row. That is, it is indicated that all the failure cause determination conditions necessary for specifying each failure cause candidate are six. In the drawing, blocks indicated by shading represent blocks corresponding to the failure cause determination condition for which the determination condition is satisfied. In the actual display screen, this block is displayed as a highlight block (a block having relatively high luminance) or a block colored in a specific color (for example, yellow). On the other hand, a white block represents a block corresponding to a failure cause determination condition for which the determination condition is not satisfied. Further, the arrangement of the display blocks here matches the order of the failure cause determination conditions described in the failure cause determination condition list 403 (FIG. 4).

  As described above, the normal display screen is very convenient for roughly understanding the establishment status (matching and mismatching) of determination conditions for a plurality of failure causes. On the other hand, the display column of the failure cause determination condition establishment status 505 is limited in terms of assignable display width and display area, and it is possible to confirm the content of the failure cause determination condition corresponding to each block. Can not. Actually, the contents of the corresponding failure cause determination conditions are not the same even in blocks adjacent vertically in the display screen. Therefore, as described above, in the case of this embodiment, an enlarged display screen as described later is prepared.

  The enlarged display screen is a display screen that enlarges and displays at least the contents of the failure cause determination condition establishment status 505 among the specific items on the screen designated by a pointer or the like. In the case of FIG. 5, the enlarged display screen is displayed so as not to overlap the above-described normal display screen. Of course, the enlarged display screen can be displayed on the normal display screen. This enlarged display screen is different from the normal display screen in that characters and names representing the contents of the corresponding failure determination condition are individually displayed in individual blocks corresponding to the failure cause determination condition. Of course, highlight display and color display on the normal display screen are also carried over to the enlarged display screen. By adding the character information, the user can easily confirm which failure determination condition is satisfied and which failure determination condition is not satisfied.

(F) Example of Data Structure of Failure Cause Partial Match Determination Rule FIG. 6 shows a description example of the failure cause partial match determination rule 601 created based on the failure cause match determination rule definition template 314. The failure cause partial match determination rule 601 is a set of determination rules 602 corresponding to individual failure cause determination conditions 405 in the failure cause determination condition list 403 described in the failure cause determination rule definition template 314. The determination rule 602 describes a failure cause determination condition and an action to be executed when the failure cause determination condition is satisfied. In the case of FIG. 6, it is described that the action is to be recorded in the log database 312 such as wording information indicating establishment of the corresponding failure cause determination condition. Due to the presence of this recording command, the failure cause determination conditions that satisfy the determination conditions are accumulated in the log database 312. Further, a decision table 311 is generated based on this information.

(G) Data Structure Example of Decision Table FIG. 7 shows a data structure example of the decision table 701 (311 in FIG. 3). The decision table 701 is created by the failure cause partial match determination mechanism 308. The failure cause partial match determination mechanism 308 refers to the log database 312 in which the history of failure cause partial match determination operations is accumulated, and generates a table as shown in the figure. As described above, the decision table 311 is an important table that is used to generate a dashboard for grasping the partial establishment status of the cause of failure.

  The decision table 701 is composed of an array of element number information 704 for storing the number of elements to be stored and a failure cause determination rule corresponding element 702. In the case of FIG. 7, since the number given by the element number information 704 is N, N rows of failure cause determination rule corresponding elements 702 are arranged. The failure cause determination rule correspondence element 702 here corresponds to one of the failure cause perfect match determination rules registered in the failure cause analysis system 301 through the failure cause determination rule definition template 314.

  The failure cause determination rule correspondence element 702 includes a pointer 705 to the complete match determination rule 703 (305 in FIG. 3), a failure cause determination condition number 706, a failure cause determination condition establishment number 707, and a failure cause determination condition of the partial match determination rule. Of success / failure information 708.

  The failure cause determination condition number 706 is set to a unique value M for each failure cause determination rule corresponding element 702. Of course, the numerical value is uniquely set by counting the number of failure cause determination conditions constituting the corresponding perfect match determination rule.

  On the other hand, the failure cause determination condition satisfaction number 707 is the number K of conditions that the condition is satisfied among the M failure cause determination conditions. In the case of this embodiment, this numerical value is set through a matching operation with the log database 312.

  In the case of this embodiment, success / failure information 708 is additionally recorded every time a determination condition that satisfies the condition is detected through an analysis operation of the log database 312 described later. Therefore, the number K of success / failure information 708 and the number K of failure cause determination condition establishment number 707 coincide with each other.

  Note that the failure rate of the failure cause partial match determination rule displayed on the dashboard is calculated as a percentage obtained by dividing the failure cause determination condition establishment number 707 in the decision table 701 by the failure cause determination condition number 706.

(H) Failure Cause Determination Rule Generation Procedure FIG. 8 shows a processing procedure executed in the failure cause determination rule generation mechanism 309. That is, a procedure for generating the complete match determination rule 305 and the partial match determination rule 307 from the failure cause determination rule definition template 314 is shown.

  When the failure cause determination rule definition template 314 is input, the failure cause determination rule generation mechanism 309 converts the content into the failure cause complete match determination rule 201 and outputs it (step 801).

  Next, the failure cause determination rule generation mechanism 309 extracts one failure cause determination condition 405 from the failure cause determination condition list 403 of the failure cause determination rule definition template 314 (step 802).

  Next, the failure cause determination rule generation mechanism 309 creates a rule for the failure cause determination condition 405 that is the processing target, with the action content being to record the result in the log database 312 when the condition is satisfied. It adds to the coincidence determination rule 307 (step 803).

  Thereafter, the failure cause determination rule generation mechanism 309 determines whether or not an unprocessed failure cause determination condition 405 remains (step 804). If it is confirmed that the failure remains, the failure cause determination rule generation mechanism 309 returns to step 802 described above, and continues a series of operations for the next unprocessed failure cause determination condition 405 (step 804). If the determination operation has been completed for all the failure cause determination conditions 405, the failure cause determination rule creation operation ends.

(I) Decision Table Creation Procedure FIG. 9 shows an example of a processing procedure used when the above-described decision table 311 is created. The operation subject is the failure cause partial match determination mechanism 308. Further, this processing operation is executed after the time when the failure cause partial match determination mechanism 308 detects that the determination operation is completed for all the partial match determination rules 307.

  First, the failure cause partial match determination mechanism 308 extracts one failure cause complete match determination rule from the complete match determination rule 305 (step 901).

  Next, the failure cause partial match determination mechanism 308 newly creates and initializes a failure cause determination rule corresponding element 702 (step 902). At this time, data fields of a pointer 705, a failure cause determination condition number 706, and a failure cause determination condition satisfaction number 707 are created. Of course, zero (0) is recorded in the failure cause determination condition number 706 and the failure cause determination condition establishment number 707.

  Next, the failure cause partial match determination mechanism 308 uses the number M of failure cause determination conditions constituting the failure cause complete match determination rule extracted in step 901 as the failure cause of the failure cause determination rule corresponding element 702 created in step 902. It records in the judgment condition number 706 (step 903).

  Next, the failure cause partial match determination mechanism 308 accesses the log database 312 and extracts one of the logs stored for the partial match determination rule (step 904). In the case of this embodiment, one log relating to the failure cause determination condition that satisfies the condition is extracted.

  The failure cause partial match determination mechanism 308 compares whether the log extracted in step 904 relates to the failure cause complete match determination rule extracted in step 901 based on the log contents (step 905).

  If it is determined in step 905 that the log is not related to the failure cause complete match determination rule extracted in step 901, the failure cause partial match determination mechanism 308 returns to step 904 (step 906).

  On the other hand, if it is determined in step 905 that the log is related to the failure cause complete match determination rule extracted in step 901, the failure cause partial match determination mechanism 308 determines whether the failure cause determination condition of the partial match determination rule is successful. A flag indicating that the condition is satisfied is set in the information 708 (step 907).

  Thereafter, the failure cause partial match determination mechanism 308 updates the numerical value K of the failure cause determination condition satisfaction number 707 to K + 1 (step 908).

  Subsequently, the failure cause partial match determination mechanism 308 determines whether or not a log that has not been extracted remains in the log database 312 (step 909). At this time, if there is a log that has not been extracted, the failure cause partial match determination mechanism 308 returns to Step 904. On the other hand, if all the logs have been extracted, the failure cause partial match determination mechanism 308 proceeds to the next step 910.

  That is, the failure cause partial match determination mechanism 308 determines whether or not there remains a failure cause complete match determination rule that has not been read in step 901 in the complete match determination rule 305 (step 910). At this time, if there is a failure cause complete match determination rule that has not been extracted, the failure cause partial match determination mechanism 308 returns to step 901. On the other hand, if all failure cause complete match determination rules have been extracted, the failure cause partial match determination mechanism 308 ends the process of creating the decision table 311.

(J) Summary As described above, by adopting the failure cause analysis system 301 according to this embodiment, even if the event sequence notified from the monitoring system does not completely match the determination rule, it is partially established. The ratio of failure condition determination conditions and the contents of success or failure can be explicitly displayed on the screen.

  For this reason, the user can efficiently perform narrowing down of the failure cause determination condition with high possibility. In particular, the partial establishment status can be confirmed on the screen. That is, it is possible to individually check the failure cause determination condition that is not satisfied and the failure cause determination condition that is satisfied. For this reason, it is possible to efficiently narrow down failure cause candidates.

  In addition, by installing this function, it is possible to efficiently reach the cause of the failure even when the type and sequence of events that appear at the time of the failure are different from the assumed contents due to a change in the system configuration or the like.

(2) Others The monitoring target system 302, the monitoring system 303, and the failure cause analysis system 301 described above do not necessarily have to be physically built in one building. For example, each system may be constructed in a different building through a network.
Further, the above-described system may be constructed so as to straddle a country or a region according to the country.

  Further, each of the monitoring target system 302, the monitoring system 303, and the failure cause analysis system 301 described above may be operated not only by the same operator but also by different operators. For example, it may be operated separately by a business operator operating a data center and a business operator providing its maintenance management service.

  DESCRIPTION OF SYMBOLS 101 ... Failure cause analysis system, 102 ... Monitoring target system, 103 ... Monitoring system, 104 ... Connection adapter, 105 ... Complete match determination rule, 106 ... Failure cause complete match determination mechanism, 107 ... Failure cause determination result dashboard generation mechanism, DESCRIPTION OF SYMBOLS 108 ... Display monitor 109 ... Fault cause determination rule definition template 201 ... Fault cause complete match determination rule description 301 ... Fault cause analysis system 302 ... Monitoring target system 303 ... Monitoring system 304 ... Connection adapter 305 ... Complete Match determination rule, 306 ... Failure cause complete match determination mechanism, 307 ... Partial match determination rule, 308 ... Failure cause partial match determination mechanism, 309 ... Failure cause determination rule creation mechanism, 310 ... Failure cause determination result dashboard generation mechanism, 311 ... decision table, 312 ... 313 ... Display monitor, 314 ... Failure cause determination rule definition template, 401 ... Failure cause determination rule definition template, 402 ... Failure cause determination rule name, 403 ... Failure cause determination condition list, 404 ... Failure cause determination result action, 405 ... Failure marking determination condition, 501 ... Failure cause analysis result list display screen, 502 ... Ranking, 503 ... Match rate, 504 ... Failure cause determination rule name, 505 ... Failure cause determination condition establishment status, 506 ... Failure cause determination condition 601 ... Failure cause part determination rule set, 602 ... Failure cause part determination rule.

Claims (9)

For exact match judgment, which is a set of judgment rules that associates combinations of multiple fault cause judgment conditions defined for individual fault causes with actions to be executed when all fault cause judgment conditions of each combination are met Rules,
For partial match determination that is a set of determination rules in which individual failure cause determination conditions constituting individual determination rules of the complete match determination rule are associated with actions to be executed when each failure cause determination condition is satisfied Rules,
A first sequence for comparing a sequence of events notified in accordance with a system failure and the exact match determination rule to determine whether a determination rule in which all of the plurality of failure cause determination conditions are satisfied is detected. A determination unit;
A second determination for comparing an event sequence notified in accordance with a system failure and the partial match determination rule to determine whether or not a determination rule that satisfies each of the failure cause determination conditions is detected And
If the determination result in the first determination unit is affirmative, a screen displaying the identified cause of failure is generated, and if the determination result in the first determination unit is negative, the cause of the failure A display screen generation unit for identifying a failure cause candidate for which a determination condition is partially satisfied using a determination result of the second determination unit, and generating a screen for displaying the failure cause candidate in a list according to a predetermined rule; The display screen generation unit, according to a sorting condition that prioritizes a larger number of failure cause determination conditions constituting the failure cause candidate when the failure cause determination conditions in the failure cause candidates are the same , A failure cause analysis system that lists failure cause candidates. The display screen generation unit displays the number of failure cause determination conditions defined for each of the failure cause candidates and the individual establishment status of the determination result of the second determination unit as a set. The failure cause analysis system according to claim 1. The display screen generation unit expresses the number of failure cause determination conditions by the number of display blocks, and expresses the individual establishment status of the failure cause determination conditions by a difference in display mode of the display blocks. The failure cause analysis system according to claim 2. The failure cause analysis system according to claim 3, wherein the display screen generation unit has a display format for confirming the content of the failure cause determination condition corresponding to the display block. A determination rule that divides a plurality of failure cause determination conditions defined for individual failure causes into individual failure cause determination conditions, and stores a determination result related to each failure cause determination condition in a storage area as log information. The failure cause analysis system according to claim 4, further comprising: a determination rule generation unit that generates a determination for determination by the determination unit. Event sequence notified with system failure and exact match determination rule (Executed when a combination of multiple failure cause determination conditions defined for each failure cause and all failure cause determination conditions for each combination are satisfied. And a process for causing the first determination unit to determine whether or not a determination rule that satisfies all of the plurality of failure cause determination conditions is detected at the same time. ,
Event sequence notified when a system failure occurs and a rule for partial match determination (executed when each failure cause determination condition that constitutes each determination rule of the complete match determination rule and each failure cause determination condition is satisfied A set of determination rules associated with actions to be performed), and causing the second determination unit to determine whether or not a determination rule that satisfies each of the failure cause determination conditions is detected,
When the determination result in the first determination unit is affirmative, a process of generating a screen that displays the specified cause of failure;
When the determination result in the first determination unit is negative, the failure cause candidate in which the failure cause determination condition is partially satisfied is identified using the determination result of the second determination unit, and the failure cause Processing a computer to generate a screen for displaying candidates in a list according to a predetermined rule,
In the process of generating the screen, if the failure cause determination conditions in the failure cause candidates are the same, the cause of the failure is determined according to the sort condition that gives priority to the larger number of failure cause determination conditions constituting the failure cause candidate. A program that causes a computer to execute processing for displaying a list of candidates. When the determination result in the first determination unit is negative, for each of the failure cause candidates, the number of corresponding failure cause determination conditions and the individual establishment status thereof are displayed as a set. The program according to claim 6. When the determination result in the first determination unit is negative, the number of the failure cause determination conditions is expressed by the number of display blocks, and the individual establishment status of the failure cause determination conditions is displayed on the display block. The program according to claim 7, wherein the program is expressed by a difference in form. The program according to claim 8, wherein if the determination result in the first determination unit is negative, the program has a display format for confirming the content of the failure cause determination condition corresponding to the display block.

Images