JP2005316728A - Fault analysis device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To specify a fault part from a fault condition of a relevant work configuration appliance and resources and to immediately check a function of the specified fault part for easily specifying the fault part. <P>SOLUTION: This fault analysis device is provided with a work configuration information table 41 storing a configurative appliance constituting work of a monitoring object system and resources for each work, a monitoring method information table 42 storing monitoring methods for the respective configuration appliances and resources, a monitoring part 51 monitoring the monitoring object system, and a check method specification part 33 estimating work causing a fault by referring a work configuration information table when the monitoring part detects the fault of the monitoring object system, extracting the configuration appliance and resources constituting the work causing the fault, and applying a monitoring method for the configuration appliance and resources constituting the work. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an apparatus and a method for estimating a cause of a failure from a failure situation occurring in a system and identifying an element that is a source of the failure.

In a distributed computer system, various types of monitoring are performed in the system in preparation for occurrence of a failure. For example, there is a life / death monitoring of a system or an element constituting the system based on an inquiry from the outside of the device represented by ping monitoring. In addition to failures, there are resource monitoring such as the presence / absence of a process by an agent in a device, disk free space, CPU usage status, response time monitoring from the viewpoint of a user of business processing, and the like.
By the way, these fault monitoring is performed independently according to each monitoring setting, and there is no mutual cooperation. Some are monitored in a short cycle, and some are monitored in a long cycle, and there is a time difference in detecting a plurality of spillover faults induced by one root cause. Furthermore, since only superficial fault phenomena are collected individually, special techniques are required to analyze the source of the fault, and it takes time to identify the root cause of the fault. In addition, by shortening the monitoring cycle, the real-time property of fault detection is improved and the simultaneousness of detection is also improved, but the load of monitoring processing increases due to the collection of data and the cause analysis amount in the system.

Further, a technique for centrally managing the business process flow and collecting the execution results to quickly collect the failure data, such as “Management Manager Computer, Recording Medium, and Computer Operation Management Method” of Patent Document 1, There are “business flow operation information acquisition method and business flow system” in Document 2, which are techniques for tracking execution results from the viewpoint of business processing flow. Accordingly, even if there is a notification of the fact that a failure has occurred, the computer resource used for executing the business process, an external service, and a network path failure are not known. In other words, there is no disclosure regarding failure analysis, and no identification of the root cause of the failure is stated.
JP-A-11-134306 JP 2003-67222 A

  The conventional system monitoring apparatus is configured as described above, and simply collects the failure status individually. Alternatively, there is a problem that it is difficult to find the underlying failure only by estimating the cause of the failure by referring to a failure database in which past failures are recorded.

  The present invention has been made to solve the above-mentioned problems. The failure part is identified from the failure state of the related business component device or resource, and the function is immediately confirmed for the identified failure part, so that the failure can be easily performed. The purpose is to identify the part.

The failure analysis apparatus according to the present invention includes a business configuration information table that stores, for each business, component devices and resources that configure the business of the monitoring target system;
A monitoring method information table for storing an individual monitoring method for each component device / resource;
A monitoring unit for monitoring the monitored system;
When the monitoring unit detects a failure of the monitored system, the business configuration information table is referred to, the business in which the failure has occurred is estimated, and the constituent devices and resources that configure the business in which the failure has occurred are extracted. And a related failure confirmation method specifying unit that applies a monitoring method of component devices and resources constituting the business.

  According to the present invention, the business configuration information table, the monitoring method information table, and the business in which a failure has occurred are estimated, the constituent devices and resources constituting the business are extracted, and the monitoring method for the constituent devices and resources is applied. Since the fault confirmation method specifying unit is provided, there is an effect that the fault part can be easily identified by immediately confirming the function without requiring high-level knowledge for analyzing the fault.

Embodiment 1 FIG.
If there is a system failure, the root cause of the failure should be in one of the resource groups including the devices and software that make up the business. From this, when a system failure is detected, it is possible to immediately identify the failure of the related business component device or resource, and immediately confirm the function for the estimated device or resource, thereby identifying the fundamental failure portion in a short time. This does not increase the normal system monitoring load.
FIG. 1 is a block diagram showing the configuration of a failure analysis apparatus according to Embodiment 1 of the present invention based on the above concept.
In the configuration of FIG. 1, the monitoring target system 1 is monitored by the monitoring units 51 and 52 in a normal cycle by a failure analysis device at a constant period or the like. Similarly, the failure monitoring statuses of the components in the monitored system are monitored by the failure monitoring units 53 and 54 as needed.
As will be described in detail later, if you divide the monitoring work for each client and monitor the monitoring program up to the response of the resources used by the broken down work, you can understand at least what kind of work caused the failure. It is not difficult to do.

In addition, the failure analyzer detects the occurrence of a failure from the monitoring data obtained by these monitoring units, extracts the devices and resources related to the failure, and specifies the method of checking the failure status of the related failure. There is a confirmation method specifying unit 30. The related failure confirmation method identification unit 30 internally includes a failure occurrence business identification unit 31 that estimates the most likely failure from the failures that have occurred, and related resource extraction that extracts devices and resources for performing the business process And a confirmation method identification unit 33 for identifying a method for confirming the failure status of the devices and resources.
Further, the failure analysis apparatus includes a system information group 40 which is referred to when the related failure confirmation method specifying unit 30 estimates a failure portion. In the information group 40 regarding this system, there are a business configuration information table 41 and a monitoring method information table 42 for each target. Furthermore, there is a failure information collection control unit 20 that controls the entire series of processes described above.

FIG. 2 is a diagram illustrating a relationship between an example of a specific configuration shown as the monitoring target system 1 in FIG. 1 and a business using a specific component device or resource.
In the figure, servers 101 to 106 as computer nodes have various programs 111 to 117 therein, and are connected to each other via network devices 121 to 124 and network services 125 and 126.
A dotted line 130 in the figure is a business configuration that constitutes a business process “business 1”. “Business 1” given as an example has a configuration in which a processing request from the client 1 is processed by the business program 1 in the business server 1 101 via the business program 3 and business program 4 in the business server 3 103. The business program 3 uses a shared service program 1 115 such as a name service or an authentication service in the processing. The dotted line 130 of “business 1” represents these business configurations with lines.
As a monitoring method performed by the normal time fault monitoring unit 51 and the like, for example, the business program 1 is accessed from the client 1 as the business 1, and if the same business program 1 is used and accessed from the client 2, the business program 1 is named as the business 11 By doing so, it is possible to specify the work at the time of failure occurrence at a fairly detailed level. Furthermore, if the monitoring program still monitors the behavior of the business program 1 and the client program 1, it is easy to detect the occurrence of a failure.

FIG. 3 is a diagram showing an example of data in the business configuration information table 41 in FIG. 1, and shows the business configuration of “business 1” shown in FIG. 2 in a table format. In this table, devices and resources that constitute the business 1 and dependent devices and resources that are prerequisites for the operation of the devices and resources are represented. Of course, other business components and resources such as business 2 and business 3 are also stored and represented.
FIG. 4 is a diagram showing an example of data in the monitoring method information table 42 for each target in FIG. 1. For each component of the target system shown in FIG. It is a representation.

Next, the operation will be described.
The normal failure monitoring units 51 and 52 in FIG. 1 notify the failure information collection control unit 20 of failure detection when detecting a failure in the monitored system. Here, as an example, it is assumed that there is no business response in the client program 1 116 in FIG. That is, since no response is returned within the set time, the failure is detected.
Upon receiving this detection, the failure information collection control unit 20 passes this information to the related failure confirmation method specifying unit 30. In the related failure confirmation method identifying unit 30, first, the failure occurrence task identifying unit 31 estimates that the failure is a failure related to the task 1 from the response failure of the client program in the content of the failure. This is the trouble task estimation step S101 in FIG. Next, the related resource extraction unit 32 refers to the business configuration information table 41 shown in FIG. 3 and extracts devices and resources related to the business 1. This is the component device / resource estimation step S102 of FIG. Further, the confirmation method specifying unit 33 specifies a failure monitoring method for each device or resource from the monitoring method information table 42 for each target shown in FIG. This is the monitoring method specifying step S104 in FIG. FIG. 5 is a diagram illustrating an example of a monitoring method information table related to the job 1 created by the related failure confirmation method specifying unit 30.

Upon receiving the monitoring method information from the monitoring method information table 42b of the job 1 in FIG. 5, the failure information collection control unit 20 passes through the failure monitoring units 53 and 54 as needed, and the equipment related to the job 1 in the monitored system. And the failure status of the resource are immediately executed in order for each business component device / resource configured based on the “monitoring method (current status checking method)” of FIG. This is the individual failure confirmation execution step S105 of FIG. For example, if ping NW1-1 is executed for the NW device 1 in FIG. 5 and a predetermined response is not returned, it can be confirmed and detected that the NW device 1 is the cause of the failure. If it is confirmed and detected that a failure has occurred in the NW device 1 121 in FIG. 2 in this failure status confirmation process, the NW device 1 is the root cause where the client program 1 116 has no business response. Becomes clear.
FIG. 13 is a diagram showing an example of a table related to the normal monitoring method of this system. According to the example of the monitoring interval of each monitoring function shown as a reference in this example, the monitoring interval of the NW device 1 is 20 minutes. Without the failure analysis device in the present embodiment, the worst time difference of 20 minutes occurs until the failure of the root cause portion is detected.
In this way, compared with the situation where the conventional monitoring device only performs monitoring of individual devices and resources with the monitoring period set independently, the use of this failure analysis device This has the effect of quickly identifying the root cause of the failure.

Embodiment 2. FIG.
In the first embodiment, the example has been described in which the confirmation method specifying unit 33 performs failure confirmation individually on the business configuration devices and resources shown in FIG. However, the method of sequentially checking the failure is inefficient. In order to estimate a component device / resource in which a failure has occurred, it is natural to refer to a failure that has occurred in the past and to estimate that the component device / resource is in the same situation. In the present embodiment, the work in which a failure has occurred is estimated based on such past history.
FIG. 6 is a configuration block diagram showing the failure analysis apparatus according to the present embodiment. In the figure, a failure history information table 43 is added as a new component added to the previous embodiment. The failure history information table 43 is referred to when the related failure confirmation method specifying unit 30 performs processing.
FIG. 7 is a diagram showing an example of specific data stored in the failure history information table 43, and records the failure history for each monitored system device and resource.

Next, the operation will be described.
8 is a table in which “failure occurrence date” and “failure severity” information obtained from the failure history information table 43 in FIG. 7 is added to the monitoring method information table in FIG. 5 created in the first embodiment.
If it is estimated that a specific task is faulty, a large number of devices and resources constituting the estimated task may be extracted. In that case, it takes a long time to obtain the result by applying the monitoring method shown in FIG. 5 to all the extracted devices and resources in order without giving priority control to the failure statuses of all these extracted devices.
Therefore, it is assumed that a portion where a failure has occurred in the past has a high probability that a failure will occur again. First, the failure is preferentially checked and primary information is reported. After confirming all the cases, there is an effect of speeding up the disaster recovery measures.
In the example of FIG. 8, failure confirmation is preferentially performed for the business program 1, the NW device 1, and the shared service program 1.

Embodiment 3 FIG.
In the third embodiment, failure monitoring priority processing is performed in the same manner as in the second embodiment. Information used for priority processing determination is a failure history information table 43c illustrated in FIG. Although the functional block diagram is omitted here, this failure occurrence frequency information table includes this frequency item in the failure history information table 43 of FIG. Refer to the frequency field.

As an operation in this case, as in the second embodiment, it is assumed that there is a high probability that a failure will occur in a portion where the failure occurrence frequency is high. Confirm failure and report primary information. After confirming all the cases, there is an effect of speeding up the disaster recovery measures.
In the example of FIG. 9, if priority monitoring is performed with a threshold value of three times or more, for example, failure check is first performed on the NW device 1 and the business program 1. However, the NW service 2 does not correspond to the business 1.

Embodiment 4 FIG.
In this embodiment, failure monitoring priority processing is performed in the same manner as in the second embodiment. Information used for priority processing determination is a system change history information table 44 shown in FIG. Although the functional block diagram is omitted here, the system change history information table 44 is a table equivalent to the failure history information table 43 in FIG. 6 and is provided in the same portion as the failure history information table 43. The system change history information table 44 is referred to when the related failure confirmation method specifying unit 30 performs processing.

As in the second embodiment, it is assumed that there is a high probability that a failure has occurred in a portion where the system has been changed. First, the failure is preferentially checked and primary information is reported. Confirming all cases after that has the effect of speeding up failure recovery.
In the example of FIG. 10, if priority is given to all records with change records, failure check is preferentially performed for the business program 3, the business program 1, the NW device 3, the NW device 1, and the shared service program 1. However, the NW service 2 does not correspond to the business 1.

Embodiment 5 FIG.
In the present embodiment, failure monitoring priority processing is performed in the same manner as in the second, third, and fourth embodiments, but the information used for priority processing determination is the system device / resource importance level information table shown in FIG. 45 information. Although the functional block diagram is omitted here, the system device / resource importance level information table 45 is a table equivalent to the failure history information table 43 in FIG. 6 and is provided in the same portion as the failure history information table 43. When the related failure confirmation method specifying unit 30 performs processing, the system device / resource importance level information table 45 is referred to.

When identifying the root cause of a failure, the failure of a device or resource that is greatly affected by the failure should be detected and countermeasures taken immediately. For this purpose, the impact of the failure on the work can be minimized by first checking the failure and reporting the devices and resources with high importance first.
In the example of FIG. 11, first, the failure check of the business program 1 having the highest severity level 10 is performed, and then sequentially confirmed in the order of higher severity level.

Embodiment 6 FIG.
In this embodiment, priority processing for fault monitoring is performed in the same manner as in the second, third, and fourth embodiments, but the information used for priority processing determination is the usage per unit time for each resource shown in FIG. This is a frequency information table 46. For example, in the configuration of FIG. 1, the failure information collection control unit 20 periodically checks the usage frequency of each business component device and resource by using the failure monitoring unit 53 as needed. The investigation result is recorded and managed in the usage frequency section of the resource usage frequency information table 46 of FIG. The frequency of use may be surveyed for an arbitrary period. The failure information collection control unit 20 activates the failure monitoring unit 53 at any time and counts whether the target resource is open (start) or closed (end). I understand. If this is further integrated, the relative usage frequency can be determined. This resource usage frequency table is provided in the same portion as the failure history information table 43 of FIG.
The resource usage frequency information table 46 is referred to when the related failure confirmation method specifying unit 30 performs processing.
When identifying the root cause of a failure, a resource with a lower use frequency described in the resource use frequency information table 46 may have a remaining bug or the like and may have a failure.

Embodiment 7 FIG.
The apparatus of the present invention is intended to efficiently detect a system failure. However, depending on the monitoring method, there may be a case where a false detection is not caused by a failure other than the failure to be monitored itself.
FIG. 13 shows an example in which life / death is confirmed from the outside of the network device by ping, and the NW device 3 124 is monitored from the ping monitoring server 107. At this time, a ping response error (no response) to the NW device 3 is detected not only by the failure of the NW device 3 but also by the failure of the NW device 2 or the NW device 4 on the monitoring path. That is, it can be said that ping monitoring from the position of the ping monitoring server 107 to the NW device 3 depends on the NW device 2 and the NW device 4.

In the present embodiment, before performing the processing of the first to sixth embodiments, it is first confirmed whether there has been a false detection state in detecting a system failure.
FIG. 14 is a diagram showing an example of the monitoring dependency relationship table 57 showing the monitoring dependency relationship regarding normal monitoring. For example, when a failure of the NW device 3 is detected, first, the failure status of the NW device 2 and the NW device 4 registered in the “dependent part of the monitoring function causing the false detection” is confirmed. If there is no failure in these, the processing in the first to sixth embodiments is performed.
By performing this preliminary process, the accuracy of the root failure location detection process is improved.

Embodiment 8 FIG.
In the processing of the first to sixth embodiments, the failure status of the device or resource extracted from the business configuration is confirmed by the registered confirmation method, but there may be a case where no failure status is recognized for all the devices and resources. . In this embodiment, even in such a case, the secondary step, the tertiary, and the like, which change the viewpoint of the failure confirmation method or prepare for collecting failure analysis information and performing human analysis for more detailed analysis. It has means that can execute the processing of the step.
FIG. 15 is a monitoring method information table 42d obtained by extending the monitoring method information table 42 for each target shown in FIG. 4, and is an example in which secondary actions are registered. In this example, an information collection method for failure analysis is registered as a secondary action.

FIG. 16 is a flow for executing the operation in the present embodiment. Even if the failure status of the related devices and resources is confirmed according to the primary action list, if neither of them is an obvious failure state, the secondary action is switched to and the treatment is executed again through steps S61 to S65.
In order to implement this example, it is necessary to add elements according to the processing to be performed, such as newly providing failure analysis information collection units 55 and 56 in FIG. 16 showing the device to which the component is added. .
Even if the failure location cannot be detected by this processing, alternative measures such as collecting failure analysis information for human analysis in advance can be executed, and system failure countermeasures can be accelerated. Is peeled off.

Embodiment 9 FIG.
In each of the above embodiments, the failure analysis apparatus has been described as being configured by hardware. However, the apparatus is not limited to this, and a general-purpose processor and memory may be used to describe steps in a software program in the memory, and equivalent operations may be executed by these program steps.
FIG. 18 is a diagram showing a flowchart for realizing the operation in the first embodiment by such program steps. In the figure, as a program step, a function corresponding to the failure occurrence work specifying unit 31 is assembled in S101. When the normal-time failure monitoring unit 151 notifies failure detection, monitoring is started as step S100 for monitoring this notification. Hereinafter, the function corresponding to the related resource extraction unit 32 is assembled in S1 and S2, and the function corresponding to the confirmation method specifying unit 33 is combined in S104 and S105. In addition, the functions of the individual component device / resource selection priority in the second to sixth embodiments are assembled in S103.
Furthermore, if the function of the flowchart shown in FIG. 18 is created as a program, the failure analysis apparatus described in the above embodiments can be configured by loading the program into a general-purpose computer.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows the failure analysis apparatus in Embodiment 1 of this invention. 3 is a diagram illustrating a relationship between a business 1 that is a monitoring target system according to the first embodiment and a specific component device / resource. FIG. 6 is a diagram illustrating an example of data in a business configuration information table according to Embodiment 1. FIG. 6 is a diagram illustrating an example of data in a monitoring method information table in the first embodiment. FIG. 6 is a diagram illustrating an example of monitoring method information created by a related failure confirmation method specifying unit in Embodiment 1. FIG. It is a block diagram which shows the failure analysis apparatus in Embodiment 2 of this invention. 10 is a diagram illustrating an example of data in a failure history information table according to Embodiment 2. FIG. 10 is a diagram illustrating an example of monitoring method information created by a related failure confirmation method specifying unit in Embodiment 2. FIG. FIG. 10 is a diagram illustrating an example of data in a failure history information table in the third embodiment. FIG. 20 is a diagram illustrating an example of data in a system change history information table in the fourth embodiment. FIG. 20 is a diagram illustrating an example of data in a system device / resource importance level information table according to the fifth embodiment. FIG. 20 is a diagram illustrating an example of data in a usage frequency information table in the sixth embodiment. FIG. 24 is a diagram for explaining an example of a method for confirming component / resource failure from the outside in the seventh embodiment. FIG. 20 is a diagram illustrating an example of data in a monitoring dependency relationship table in the seventh embodiment. FIG. 20 is a diagram illustrating an example of data in a monitoring method information table in the eighth embodiment. FIG. 20 is a flowchart showing a failure analysis operation in the eighth embodiment. FIG. 10 is a configuration block diagram illustrating a failure analysis device according to an eighth embodiment. It is an operation | movement flowchart which shows the failure analysis method in Embodiment 9 of this invention.

Explanation of symbols

  20 fault information collection control unit, 30 related fault confirmation method identification unit, 31 fault occurrence business identification unit, 32 related resource extraction unit, 33 confirmation method identification unit, 40 system information group, 41 business configuration information table, 42, 42b , 42c, 42d Monitoring method information table (for each target), 43, 43c Failure history information table, 44 System change history information table, 45 System device / resource importance information table, 46 Usage frequency information table, 51, 52 Normal time Fault monitoring unit, 53, 54 Anytime fault monitoring unit, 55, 56 Fault analysis information collection unit, 57 Monitoring dependency relationship table, S61 Extraction of previous work related parts and extraction of primary step actions, S62 Action execution for all targets, S63 fault location detection step, S64 next action registration step , S65 next action extraction step, S101 failure task estimation step, S102 component device / resource extraction step of relevant job, S103 component device / resource priority selection step, S104 component device / resource confirmation method specifying step, S105 priority Individual failure confirmation execution step based on S106, confirmation execution end confirmation step in S106.

Claims (10)

A business configuration information table that stores the components and resources that make up the business of the monitored system for each business;
A monitoring method information table for storing an individual monitoring method for each component device / resource;
A monitoring unit for monitoring the monitored system;
When the monitoring unit detects a failure of the monitored system, the business configuration information table is referred to, the business in which the failure has occurred is estimated, and the constituent devices and resources that configure the business in which the failure has occurred are extracted. A failure analysis apparatus comprising: a related failure confirmation method identification unit that applies a method for monitoring component devices and resources constituting the business. It has a failure history information table that stores past failure histories of component devices and resources.
The failure analysis apparatus according to claim 1, wherein the related failure confirmation method specifying unit applies a monitoring method of component devices / resources constituting a business with reference to the failure history information table. A system change history information table for storing past system change history of component devices / resources is provided.
The failure analysis apparatus according to claim 1, wherein the related failure confirmation method specifying unit applies a monitoring method of component devices and resources constituting a business with reference to the system change history information table. It has a component / resource importance information table that specifies the importance of component devices / resources in the monitored system.
The failure analysis apparatus according to claim 1, wherein the related failure confirmation method identification unit applies a monitoring method of component devices / resources that constitute a business with reference to the component / resource importance information table. A monitoring dependency table that stores the mutual influence relationship between component devices and resources in the monitored system
The failure analysis apparatus according to claim 1, wherein the related failure confirmation method specifying unit applies a monitoring method of component devices / resources constituting a business with reference to the monitoring dependency relationship table. In addition to storing the individual monitoring method of each component device / resource as a primary action, the monitoring method information table stores other monitoring methods as secondary actions,
The related failure confirmation method specifying unit applies a monitoring method of component devices / resources constituting a business based on the secondary action if necessary following the primary action of the monitoring method information table. The failure analysis apparatus according to claim 1. Analysis method in an analysis apparatus comprising: a business configuration information table for storing, for each business, component devices / resources that configure a business of the monitored system; and a monitoring method information table for storing individual monitoring methods for each component device / resource. In
Monitoring the monitored system;
When the monitoring step detects a failure of the monitored system, referring to the business configuration information table to estimate the business in which the failure has occurred;
A step of extracting component devices / resources constituting the business in which the failure has occurred;
A failure analysis method comprising: a step of confirming and executing a monitoring method of component devices and resources constituting the business. Furthermore, in the analysis method of the analysis apparatus provided with the failure history information table for storing the past failure history of the component device / resource,
Following the step of extracting component devices / resources, there is provided a component device / resource priority order selection step for selecting the priority order of component devices / resources to be checked for association with reference to the failure history information table, and the priority order selection step 8. The failure analysis method according to claim 7, wherein a step of confirming and executing a monitoring method of component devices / resources is performed after execution of the step. In the monitoring method information table, in addition to storing the individual monitoring method of each component device / resource as a primary action, in the analysis method of the analysis device that stores other monitoring methods as secondary actions,
A step of confirming and executing the monitoring method of the component device / resource based on the secondary action, if necessary, following the step of the primary action of confirming and executing the component device / resource monitoring method is provided. The failure analysis method according to claim 7. Configure as a computer executable program,
Storing the component devices and resources that constitute the business of the monitored system for each business and configuring the business configuration information table;
Storing an individual monitoring method for each component / resource and configuring a monitoring method information table;
Monitoring the monitored system;
When the monitoring step detects a failure of the monitored system, referring to the business configuration information table to estimate the business in which the failure has occurred;
A step of extracting component devices / resources constituting the business in which the failure has occurred;
A failure analysis program comprising: a step of confirming and executing a monitoring method of component devices and resources constituting the business.

Images