KR20170127876A - System and method for dealing with troubles through fault analysis of log - Google Patents

Abstract

The present invention relates to a system and a method for analyzing a fault in a cluster-shaped high-performance computing system, in which a large-scale computing system node formed of various resources is connected through a network, and responding to a trouble. According to the present invention, the trouble responding system based on a log fault analysis comprises: a resource log collector for collecting an event or log information from a configured node; a fault log analysis device transmitting fault related information when it is recognized by analyzing the log information that the system has a trouble, analyzing associated resources of a resource with a fault, and analyzing the severity of the fault; a resource fault processing device resolving a faulty state of the resource when a trouble is recognized; a fault log reporting device reporting fault related information to a user; and a policy and filter management device setting and storing an event or a log collection policy, information on a fault analysis filter, resource association information, and a failure notification policy.

Description

TECHNICAL FIELD [0001] The present invention relates to a system and method for failure diagnosis based on log defect analysis,

The present invention relates to a system and method for analyzing defects in a cluster-like high-performance computing system in which a large-scale computing system node made up of various resources is network-connected, and for responding to a failure.

In a high performance computing system composed of a large number of nodes, a large number of managed nodes operate in an interlocked manner. Therefore, a failure of a system or a service is recognized and a service system is diagnosed to determine whether the entire system is a failure or a failure of a partial system.

When an error or a log is generated, the node becomes a state in which it is difficult to proceed with a job or service that was being executed. According to the related art, the error state is simply transmitted to the manager, Or the availability and safety of the service is deteriorated.

SUMMARY OF THE INVENTION The present invention has been proposed in order to solve the above-described problems, and it is an object of the present invention to provide a high performance computing cluster system including a computing node having a plurality of resources connected by a high- And to provide a failure response system and method based on log defect analysis capable of responding to pre-existing faults and errors.

The fault analysis system based on log defect analysis according to the present invention comprises a resource log collector for collecting event or log information from a configuration node, and a fault log analyzer for analyzing log information and transmitting defect related information when a system fault is recognized, A fault log analyzer for analyzing the associated resources of the resource and analyzing the severity of the fault, a resource fault handler for resolving the fault state for the resource when the fault is recognized, and a fault log report for reporting the fault related information to the user And a policy and filter manager for setting and storing an event or log collection policy, information on a defect analysis filter, resource association information, and a failure notification policy.

A method for responding to a fault based on log defect analysis according to the present invention comprises the steps of: setting a collection policy, an event filter and a defect filter information and resource association information for an event or a log; collecting event logs and software log information of a hardware resource from a computing node; Analyzing the defect using the defect filter and the resource association information, and storing the defect log analysis information in a normalized form when it is determined that a defect has occurred as a result of defectiveness analysis, notifying occurrence of a defect, And performing the steps of:

The system and method for fault-tolerance-based failure analysis according to the present invention analyze a hardware resource or software defect of a node and its severity in a high performance computing cluster system environment composed of a computing node having a plurality of resources, By executing the failure handling routine and notifying the administrator of the failure, it is possible to proactively respond to a system failure, to recover the system in question in a short period of time, and to improve the stability and management efficiency of the system.

In addition, it is possible to shorten the time from failure recognition to recovery through the pre-registered failure and defect response processing routine, thereby improving the availability and reliability of the system and the service.

In addition, it is possible to effectively grasp and manage the fault and operation state of the system by using the statistical data on the occurrence of defects.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating an entire system to which a fault analysis system based on a log defect analysis according to an embodiment of the present invention is applied.
2 is a detailed block diagram illustrating an entire system to which a fault analysis system based on a log defect analysis according to an embodiment of the present invention is applied.
3 is an exemplary diagram illustrating resource associations according to an embodiment of the present invention.
4 is a diagram illustrating defect severity according to an embodiment of the present invention.
5 is a flowchart illustrating a method of responding to a fault based on log defect analysis according to an embodiment of the present invention.
6 is a flowchart illustrating a log-based defect analysis and a defect-handling operation according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, advantages and features of the present invention and methods of achieving them will be apparent from the following detailed description of embodiments thereof taken in conjunction with the accompanying drawings.

The present invention may, however, be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, And advantages of the present invention are defined by the description of the claims.

It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. &Quot; comprises "and / or" comprising ", as used herein, unless the recited component, step, operation, and / Or added.

Prior to describing the preferred embodiments of the present invention, the background of the present invention will be described below in order to facilitate the understanding of those skilled in the art.

In high-performance computing systems consisting of large-scale nodes, dozens of petaflops (PFlops) systems consisting of thousands of nodes are actually being built and serviced, and within a few years from now, exascale systems will emerge.

In an ExScale system that uses a large number of nodes, it can be an event in which rare errors and failures occur each time, which can not be found in a conventional low-level hybrid computing system.

The ExaScale system has a large number of computing resources, which together constitute a large scale networking, and these resources are software managed systems that cause resiliency problems across multiple domains and layers .

Various hardware resources constituting the node and software running on the operating system have different forms and contents of events and log messages that occur in an error or error situation and do not have a consistent form.

Researches have been made to standardize defect related messages. However, since the implementation and development must be performed in compliance with the provided interface, there is a problem that it is difficult to apply to various devices and software that have already been developed.

Although industry standard log information such as the system event log of the hardware device or the system log record of the daemon software is used but not all devices or software actually exist, There is a problem that it is difficult to process with messages or information.

According to the related art, when an error or a log is generated and a job or service becomes difficult to proceed in a node, such an error situation is simply handed to the manager, and the manager is processed in a manner corresponding to the error situation. There is a problem that the availability and stability of the service are deteriorated.

According to the related art, there is provided a function that a threshold value or a condition for an error situation is set in some commercial management systems and an error processing routine is automatically performed when a corresponding situation occurs. However, In case of an error, normalization can be performed by re-executing the resource, but in the case of an error caused by hardware or other resources, a fundamental problem can not be solved.

The present invention has been proposed in order to solve the above problems. In the high performance computing cluster system composed of a computing node having a large number of resources connected by a high-speed low-latency network, a fault or error is generated based on hardware resources or logs collected from software Analyzing the fault, and performing a preliminary response to the fault and reporting the fault.

According to the present invention, log information generated during the operation of hardware and software resources is collected, a filter is applied to analyze and store the occurrence of defects and errors, performs defect repair by performing a set fault processing routine, And provides fault analysis and monitoring to the administrator periodically.

Also, it is possible to check the defect and normal operation of the resource associated with the defective resource through the resource association information, execute the error processing routine for the defect, and repeat the same problem repeatedly If it does occur, it does not just execute notification or error handling routine, but stops use and operation of the resource, and makes a separate notification to the administrator of a serious level of failure.

According to the present invention, a fault-related filter and a filter defined by a user are fundamentally analyzed to analyze the occurrence and failure of a defect, and a correlation between hardware resources or software to be log- Analyzes faults for resources associated with defective resources, and performs fault handling routines for normal operation.

1 is a block diagram illustrating an entire system to which a fault analysis system based on a log defect analysis according to an embodiment of the present invention is applied.

The fault analysis system 100 according to an embodiment of the present invention periodically collects event or log information from the entire configuration node 200. At this time, Collect.

In each node 210 including the hardware resource 230 and the software resource 220, event and log information are generated in various situations.

The log defect analysis-based fault response system 100 analyzes the collected information by applying a filter for defect analysis, stores the analyzed information and the defect information in the database 110, To the user.

The user confirms the notification of the defect log through the defect notification receiving unit 320 of the user tool 300 and confirms the detailed defect information through the defect log viewer 310. [

The log defect analysis-based failure response system 100 performs defect and failback by calling a defect processing routine registered by the user rather than merely performing defect notification.

In addition, the fault analysis system 100 based on the log defect analysis uses the association mapping information between the resources to check whether or not a defect associated with a resource associated with the fault occurrence resource is defective, and sequentially performs the correspondence thereto.

2 is a detailed block diagram illustrating an entire system to which a fault analysis system based on a log defect analysis according to an embodiment of the present invention is applied.

The log defect analysis based fault handling system according to an embodiment of the present invention includes a resource log collector 120 for collecting event or log information from a configuration node, A defect log analyzer 130 for analyzing an associated resource of the defective resource and analyzing the severity of the defect, and a fault log analyzer 130 for detecting a fault, A defect log processor 150 for reporting defect related information to a user and a policy setting and storing information for event or log collection policy, defect analysis filter, resource association information and fault notification policy, And a filter manager (160).

The policy and filter manager 160 according to an embodiment of the present invention stores information on a defect analysis filter including a defect-related filter and a user-defined filter that are basically provided in the system.

The resource log collector 120 according to an exemplary embodiment of the present invention collects events or log information generated from hardware resources and software resources of a node. The resource log collector 120 periodically collects events and log information from the policy and filter manager 160, Or log information.

At this time, the resource log collector 120 parses the hardware event log and the software log, and stores them in the normalized format.

The normalized format for storing log information of hardware resources is shown in [Table 1] below.

In this case, filter is a field indicating error level, seq is a number field, host is a field indicating host name, date is a field indicating the date when the event occurred, time is a field indicating time at which the event occurs, A tag is a field for storing tag information, and mail_yn is a field for indicating whether mail is sent to an event that has occurred.

The normalized format for storing the log information of the software is shown in [Table 2] below.

At this time, host is a field indicating the host name, facility is the field where the event occurs, daemon, authopriv, kern, etc., priority is a field indicating error level (used as filter), tag is a field indicating tag information, date is a field indicating the date when the event occurred, time Msg is a field in which the contents of the generated event are stored, seq is a field in which a number is indicated, and mail_yn is a flag indicating whether or not mail is sent to the generated event Field.

The defect log analyzer 130 according to an embodiment of the present invention analyzes log information by applying a defect analysis filter set from the policy and filter manager 160 and extracts and classifies the collected information to infer the root cause of the failure .

The defect log analyzer 130 identifies a defect for a resource associated with the corresponding defect resource by using the resource association information as the association mapping information between the resources.

FIG. 3 is a diagram illustrating resource associations according to an embodiment of the present invention. Processes A, B, C, and D, which are software resources, require processes F, G and H for operation.

If the occurrence of a defect is confirmed in process F as a result of the defect analysis, the defect log analyzer 130 checks the defect analysis or operation state for processes A and B that are related to process F. [

Similarly, if a problem is identified in process G, the defect log analyzer 130 performs defect analysis and verification for processes B and C.

According to the present invention, by analyzing and responding to other related resources that are affected by a defect occurrence resource through mapping setting of an associated resource, rather than simply analyzing or responding to a defect occurrence occurrence for a defect occurrence resource, It has the effect of supporting defect management function.

The resource defect processor 140 according to the embodiment of the present invention analyzes the defect logs of the corresponding resources as defect or defect as a result of analysis by the defect log analyzer 130, And a statistical information storing unit 110. The defect management unit 110 provides a function of calling a defect processing routine registered in advance to resolve a defect state for a corresponding resource and to perform a normal operation.

Also, as described above, the resource defect processor 140 normalizes the associated resource by eliminating the defect state according to the analysis result of the resource associated with the corresponding resource in which the defect occurs.

The defect log report unit 150 according to an embodiment of the present invention may output defect log information to a screen according to a user's request or may transmit defect related information through a mail or a message according to a preset failure notification policy send.

The user can confirm the detailed defect information through the defect log viewer 310 shown in FIG. 1, and check the mail or the message received from the defect log reporter through the defect notification receiving unit 320.

The defect log analyzer 130 according to the embodiment of the present invention analyzes the defect severity using the number of occurrences of defects, the period, and the defect log information.

4 is a diagram illustrating defect severity according to an embodiment of the present invention.

The defect severity is determined by the defect log analysis information and the defect level extracted from the defect statistical information, the defect occurrence frequency, and the frequency of occurrence of defects.

The set value of the fault occurrence frequency and the frequency of occurrence of the fault can be set separately by the administrator according to the system or resource situation.

The defect level is determined by the level, as shown in FIG. 4, and the failure response mechanism is determined.

Levels 3 and 4 are the levels at which failure handling routines are required, level 3 identifies the operating status of the resource, invokes the fault handling routine, and level 4 acts as the fault handling routine.

The defect severity is calculated to be high if the defect occurrence cycle is shortened even if the defect occurrence frequency is small even if the defect occurrence frequency is small.

In addition, even if it is located in the same occurrence frequency section, the defect severity varies depending on the change in the generation period and the generation period.

If the defect severity exceeds the threshold value, the resource defect processor 140 stops using the corresponding defect resource, and the defect log reporter 150 notifies the user of the interruption of the use of the corresponding defect resource according to the defect severity.

According to the present invention, by introducing the concept of defect severity, when the same problem repeatedly occurs by managing statistical information on a defect or an error, not only a notification or a defect processing routine is performed, but also the use and operation And the user (manager) is notified of a serious level of failure.

5 is a flowchart illustrating a method of responding to a fault based on log defect analysis according to an embodiment of the present invention.

A method for responding to a fault based on log defect analysis according to an embodiment of the present invention includes setting a collection policy, defect filter information and resource association information for an event or a log (S100) from a computing node, (S300) of determining whether or not a defect log is generated (S400); and a step of determining whether a defect log is generated The defect log analysis information is stored in the normalized form, the defect occurrence is notified, and the defect processing routine is performed (S500).

In step S100, a collection policy, which is a collection period for an event or a log stored in the policy and filter manager, and defect filter information provided from the system or user, stored in the policy and filter manager, and resource association information, .

In operation S200, the event log and the software log of the hardware resource are parsed and stored in a normalized format.

In operation S300, the defect information and the fault occurrence are analyzed from the collected log information based on the defect filter and the resource association information.

If it is determined in step S400 that a defect and a fault have occurred in the specific resource, step S500 is to notify the user (administrator) of the log defect analysis result through the defect log report 150, ), And transmits the result.

At this time, the step S500 notifies the designated user (manager) of the occurrence of the defect in the form of a mail or a message in accordance with the preset notification policy (cycle, repetition number, etc.).

FIG. 6 is a flowchart illustrating steps S500 according to an embodiment of the present invention, that is, a log-based defect analysis and a defect processing operation.

As a result of analyzing the log information to which the defect filter is applied, if the occurrence of the defect is confirmed (S510), it is checked whether a repetitive defect occurs (S520).

If it is determined that a repetitive defect has occurred, the defect severity is analyzed using the number of occurrences of defects, the period, and the defect log information (S530).

If the defect severity exceeds the threshold value, it is determined that the defect resource does not operate normally or is difficult to be serviced, and the use of the defective resource is interrupted (S550).

Subsequently, after the use of the defective resource is stopped, the user (administrator) is notified of the occurrence of the defect and the discontinuation of the resource use through the defect log report unit, and the related information is stored (S560).

If it is determined that the repetitive defect does not occur in step S520 and the defect severity is equal to or less than the threshold value in step S540, the occurrence of the defect is reported to the user in step S570.

In step S570, the step defect occurrence notification and the defect analysis information are stored. In step S580, the previously registered defect corresponding routine is called and performed.

Then, in step S590, it is determined whether a resource associated with the defective resource exists.

At this time, if there is no associated resource, the defect processing is completed. If there is an associated resource, the defect analysis step S510 and the defect handling step S580 are performed until all the related resources are completed.

According to the present invention, when a defect state of a resource is serious, the effect of isolating the resource by suspending the use of the resource, stopping the operation on the associated resource, and reducing the load on system resources and services have.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Log fault analysis based failure response system
110: Database 120: Resource Log Collector
130: Defect log analyzer 140: Resource defect processor
150: Defect Log Reporter 160: Policy & Filter Manager
200: node set 210: node
220: software resource 230: hardware resource
300: User Tools 310: Defect Log Viewer
320: Fault notification receiver

Claims (18)

A resource log collector for collecting event or log information from the configuration node;
A defect log analyzer that analyzes the log information and transmits defect related information when it is recognized that a system failure has occurred, and analyzes an associated resource of the defective resource and analyzes the defect severity;
A resource fault handler for resolving a fault condition for a resource when the fault log analyzer recognizes a fault;
A defect log reporting unit for reporting defect related information to a user; And
A policy and filter manager for setting and storing the event or log collection policy, the information about the defect analysis filter, the resource association information,
Wherein said fault diagnosis system comprises:
The method according to claim 1,
The resource log collector periodically collects the event or log information according to the log collection policy set from the policy and filter manager
Failure response system based on log defect analysis.
The method according to claim 1,
The resource log collector parses hardware event logs and software logs and stores them in a normalized format
Failure response system based on log defect analysis.
The method according to claim 1,
The defect log analyzer analyzes the log information by applying the defect analysis filter set from the policy and filter manager, extracts and classifies the collected information, and deduces the root cause of the failure
Failure response system based on log defect analysis.
The method according to claim 1,
The defect log analyzer uses the resource association information, which is association mapping information between resources, to check a defect for the associated resource
Failure response system based on log defect analysis.
6. The method of claim 5,
The resource defect processor calls a defect processing routine previously registered in the database to resolve the defect state for the resource and the associated resource to perform a normal operation
Failure response system based on log defect analysis.
The method according to claim 1,
The defect log reporter outputs the defect log information to the screen according to a user's request or transmits the defect related information through a mail or a message according to the predetermined failure notification policy for the notification period
Failure response system based on log defect analysis.
The method according to claim 1,
The defect log analyzer analyzes the defect severity using the number of occurrences of defects, the period and the defect log information
Failure response system based on log defect analysis.
9. The method of claim 8,
The resource defect processor may stop using the defect resource if the analyzed defect severity exceeds a threshold value
Failure response system based on log defect analysis.
10. The method of claim 9,
The defect log reporter notifies the user of the discontinuation of use of the corresponding defective resource in accordance with the defect severity
Failure response system based on log defect analysis.
The method according to claim 1,
The policy and filter manager stores information about the defect analysis filter including a defect-related filter and a user-defined filter that are basically provided in the system
Failure response system based on log defect analysis.
(a) setting a collection policy and defect filter information and resource association information for an event or a log;
(b) collecting event logs and software log information of hardware resources from the computing nodes;
(c) analyzing whether the defect is defective using the defect filter and the resource association information; And
(d) if it is determined that a defect has occurred as a result of the defectiveness analysis, storing the defect log analysis information in a normalized form, notifying occurrence of a defect, and performing a defect processing routine
Based on the log defect analysis.
13. The method of claim 12,
The step (a) may comprise setting the collection association policy, which is a collection cycle for the event or log, the defect filter information provided from the system or the user, and the resource association information, which is association mapping information between resources
Fault response based fault analysis.
13. The method of claim 12,
Wherein the step (b) includes parsing the event log and the software log of the hardware resource and storing the same in a normalized format
Fault response based fault analysis.
13. The method of claim 12,
In the step (c), the defect severity is analyzed using the number of occurrences of defects, the period and the defect log information
Fault response based fault analysis.
16. The method of claim 15,
The step (d) may include suspending the defective resource if the defect severity exceeds a threshold, and performing a notification of the suspension
Fault response based fault analysis.
13. The method of claim 12,
The step (c) may include analyzing whether a defect associated with the corresponding resource is defective using the resource-related information
Fault response based fault analysis.
13. The method of claim 12,
The step (d) includes notifying the occurrence of the defect according to a user's request, or notifying occurrence of the defect through a mail or a message in accordance with a predetermined notification cycle
Fault response based fault analysis.

Images