JP2009252006A - Log management system and method in computer system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method capable of integratedly managing logs collected and stored by a plurality of cell nodes due to a single fault in a computer system as one log. <P>SOLUTION: In the computer system with a plurality of cell nodes, when a fault is detected in a component in any cell node, the fault is notified to a management controller in the cell node, the detection of the fault is transferred to the other cell nodes in the computer system, each cell node stores faults detected in its own cell node and the detection of faults transferred from other nodes as local log data, the local log data is transferred to an integrated management controller in the computer system, a fault factor in the local log data to be transferred from each cell node is subsequently estimated on the basis of the transferred local log data, and the local log data is accumulated according to the estimation and stored as global log data into a memory. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an apparatus, a method, and a program for collecting and managing a failure log in a computer system composed of a plurality of cell nodes.

  The basic unit is an assembly called a cell node equipped with hardware components such as a microprocessor, a memory, an I / O device, and an interconnect controller, and a management controller that manages and controls these hardware components. There is a computer server with excellent expandability that can be integrated to form one computer system. In operation management of such a computer server, it is necessary to detect a failure occurring in any cell node and quickly identify the cause.

  In such a computer server, when a computer system is formed by a plurality of cell nodes, if a fatal failure occurs in the computer system, the management controllers in the cell nodes forming the computer system are independent of each other. Collects and holds the status (log) of hardware components mounted on its own cell node. However, there is a failure in which the cause of the failure cannot be identified even if the logs of a plurality of cell nodes collected and held independently of each other are analyzed alone. For example, there is a case where the computer system is stalled or the interface connecting each cell node is faulty.

  In this document, a computer system is defined as an aggregate that operates on one operating system (OS), and a computer server is defined as an aggregate of the entire cell nodes that are physically interconnected. That is, a plurality of computer systems can be formed in one computer server.

  One form of such a computer server is called a blade server. The relationship between an example of a blade server and a computer system is described in FIG.

  The blade server 300 has as its basic unit an assembly called blades 301-1 to 301-4 on which hardware components such as processors, memories, and I / O devices and management controllers that manage and control these hardware components are mounted. A computer server in which a plurality of blades are collected.

  In addition, Japanese Patent Laid-Open No. 2005-28452 (Patent Document 1), Japanese Patent Laid-Open No. 02-2749 (Patent Document 2) and Japanese Patent Laid-Open No. 11-143738 (Patent Document 3) include a plurality of processors or computers. Discloses a technique for centrally managing or simultaneously monitoring the failure information.

JP 2005-284520 A JP 02-27449 A Japanese Patent Application Laid-Open No. 11-143738

  However, since a blade server forms a computer system with a single blade and does not consolidate multiple blades into a single computer system, only a single blade log is collected if a fatal failure occurs in the computer system. It is not necessary to collect logs with multiple blades. In addition, the logs collected by each blade are collected when different failures occur, and these logs are not related.

  In addition, although the technique disclosed in the above-mentioned patent document indicates that the failure information in a plurality of processors is centrally managed, the function for classifying and managing the failure information according to the specific failure type is shown. Therefore, it is still difficult to quickly identify a problem location in a computer system composed of a plurality of cell nodes and take appropriate measures.

  The object of the present invention has been made in view of the above-mentioned problems. In such a computer server, logs collected and held in a plurality of cell nodes due to a single failure in the computer system are stored in one log. To provide a log management system and a log management method that can be managed in an integrated manner.

  A log management system according to the present invention is a log management system for managing log data when a failure occurs in a computer system having a plurality of cell nodes, and means for detecting a failure in a component in each cell node, When a failure is detected in a component in the cell node, a means for notifying the management controller in the cell node of the detection of the failure, and when a failure detection is notified to the management controller, the failure detection is performed by a computer. Means for transferring to other cell nodes in the system, and means for holding in each cell node a failure detected in the own cell node and a failure transferred from the other cell node in the local memory as local log data; Local log data held in the local memory of the cell node Means for transferring to the central management controller in the computer system, and the central management controller estimates the cause of failure in the local log data transferred from each cell node based on the local log data transferred from each cell node. And means for aggregating the local log data according to the estimation of the failure factor and holding the result in the memory as global log data.

  A log management method according to the present invention is a log management method for managing log data when a failure occurs in a computer system having a plurality of cell nodes, the step of detecting a failure in a component in each cell node, When a failure is detected in a component in the cell node, the step of notifying the management controller in the cell node of the detection of the failure, and the detection of the failure when the detection of the failure is notified to the management controller A step of transferring to other cell nodes in the system, and a step of holding in each cell node a failure detected in the own cell node and a failure transferred from the other cell node in the local memory as local log data; Local log data stored in the local memory of the cell node Of transferring the data to the central management controller in the computer system, and based on the local log data transferred from each cell node in the central management controller, the cause of failure in the local log data transferred from each cell node thereafter And a step of aggregating local log data according to the estimation of the failure factor and holding it in the memory as global log data.

  According to the present invention, in a computer system composed of a plurality of cell nodes, a failure cause can be identified with high accuracy even for a failure that could not be identified until now.

  The reason is that when a failure occurs in any of the cell nodes, the local log data recorded and held in each cell node is transferred to the central management controller, where the log data at the same failure level is a failure based on one factor. This is because it is presumed that there is a record and is kept in that state, and as a result, it is very easy to identify the cause of the failure in the maintenance management of the computer system.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

(Configuration of the embodiment)
Referring to FIG. 1, an example of a computer server according to an embodiment of the present invention, an example of a computer system, and their correlation are shown.

  The computer server 100 is composed of four cell nodes 101-1 to 101-4. These four cell nodes 101-n perform data transmission / reception between processors, I / O devices, and between processors and I / O devices across cell nodes via a transmission line 150. In addition, when communication between management controllers across cell nodes is performed, it is performed via the transmission line 151.

  The computer server 100 is divided into a computer system A and a computer system B. The computer system A is formed from three cell nodes 101-1, 101-2, and 101-3, and the computer system B is formed from one cell node 101-4. Since data transmission / reception between processors, I / O devices, and between processors and I / O devices across the computer system is not performed, the cell node 101-4 does not perform data transmission / reception with other cell nodes via the transmission line 150. .

  Since the computer system A is formed of three cell nodes 101-1, 101-2, and 101-3, it is necessary to determine one management controller that performs overall management of the entire computer system A. In this embodiment, the management controller 111-1 in the cell node 101-1 is used as the overall management controller of the computer system A.

  Referring to FIG. 2, there is shown an embodiment of a cell node which is a main part of a computer server according to the present invention.

  The processor 200-n, the memory 201-n, and the I / O device 202-n are main components of the computer. When a failure is detected in these components, a failure detection notification is issued to the management controller 111 via the transmission path 260-n. This failure detection notification includes a failure level. The failure level is the degree of influence that the failure has on the computer system. An example of the fault level classification is described below.

[Fatal failure] A failure that causes the computer system to go down. Since the failure level affects the entire computer system, it is necessary to collect logs of the entire computer system.
[Warning failure] A failure level at which some hardware components are in an abnormal state, but the computer system can continue to operate. It is necessary to collect logs only for failure detection cell nodes.
[Correctable fault] A correctable fault level such as a memory 1-bit error. It is necessary to collect logs only for failure detection cell nodes.

  The interconnect controller 110 controls data transmission / reception between the processors 200-n, the I / O devices 202-n and between the processors 200-n and the I / O devices 202-n in the same cell node, Controls data transmission / reception between cell nodes. When a failure is detected in the interconnect controller 110, a failure detection notification is issued to the management controller via the transmission path 260-5. This failure detection notification includes a failure level.

  The management controller 111 is a main part for realizing the present invention. The management controller 111 manages and controls hardware components in the own cell node. Therefore, the hardware components in the cell node such as the processor 200-n, the memory 201-n, the I / O device 202-n, and the interconnect controller 110 are connected by the transmission line 260-n.

  Further, it is connected to a non-volatile memory 204 that holds a log in the own cell node and an environment monitoring device 203 that monitors the environment (temperature, power supply, etc.) of the own cell node. Further, in order to centrally manage the computer system, it is connected to a management controller in another cell node through a transmission line 151.

  When the management controller 111 receives the failure detection notification via the transmission line 260-n from the hardware components in the cell node such as the processor 200-n, the memory 201-n, the I / O device 202-n, and the interconnect controller 110, If the failure level of the failure detection notification is a fatal failure, the failure detection notification is transferred via the transmission path 151 to all the cell node management controllers in the same computer system.

  When the management controller 111 receives a failure detection notification from the hardware component in the own cell node or from another cell node, the management controller 111 collects the log of the hardware component in the own cell node and holds it in the nonvolatile memory 204. At this time, the local log ID for identifying the log, the log collection time, and the failure level embedded in the received failure detection notification are also stored in the nonvolatile memory as a set. An example of a log structure held in the nonvolatile memory 204 is shown in FIG. Details of this will be described later.

  Furthermore, the management controller 111 holds a log of hardware parts in its own cell node in the nonvolatile memory 204, and then issues a log collection notification to the overall management controller 111 with a local log ID, a log collection time, and a failure level. To do.

  When the management controller 111, which is the overall management controller, receives a log collection notification with a failure level of fatal failure from a certain management controller 111 in the same computer system, the management controller 111 in the same computer system thereafter A log collection notification whose failure level is a fatal failure received from the management controller 111 is determined to be a log caused by the same failure, and a plurality of log collection notifications notified from these different management controllers are managed as one group.

  An embodiment of the management method is shown in FIG. The management controller 111 serving as the overall management controller has a log management table capable of centrally managing hardware component logs collected and held in its own computer system as shown in FIG.

  The global log ID = 3 shown in FIG. 4 indicates that the log collected and retained due to the coincidence failure is the local log ID = 2 in the nonvolatile memory 204 in the cell node 101-1, the cell node 101-2. It shows that the local log ID = 1 in the internal nonvolatile memory 204 and the local log ID = 0 in the nonvolatile memory 204 in the cell node 101-3.

(Operation of the embodiment)
Hereinafter, the operation of the present embodiment will be described with reference to FIGS.

  FIG. 5 is a processing flowchart from the occurrence of a fatal failure to the management of logs collected and held in the distributed log management method of the present invention, and FIGS. 6 and 7 show the operation. In this embodiment, an operation when a fatal fault is detected in the memory 201-1 in the cell node 101-3 will be described.

  When the memory 201-1 in the cell node 101-3 detects a fatal failure (step S 501 in FIG. 5), the failure level is assumed to be a fatal failure and the failure is transmitted to the management controller 111-3 in the own cell node via the transmission line 260-1. A detection notification is issued (step S502, 1 in FIG. 7).

  When the management controller 111-3 in the cell node 101-3 receives a failure notification whose failure level is a fatal failure, the management controller 111-3 sends the failure detection notification to the management controllers 111-1 and 111-2 of other cell nodes in the same computer system. Transfer (step S503, 2 in FIG. 7). When the failure level is a warning failure or a correctable failure, the failure notification is not transferred to the management controller of another cell node. This is because, when the failure level is a warning failure or a correctable failure, the failure location can be sufficiently identified only by the log in the failure detection cell node.

  Upon receiving the failure detection notification, the management controller 111-3 in the cell node 101-3 collects hardware component logs in the own cell node and holds them in the nonvolatile memory 204 under the self-management controller (step S504-1). 3 in FIG.

  In parallel with this, the management controllers 111-1 and 111-2 in all the other cell nodes in the computer system A similarly collect the log of the hardware parts in the own cell node when receiving the failure detection notification. And stored in the non-volatile memory 204 under its own management controller (step S504-2, 3 in FIG. 6). At this time, each management controller 111-n also holds the local cell ID unique in the own cell node, the log collection time, and the failure level (fatal failure) embedded in the failure detection notification.

  FIG. 3 shows an example of information held in the nonvolatile memory 204 of each cell node at this time. In the cell node 101-1, since two logs have been held in the local log ID = 0 and 1 so far, they are collected and held due to the fatal failure detected by the memory 201-1 in the cell node 101-3. The log is held at local log ID = 2.

  In the cell node 101-2, since one log has been held at the local log ID = 0 so far, logs collected and held due to a fatal failure detected by the memory 201-1 in the cell node 101-3 Is held at local log ID = 1. In the cell node 101-3, since there is no log held so far, the log collected and held due to the fatal failure detected by the memory 201-1 in the cell node 101-3 is set to local log ID = 0. Retained.

  The management controller 111-3 in the cell node 101-3, when the log collection / retention in the own cell node is completed, adds the local log ID, the log collection time, and the failure level to the management controller 111-1 as the overall management controller. A log collection notification is issued via the transmission path 151 (step S505-1, 4 in FIG. 6).

  In parallel with this, the management controllers 111-1 and 111-2 in all other cell nodes in the computer system A in the computer system A also complete the log collection / retention in their own cell nodes. Then, a log collection notification is issued via the transmission path 151 with the local log ID, the log collection time, and the failure level to the management controller 111-1 as the overall management controller (step S505-2, 4 in FIG. 6). .

  That is, the management controller 111-1 of the cell node 101-1 reports local log ID = 2, the management controller 111-2 of the cell node 101-2 reports local log ID = 1, and the management controller 111 of the cell node 101-3. -3 reports local log ID = 0.

  When the management controller 111-1, which is the overall management controller, receives a log collection notification whose failure level is fatal from any cell node in the computer system A, it starts a timer. This timer times out after a certain time. The overall management controller collects and holds a plurality of log collection notifications whose failure level is fatal failure received before the timer times out due to the fatal failure detected by the memory 201-1 in the cell node 101-3. It is determined that the log has been recorded (step S506, 5 in FIG. 6).

  The management controller 111-1, which is the overall management controller, holds the log management table shown in FIG. 4, and stores the result determined in step S506 (step S507, 6 in FIG. 6). 4 indicates that the log with the global log ID = 3 is a log collected and held due to a fatal failure detected by the memory 201-1 in the cell node 101-3.

  FIG. 8 shows a correspondence method between the log management table held by the overall management controller and the log held in the nonvolatile memory 204 by each cell node. The global log ID = 0 corresponds to the local log ID = 0 held by the cell node 101-2, the global log ID = 1 corresponds to the local log ID = 0 held by the cell node 101-1, and the global log ID = 2 corresponds to the local log ID = 1 held by the cell node 101-1, and the global log ID = 3 is the local ID = 2 held by the cell node 101-1, and the local log ID = 1 held by the cell node 101-2. And the local log ID = 0 held by the cell node 101-3. In other words, a new global ID is added to the log management table held by the overall management controller every time a failure is detected, and it is possible to determine where and where logs collected and held due to the failure detection exist.

  As described above, a series of processes from occurrence of a failure in any cell node to log registration is completed.

(Effect of embodiment)
As described above, according to the present embodiment, logs collected and held in a plurality of cell nodes due to a single failure in the computer system can be managed as one log. As a result, it becomes possible to analyze the logs across the logs, and as a result, it is possible to identify the cause of the failure even for the failure that could not be identified until now.

  The management controller of each cell node according to the embodiment of the present invention is not only realized in hardware, but also stores a log management program providing its function in an auxiliary storage unit such as a hard disk device or ROM, and It can also be realized in software by executing the program with a processor.

  Although the present invention has been described with reference to the preferred embodiments, the present invention is not necessarily limited to the above embodiments, and various modifications can be made within the scope of the technical idea. .

  For example, in the above embodiment, the overall management controller is used as a management controller in one selected cell node, but this may be realized on a management controller outside the cell node.

  Further, the interconnect controller may have the function of the management controller. Since data transmission between interconnect controllers is generally faster than data transmission between management controllers, this example is preferable in order to reduce the difference in log data transfer time between cell nodes.

It is a block diagram which shows the structure of the log management system in the computer system by one embodiment of this invention. It is a block diagram which shows the structure of the cell node in embodiment of FIG. It is a figure which shows an example of the information hold | maintained at the non-volatile memory of each cell node. It is a figure which shows an example of the log management table which centrally manages the hardware component log collected and hold | maintained with the non-volatile memory in the cell node which has an integrated management controller. It is a figure which shows the processing flowchart until it manages the log collected and hold | maintained from fatal failure generation in the log management method by one embodiment of this invention. It is a block diagram which shows the structure of the log management system by one embodiment of this invention which added the process of FIG. It is a block diagram which shows the structure of the cell node which added the process of the process in FIG. It is a figure which shows the correspondence method of the log management table which a general management controller hold | maintains, and the log which each cell node hold | maintains at non-volatile memory. It is a block diagram which shows the structure of the blade server by related technology.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100: Computer server 101-n: Cell node 110-n: Interconnect controller 111-n: Management controller 111-1: General management controller 150: Transmission path 151: Transmission path 200-n: Processor 201-n: Memory 202-n : I / O device 203: Environmental monitoring device 204: Non-volatile memory 260-n: Transmission path

Claims (16)

A log management system for managing log data when a failure occurs in a computer system having a plurality of cell nodes,
Means for detecting a failure in a component in each cell node;
Means for notifying the detection of the failure to the management controller in the cell node when a failure is detected in a component in any of the cell nodes;
Means for transferring the failure detection to another cell node in the computer system when the failure detection is notified to the management controller;
In each cell node, means for holding in a local memory as a local log data detection of a failure detected in the own cell node and a failure transferred from the other cell node;
Means for transferring local log data held in a local memory of each cell node to a general management controller in a computer system;
The overall management controller is
Means for estimating the cause of failure in the local log data transferred from each cell node based on the local log data transferred from each cell node;
Means for aggregating the local log data according to the estimation of the failure factor and holding it in memory as global log data;
A log management system in a computer system, comprising: Means for notifying the management controller in the cell node of the detection of the failure,
2. The log management system according to claim 1, wherein the management controller is notified of information for identifying a component in which a failure has occurred and information for identifying the level of the failure. Information identifying the level of failure is
3. The log management system according to claim 2, wherein at least the failure includes information indicating a type of whether or not the failure is a fatal failure affecting the entire computer system. Means for forwarding the failure detection to another cell node in the computer system;
The detection of the failure is transferred to another cell node in the computer system only when the information specifying the level of the failure indicates a fatal failure. The log management system according to claim 3. Means for holding the failure in the local memory as local log data,
5. The log management system according to claim 1, wherein the log management system retains information specifying a unique log ID, a log data collection time, and a failure level for each failure. Means for estimating the cause of failure in local log data transferred from each cell node of the overall management controller,
When the failure level in the first local log data is a fatal failure, the local log data at the fatal failure level transferred within a predetermined time from the transfer is caused by the same factor as the first failure. The log management system according to any one of claims 1 to 5, wherein the log management system is estimated.   The log management system according to claim 1, wherein the management controller is an interconnect controller in each cell node.   8. The log management system according to claim 1, wherein one of the selected management controllers in any of the cell nodes is used as the overall management controller. 9. A log management method for managing log data when a failure occurs in a computer system having a plurality of cell nodes,
Detecting a failure in a component in each cell node;
A step of notifying the detection of the failure to a management controller in the cell node when a failure is detected in a component in any of the cell nodes;
Transferring the failure detection to another cell node in the computer system when the failure detection is notified to the management controller;
In each cell node, holding a failure detected in its own cell node and a failure transferred from the other cell node in local memory as local log data; and
Transferring local log data held in a local memory of each cell node to a general management controller in a computer system;
In the overall management controller,
Based on the local log data transferred from each cell node, and thereafter estimating the cause of failure in the local log data transferred from each cell node;
A log management method in a computer system, comprising the step of aggregating the local log data in accordance with the failure factor estimation and holding it in a memory as global log data. Informing the management controller in the cell node of the detection of the failure,
The log management method according to claim 9, wherein the management controller is notified of information for identifying a failed component and information for identifying the level of the failure. Information identifying the level of failure is
The log management method according to claim 10, comprising at least information indicating whether the failure is a fatal failure affecting the entire computer system. Transferring the detection of the failure to another cell node in the computer system;
The detection of the failure is transferred to another cell node in the computer system only when the information specifying the level of the failure indicates a fatal failure. The log management method according to claim 11. Holding the failure as local log data in local memory;
The log management method according to any one of claims 9 to 12, wherein information specifying a unique log ID, a log data collection time, and a failure level is held for each failure. In the step of estimating the cause of failure in local log data transferred from each cell node of the overall management controller,
When the failure level in the first local log data is a fatal failure, the local log data at the fatal failure level transferred within a predetermined time from the transfer is caused by the same factor as the first failure. The log management method according to any one of claims 9 to 13, wherein the log management method is estimated.   The log management method according to any one of claims 9 to 14, wherein the management controller is an interconnect controller in each cell node.   The log management method according to claim 9, wherein one of the selected management controllers in the cell node is the overall management controller.

Images