JP2000305914A - Cluster system, specification assisting device of system fault factor and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To promptly and exactly specify a system component to be a fault generating factor without troubling a maintenance staff. SOLUTION: A state value including a state value number recorded in recording means 4a, 4b,... of all nodes 3a, 3b,... to constitute a cruster system is fetched by a state value reading and storage means 21 and stored in a state value storage part 12. In addition, the system is preliminarily provided with a defining means (13, factor attribute definition table, relational attribute table between factor and state value) to define certainty of generation of a fault by every system component and certainty of the fault to be reflected on the state value by every system component. And an abnormal value is found from among the state values to be stored in the state value reading and storage means 21 and appropriateness of fault of the system component is calculated by using the state value as the abnormal value, the certainty of generation of the fault and the certainty of the fault to be reflected on the state value by a fault possibility processing means 22.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a cluster system in which a plurality of computers (hereinafter, referred to as nodes) execute predetermined processing of a program in cooperation with each other via a network or the like, a system failure factor identification support device, and a recording medium.

[0002]

2. Description of the Related Art Conventionally, as shown in FIG. 8, a plurality of nodes 52 ₁ , 52 ₂ ,... Are connected on, for example, a LAN 51 having duplicated transmission lines L 1 and L ₂ , and one node 52 ₁ executes a program. when a failure occurs during, it has been made _{that the 2} next node 52 executes a predetermined process takes over the execution of the program of the same function.

In such a cluster system, when analyzing the cause of the occurrence of a failure, the maintenance staff needs to operate the nodes 52 ₁ , 5 2
Read the error / execution log (log: history state), which is the program execution state, from the storage devices 53 ₁ , 53 ₂ ,... Attached to 22 ₂ ,. Is estimated.

[0004]

However, in the above-described cluster system, not only the configuration and operation are more complicated than a system including a single node, but also
Since one business program is executed while moving on a plurality of nodes, it is often difficult for maintenance personnel to manually analyze the cause of the failure.

The present invention has been made in view of the above circumstances, and records a detectable state value of each node together with a predetermined state value number, so that it is possible to easily specify a system component which causes a failure. To provide a cluster system.

It is another object of the present invention to provide a system failure factor identification support device for quickly and accurately identifying a system component that causes a failure without the need for maintenance personnel.

It is a further object of the present invention to provide a computer-readable recording medium in which a failure factor identification processing program for quickly and accurately identifying a system component causing a failure is recorded.

[0008]

In order to solve the above-mentioned problems, the present invention provides a cluster system in which a plurality of nodes execute required processing while cooperating with each other via a transmission line. A recording means is provided for recording a time, a predetermined state value number, and a state value each time a possible state value is abnormal or every certain time has elapsed.

According to the present invention, by taking the above measures, each node records a time, a predetermined state value number and a state value each time the state value is abnormal or every time a certain time elapses. Based on the status value number of the status value determined to be abnormal, it is possible to specify the system component causing the failure.

[0010] Further, another invention is characterized in that all nodes constituting the cluster system change the time, a predetermined state value number and a state value each time the detectable state value of each node is abnormal or every certain time has elapsed. In the case where a recording means for recording is provided, a state value reading / storage means for taking in and storing a state value including a state value number from a recording means attached to each node, and a reliable occurrence of a failure for each system component. Means (real number from 0 to 1) and certainty (real number from 0 to 1) to be reflected in the state value of a fault for each system component, and a state stored in the state value reading storage means An abnormal value is found from the values, and the fault validity of the system component is calculated using the status value that is the abnormal value, the certainty of the occurrence of the fault, and the certainty reflected in the status value of the fault. And a validity output means for displaying the validity of each system component calculated by the failure possibility processing means on these components and enabling the identification of the cause of the failure. This is a system failure factor identification support device.

According to the present invention, by taking the above means, the state value including the state value number is read from the recording means attached to each node by the state value reading / storing means, and the state values are processed and stored in chronological order. After that, the failure possibility processing means finds a state value regarded as abnormal from the state values, and using both this state value and the certainty of the system component related to the state value defined by the definition means, By calculating the fault validity of the system component, when the validity is set to "1", the multiplicity of the two certainty values is subtracted from this "1" to specify the system component having the highest fault validity. It is.

Further, as another invention, if the configuration of the system failure factor identification support device is applied to at least one node constituting the cluster system as it is, the cluster system itself can identify the system component causing the failure. .

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Prior to the description of embodiments of the present invention, basic matters in the case of mechanically analyzing a failure factor of a cluster system will be described.

When mechanically analyzing the cause of a failure in the cluster system, it is necessary to accurately grasp the detectable state value of each node constituting the cluster system. This detectable status value includes, for example, periodic communication between each node that is performed periodically via a network, the execution status of a program at a certain node, and the like. It is necessary to analyze from the operation state across the system components. For example, when an abnormality appears in a certain state value, it means that an abnormality appears in at least one of the components related thereto, and conversely, when the state value of a certain node is normal, the related system component It means that everything is likely to be normal.

In general, considering a redundant configuration and a handover of a program which are features of a cluster system,
The various states of each system component will appear in the combined detected state values. Therefore, the states of these components are set as simultaneous equations, and a system component that becomes a failure factor is specified from among the system components.

FIG. 1 is a configuration diagram showing one embodiment of a cluster system according to the present invention.

This system comprises a plurality of nodes 3a each having a cluster control program 2a, 2b, 2c,... Which takes in and processes a detectable state value of a node on, for example, LAN1 composed of duplicated transmission lines L1 and L2. , 3
, 3c,... are connected, and these nodes 3a, 3b,.
Recording devices 4 for recording data such as status values are stored in 3c,.
a, 4b, 4c,... are connected.

Each of the nodes 3a, 3b, 3c,.
A time generation source (not shown) for taking in the state value abnormality detection time when the system component is abnormal is provided, and the recording devices 4a, 4b, 4c,. The corresponding state value number has been set.

Here, the detectable state value of each node is a simple value as much as possible. For example, "normal" or "abnormal"
, That is, “1” or “0”. Also, in the program execution state in each node, only the state of the program that has been executed or has failed in execution is recorded as “1” or “0”, for example.

The number of types of the detectable state values includes the following types when the heartbeat between nodes and the state of the program at the time of executing the node are considered separately.

In the case of an inter-node heartbeat, from a certain node, whether or not the monitored node is operating is determined by the number of nodes × the number of communication paths × the number of nodes (1). However, if the transmission line is not duplicated, for example, there are the number of nodes × the number of communication paths.

* In the case of the state of the program running on the node, there are as many as the number of nodes × the number of programs (2).

Therefore, in order to organize the state values as described above, the state value numbers are assigned by the number of types of the state values.

In the above (1), the square equation of the number of nodes is used. For example, in the case of a duplex transmission line as shown in FIG. 2, two inquiry communication paths R1 (solid line) and R2 (dotted line) , The inquiring node sends different communication paths R1, R
This is because it is necessary to make an inquiry through 2. By executing such an inquiry, there is a response from the inquiry destination node to the inquiry via one communication route R1,
For example, if there is no response from the inquiry destination node to the inquiry via the other communication path R2, at least the communication path R2
Can be detected as an abnormal state value.

Next, the operation of the above system will be described with reference to FIGS. 3 (a) and 3 (b).

When each node 3 (3a, 3b,...) Receives an interrupt or other various factors during the original data processing of the node and the operation of the cluster control program 2 (2a, 2b,. The processing as shown in FIG.

That is, each node 3 (3a, 3b,...)
Performs initialization processing such as unnecessary data clear processing and other necessary data setting (S1), and then determines whether or not there is an abnormality in the state value (S2). When there is an abnormal value among the detectable state values of each node, it is determined that there is an abnormality. Here, the abnormal value may be, for example, communication failure or other data transfer failure, power supply down, OS down, cluster control program inoperable, etc., but other various failures may be considered. .

When the status value takes an abnormal value as described above, the time and the status value number at that time are fetched and recorded in the recording device 4 (4a, 4b,...) Together with the status value (S).
3). At this time, if there are a plurality of status value types related to the abnormality, the status values are stored for each type using the respective status value numbers.

The above series of processing is repeatedly executed until a detection end instruction is given (S4).

As a result, as shown in FIG. 4, every time an abnormal value occurs, the recording device 4 sequentially records the time, the status value number, and the status value record in time series.

FIG. 3A shows a process of recording a status value when the status value takes an abnormal value. However, the status value may be recorded every time a predetermined time elapses.

The recording of the state value at regular intervals is shown in FIG.
As shown in (b), after each node 3 (3a, 3b,...) Performs an initialization process (S11), it is determined whether a certain time Δt has elapsed (S12). When the predetermined time Δt has elapsed, the time, the state value number, and the state value at that time are sequentially stored in the recording device 4 (S13). Then, a series of processing as described above is repeatedly executed until there is a detection end instruction (S14).

Therefore, according to the above-described embodiment, when the detectable state value of each node 3 takes an abnormal value, or every time a certain time elapses, the state value is changed for each state value number. Since such a value is recorded, it is possible to identify the system component causing the failure based on the status value number of the status value regarded as abnormal.

FIG. 5 is a block diagram showing an embodiment of a system failure factor identification support apparatus according to the present invention.
Note that this apparatus is an independent support system is a cluster system shown in FIG. 1, may be imparted the functions shown in FIG. 5 to a particular node for example 3 ₁ shown in FIG. 1 for example, or to each node The function shown in FIG. 5 may be provided. In this case, a cluster system having a failure factor identification support function can be realized.

The system failure factor identification support device includes the recording devices 4a, 4b, 4 of the nodes 3a, 3b, 3c,.
a status value storage unit 12 for reading and storing status values and the like stored in c,... through the interface 11, a definition table 13 for defining element attributes and element-state value relationship attributes in advance, and a failure factor identification processing program , A failure factor identification processing unit 15 which is configured by a CPU, reads a failure factor identification processing program recorded on the recording medium 14 and executes a predetermined process, and before, during, and after the process. A data buffer 16 for storing result data, an input device 17, and a display unit 18 are provided.

Although the state value storage section 12, the definition table 13, and the data buffer 16 are provided separately, they may be used in the same recording medium by dividing the area.

As shown in FIG. 6 (a), the time and state recorded in each of the recording devices 4a, 4b, 4c,... Attached to the nodes 3a, 3b, 3c,. Among the value numbers and the state values, the state value numbers and the state values are stored in a time-series table. Hereinafter, the storage table of the state value storage unit 12 is referred to as a state value table T.
Called r.

The definition table 13 is an element attribute definition table Ta for defining element attributes as shown in FIG.
And an element-state value relation attribute definition table Tr that defines the attribute of the element-state value relation as shown in FIG. 6C.

This element attribute definition table Ta is a table for setting in advance the certainty of failure occurrence of system components. Specifically, a system component (system component number) that may cause a failure is used as an attribute, and the degree of whether the failure of the system component appears as a symptom is continuous or intermittent. That is, the degree of certainty of the symptom continuity (repetition) for each system component
It is represented by a real number of １1, and the certainty of the onset of the so-called failure over time is defined as an attribute.

The system components are components which may cause a failure among all components constituting the system. For example, each node, each transmission line (communication path), program, OS, A power source, resources used by the program, and the like are listed, and these system components are respectively assigned system component numbers. Regardless of which node executes a program having the same function, the contents of the program are the same. Therefore, the programs are regarded as logically the same system components, and the same element numbers are used.

On the other hand, the element-state value relationship attribute definition table Tr is a table for setting in advance the certainty that a failure of a system component is reflected in the state value. Status value number,
It is an array of records consisting of a set of system component numbers and certainty. The fields of this record are Tr [i]. s, Tr [i]. c, Tr [i]. w
Shall be referred to. s indicates a state value number, c indicates a system component number, and w indicates a certainty.

More specifically, the element-state value relation attribute definition table Tr has, as attributes, a state value number and a system element number which is a system element which may cause a failure. It is a certainty that the failure of the element is actually reflected in the state value as a real number of 0 to 1, and the so-called failure occurrence in the space of the failure is defined as an attribute.

The recording medium 14 stores a failure factor identification processing program to be executed by the failure factor identification processing unit 15 as shown in FIG. As the recording medium 14, a CD-ROM or a magnetic disk is generally used, but other than that, for example, a magnetic tape, DVD-ROM, floppy (registered trademark) disk, MO, CD-R, memory card Or the like may be used.

The failure factor identification processing section 15 is provided with a state value reading and storing means 21, a failure possibility processing means 22, and a validity output means 23.

The state value reading and storing means 21 includes recording devices 4a, 4a,
has a function of reading the data recorded in b, 4c,... and sequentially storing the data in the data buffer 16, performing a time-series rearrangement process after storing the data, and storing the data in the state value storage unit 12.

The fault possibility processing means 22 reflects the status value stored in the status value storage unit 12, the reliability of occurrence of a fault of each system component set in the definition table 13 and the status value in advance. It has a function of expressing the possibility of a failure of each system component, that is, the validity of the failure, by numerical values using certainty and the like.

The validity output means 23 has a function of displaying the validity of the failure of each system component on the display unit 18 to notify maintenance personnel.

Next, a description will be given of a process of identifying a system component which causes a failure in the apparatus configured as described above.

Note that the process of specifying the system component which causes the obstacle is an example of processing using the deduction method. In this example, in the initial state, the possibility of failure (failure validity) of all system components is set to “1”, and when it is assumed that a certain system component has a failure, an abnormality may be detected. Regarding the state value, a method is adopted that reduces the possibility of an assumed failure unless an abnormality is actually detected.

Hereinafter, a specific operation will be described with reference to FIG.

When the operation of the apparatus is started, an initialization process is performed (S21), and then a failure factor identification processing program recorded on the recording medium 14 is read and stored in, for example, the data buffer 16. Here, the input device 1 is
When a time zone to be analyzed is input from step 7, the analysis time zone is set in the data buffer 16 and other storage means (S22; analysis time zone setting function).

Thereafter, i = 1 is set as the first node among the plurality of nodes (S23), and the time and state values within the time zone are obtained from the recording device 4a attached to the node 3a corresponding to i = 1. The number and the state value are sequentially read and stored in the data buffer 16.

When the reading of the status value of the recording device 4a is completed, the time, the status value number and the status value within the time zone are sequentially read from the storage device 4b attached to the next node 3b, and the data buffer 16 is read. To be stored. Such processing is performed for all nodes. Thereafter, the data of all the nodes stored in the data buffer 16 are arranged in chronological order from the time record, the time record is removed, and the record composed of the set of the state value number and the state value is sequentially arranged. (S24 to S28; status value reading storage function). The fields of this state value record are Tf [i]. s, Tf [i]. v.

Next, for example, the validity table Tw formed in the definition table 13 or the data buffer 16 or the like.
, "1" is set as the fault validity of all system component numbers. That is, in the initial state, the failure validity of all system components is set to “1” (S29; initial failure possibility setting function). The validity table Tw indicates the validity of the assumption that each system component has a fault, and is an array of real numbers of fault validity for each system component as described later.

After setting the validity of all the system component numbers in the validity table Tw to “1” as described above, whether the status values stored in the status value storage unit 12 are successively abnormal or not. (S3)
0, S31; status value abnormality presence / absence determination function).

If the i-th record is “Tf [i].
If “v == abnormal”, the state value numbers of all records in the element-state value relation attribute table Tr are searched to determine whether or not there is a corresponding state value number (S32, S32).
33; applicable state value number search function). Now, for all records in the element-state value relationship attribute table Tr, suppose that the j-th record is “Tr [j] .s == Tf
[I]. v ", then c = Tr [j]. c,
The fault validity of the corresponding system component is calculated (S3
4: validity calculation function). The failure validity of this system component is Tw [c] = Tw [c] * ｛1- (Ta [c] * Tr
[J]. It is calculated from w｝.

If the state value abnormality search is not completed, the process returns to step S31, and the same processing is repeatedly executed for the next state value abnormality (S31 to S36).

When the validity of the system component is calculated for each abnormal state value and stored in the validity table Tw as described above, the validity of the system component (element number) is read from the validity table. Is displayed on the display unit 18 (S37; validity output function). That is, it can be considered that the system component having the highest validity value among the system component numbers in the validity table Tw has a high possibility of having a failure.

Therefore, according to the above-described embodiment, the status values are sequentially read out together with the status value numbers from the recording devices attached to the respective nodes constituting the cluster system. Element-state value relation attribute table Tr based on abnormal state values
The corresponding system component is found from the record of the status value number, the system component number, and the reliability reflected in the failure status value of the system component, and the reliability of the system component is set in advance in the element attribute table Ta. Failure validity is calculated numerically using the certainty of the occurrence of failure of the corresponding system component, and the system component with a high possibility of failure is specified. System components that can be analyzed and whose validity value is large can be identified as possibly having a failure, and the validity value of the system component related to the system component that is considered to have a possible failure is also calculated at the same time. Therefore, the influence of each system component can be easily grasped.

[0060]

As described above, according to the present invention, each node sets a time, a predetermined state value number and a predetermined state value every time a detectable state value of each node is abnormal or every predetermined time. By recording the value, it is possible to provide a cluster system that can easily specify a system component causing a failure.

Another aspect of the invention can provide a system failure factor identification support device that can quickly and accurately identify a system component causing a failure without the need for maintenance personnel.

Still another aspect of the present invention can provide a computer-readable recording medium that stores a failure factor identification processing program for quickly and accurately identifying a system component that causes a failure.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing an embodiment of a cluster system according to the present invention.

FIG. 2 is a diagram illustrating the number of state value types for assigning state value numbers.

FIG. 3 is a flowchart illustrating the operation of the cluster system according to the present invention.

FIG. 4 is a diagram showing a data recording state of a recording device attached to each node constituting the cluster system.

FIG. 5 is a configuration diagram showing an embodiment of a system failure factor identification support device according to the present invention.

FIG. 6 is a view showing a state of table data used in the apparatus of FIG. 5;

FIG. 7 is a flowchart for explaining the operation of the apparatus shown in FIG. 5;

FIG. 8 is a configuration diagram illustrating a conventional cluster system.

[Description of References] 3a, 3b,..., Nodes 4a, 4b,... Recording device 12,... Status value storage unit 13,... Definition table 14,. Processing means 23 ... validity output means

Claims (7)

[Claims] In a cluster system in which a plurality of nodes execute required processing while cooperating with each other on a transmission line, each of the nodes is configured to detect when a detectable state value of each of the nodes is abnormal or every predetermined time. A cluster system comprising recording means for recording a time, a predetermined state value number, and a state value. 2. A state value reading and storing means for fetching and storing a state value including a state value number from said recording means of all nodes constituting the cluster system according to claim 1, and a fault for each system component. Defining means for defining the certainty of the onset and the certainty reflected in the status value of the fault for each system component; finding an abnormal value from the status values stored in the status value reading storage means; Fault possibility processing means for calculating the fault validity of a system component by using a status value that is a value, the certainty of the occurrence of the fault, and the certainty reflected in the status value of the fault. System failure factor identification support device to be used. 3. The system failure factor identification support device according to claim 2, wherein the definition means for defining the certainty of the occurrence of the failure of each system component has each of the system components as an attribute. A system failure factor identification support device, which defines a degree of continuity of a symptom when an element fails. 4. The system failure factor identification support device according to claim 2, wherein the definition means for defining the reliability reflected in the failure status value of each of the system components includes the system components and the respective statuses. A system failure factor identification support device, wherein a relationship with a value is defined as an attribute, and the failure of each system component is defined with certainty reflected in the state value. 5. The system failure factor identification support device according to claim 2, wherein the failure possibility processing means sets the failure validity of each of the system components to 1, and defines each system configuration defined by the definition means. The fault validity of each of the system components is determined by using a deduction method in which the two certainties of the elements are real numbers of 0 to 1 and the multiplied value of the two certainties is subtracted from the fault validity of a certain system component. A system failure factor identification support device characterized by calculating. 6. The system failure factor identification support device according to claim 2, wherein the failure possibility processing means relates to the status value based on a status value which is an abnormal value found from each status value. A system failure factor identification support device, wherein a failure validity of each system component is calculated using the above-mentioned two certainties of each system component. 7. A first definition table defining a degree of continuity of a symptom when a failure occurs in a system component, a predetermined state value number, and a system component as attributes. And a second definition table for defining the reliability reflected in the status value for each status value number, and a failure factor identification processing program for identifying a failure factor of the system component is recorded on the recording medium. An analysis time zone setting function for setting the time zone for analyzing the cause of the failure in the computer for processing the cause of the failure, and when the detectable state value is abnormal or constant within the set time range. A status value reading and storing function for sequentially reading and storing status values including status value numbers recorded in all nodes at every elapse of time, and fault validity of all system components An initial failure possibility setting function to be set as "1"; a state value abnormality presence / absence determination function to detect a state value indicating that there is an abnormality from among the state values stored by the state value reading / storing function; In connection with the status value detected by the abnormality presence / absence determination function, the first
First and second from the and second definition tables, respectively
Is obtained by subtracting the multiplied value of the first and second certainties from the set fault validity “1” of each set system component and calculating the fault validity of the corresponding system component. A computer-readable recording medium that stores a failure factor identification processing program for realizing a calculation function.