JP2000353154A - Fault monitoring system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately grasp failures generated in a large scale parallel computer system in the order of generation time series. SOLUTION: The failure monitoring system is provided with a plurality of nodes 10 being respectively independent computers, cross bar data switches 20 for switching a signal route between the nodes, a cross bar controller 30 for controlling the drive of the switches 20, slave service processors 40 connected to the nodes 10, the switches 20 and the controller 30 and allowed to monitor failures and output prescribed failure information immediately after detecting a failure, and a master service processor 50 for monitoring the generation time series of the failure by receiving the failure information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a fault monitoring system, and more particularly, to a fault monitoring system used in a large-scale parallel computer system.

[0002]

2. Description of the Related Art Conventionally, large-scale numerical calculations such as numerical simulations of flows have been performed using large-scale parallel computer systems. That is, the calculation of the simultaneous linear equations and the like is shared among a plurality of computers (hereinafter, referred to as nodes) to perform the calculation at high speed. As an example of such a large-scale parallel computer, for example, there is a supercomputer SX-5 series manufactured by Nippon Electric Co., Ltd.
This achieves the computational performance of 4TFLOPS by installing a maximum of 32 nodes equipped with 16 CPUs.

Incidentally, such a large-scale parallel computer system is generally provided with a fault monitoring device called a service processor in order to monitor and recover from a fault occurring in the system. The service processor
Generally, a plurality of devices are installed in the system. Each device (for example, a node, a crossbar data switch, a crossbar control device, and the like) that configures the system is provided, or the devices in the system are divided into a plurality of groups. Each of them is provided separately. Accordingly, each service processor independently monitors a fault of a device (or a device group) in charge, and individually stores fault information in a hard disk drive or the like.

[0004] As a failure that occurs in the system, for example, there is a failure of a register of a CPU in a node. When a failure occurs in a register, the calculation at the failed node fails first, and the calculation performed at each node in the parallel calculation is related to each other as described above. It propagates to other nodes one after another, and the calculation of the entire system stops instantly.

When such a situation occurs, in order to recover the stopped calculation, it is necessary to identify the failed node at an early stage and replace the CPU.

[0006]

However, in the conventional parallel computer system, since a plurality of service processors function independently, there is a problem that it is difficult to uniquely determine a time series of occurrence of fault information. . That is, in each service processor, the fault information with the fault occurrence time added is recorded on a hard disk drive or the like, but the clocks built in each service processor are not always completely synchronized. When examining fault information after an accident, the order in which faults occurred close in time may become unknown.

[0007] Even if the clocks of the service processors are completely synchronized, it is difficult to accurately grasp the order of occurrence of faults that occur within the resolution of time measurement.

SUMMARY OF THE INVENTION The present invention has been made to solve such a problem, and provides a fault monitoring system in a large-scale parallel computer system capable of accurately grasping faults that have occurred in the system in the order of occurrence in chronological order. The purpose is to do.

[0009]

In order to achieve the above object, a fault monitoring system according to the present invention comprises a plurality of nodes, each of which is an independent computer, and a crossbar data for switching a signal path between the nodes. A switch, a crossbar control device for controlling the driving of the crossbar data switch, and a node connected to the node and the crossbar data switch and the crossbar control device for monitoring a fault and outputting predetermined fault information immediately upon detecting the fault. It has a slave service processor and a master service processor that monitors the time series of the failure by receiving the failure information.

[0010] Other aspects of the present invention include the following. That is, the slave service processor may have a hard disk drive for recording the failure information. Further, the master service processor may include a hard disk drive for recording the failure information. Further, the failure information may include a number of a reporting slave service processor, a number of a device in which the failure has occurred, and a registration number of the failure at the reporting source. Further, the failure information may further include a number indicating a degree of the failure. Further, the fault monitoring system may be applied to a massively parallel computer system.

[0011]

Next, one embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing one embodiment of the present invention. As shown in the figure, the fault monitoring system according to the present embodiment includes a plurality of nodes 10, a plurality of crossbar data switches 20,
Crossbar controllers 30 and a plurality of slave service processors 4 having a hard disk drive 40a
0 and one master service processor 50 having a hard disk drive 50a.

The node 10 is a computer equipped with one or a plurality of CPUs, and functions as an independent computer for each node. In the figure, a total of eight nodes 0 to 7 are installed.

The crossbar data switch 20 is a device that is connected between the nodes 10 and switches the path of a signal transmitted between the nodes. Therefore, each node 10
The crossbar data switch 20 enables data transmission and reception between the nodes 10 and communication between processors. In the figure, crossbar data switches 0 to
There are a total of 4 units, 3 units. Each node 10 and the crossbar data switch 20 existing in the system
Is a communication line dedicated to failure processing that is independent of a communication line that mediates the node 10 and the crossbar data switch 20,
Connected to slave SVP40

The crossbar control device 30 is connected to each node and controls the driving of the crossbar data switch 20.

A slave service processor (hereinafter, referred to as a slave SVP) 20 having a hard disk drive (hereinafter, referred to as an HDD) 40a is a fault processing device that is connected to each device in the system and monitors a fault. When a failure is found, failure information to be described later is output and stored in the HDD 40a and the master SVP
Send to 50. In the figure, slaves SVP0-4
Are installed and connected to the node 10, the crossbar data switch 20, and the crossbar control device 30.

A master service processor (hereinafter, referred to as a master SVP) 50 having an HDD 50a is an integrated fault processing device that receives fault information sent from the slave SVP 40 and monitors a fault in the entire system. Therefore, the master SVP 50 is connected to all the slave SVPs 40 and transmits the received fault information to the HDD 50.
Stored in a.

As described above, the fault monitoring system according to the present embodiment comprises a plurality of nodes 10, a crossbar network device between nodes (crossbar data switch 20 and crossbar control device 30), and a plurality of fault processing devices (slave SVP40). And a master SVP 50).

The nodes 1 constituting these systems
0, crossbar data switch 20, crossbar controller 3
0, slave SVP40 and master SVP50,
Each device is assigned a unique device number within the system. Therefore, each slave SVP 40 performs a failure process using the same device number table (Table 1).

[Table 1] {Device name Device number} ─ Master SVP 000 Slave SVP0 010 Slave SVP1 011 Slave SVP2 012 Slave SVP3 013 Slave SVP4 014 Node 0 020 Node 1 021 Node 2 022 Node 3 023 Node 4 024 Node 5 025 Node 6 026 Node 7 Data switch Crossbar Control 0 031 Crossbar data switch 1 032 Crossbar data switch 2 033 Crossbar data switch 3 034 ────────────────────

Next, the operation of the present invention will be described.

[When a Single Failure Occurs] Here,
The flow of the failure information collecting process will be described by taking as an example a case where a failure has occurred in a single node (it is assumed that a failure has occurred in the node 7 in FIG. 1).

First, the slave SVP 4 monitoring the failure of the node 7 transmits failure information to the master SVP 50 when the occurrence of the failure is detected in the node 7. That is,
The device number “014” of the slave SVP4, the device number “027” of the failed node 7 and the slave SVP
The failure information including the sequential registration number “00000000214” of the failure information managed by the VP 4 and the number “4” indicating the degree of the failure is transmitted.

FIG. 2 shows that the slave SVP 4 is the master SVP
FIG. 9 is an explanatory diagram showing communication contents (failure information) for notifying the occurrence of a failure, which is transmitted to the communication device 50; In the figure, the device number is determined based on Table 1, and the number indicating the degree of failure is determined based on Table 2.

[Table 2] 度 合 い Degree of failure Abbreviation No. ────── ────────────────────────── Severe failure CHECK 0 Failure that could be severe in the future CAUTION 1 Mild failure WARNING 2 Signal other than failure ATTENTION 3 ────────────────────────────────

When it is desired to know only the time series of the occurrence of a fault, it is not necessary to add a code indicating the degree of the fault. It can be said that reducing the data size of the fault information as much as possible is convenient for performing fault notification at high speed.
If there is no transmission problem, other information may be added to the failure information.

Thereafter, the master SVP receiving the failure information
5 assigns a sequential registration number managed by the master SVP 50 to the received fault information, and further assigns time information by a clock managed by the master SVP 50 as a fault occurrence time to configure the fault registration information. S
Recording is performed on the HDD 5a connected to the VP5.

An example of the failure registration information is as follows. In order from the left, the date of failure (master SV
P50), time of failure (added by master SVP), failure registration number in master SVP 50, number of slave SVP in which failure was detected, name of device in which failure occurred, failure registration number in slave SVP, degree of failure Is an abbreviation that indicates Node *** indicates node 10, IXS
** indicates the crossbar data switch 20.

99-02-15 20:51:40 0000005409 SVP02 Node000 0000001027 WARNING 99-02-15 21:35:48 0000005410 SVP03 Node005 0000000873 ATTENTION 99-02-15 23:54:39 0000005411 SVP04 Node006 0000001354 WARNING 99- 02-16 01:15:42 0000005412 SVP00 IXS00 0000001161 CAUTION 99-02-16 10:38:09 0000005413 SVP00 IXS11 0000001162 CHECK 99-02-16 11:22:50 0000005414 SVP03 Node004 0000000874 ATTENTION 99-02-17 11: 22:47 0000005415 SVP02 Node000 0000001028 CAUTION 99-02-18 15:23:53 0000005416 SVP02 Node002 0000001029 WARNING 99-02-19 14:16:50 0000005417 SVP04 Node006 0000001355 ATTENTION

FIG. 3 is an explanatory diagram showing fault information stored in the master SVP and the slave SVP, respectively.
As shown in the figure, the master SVP 5 can refer to detailed failure information managed by the slave SVP 4 based on the failure report notification device number and the report failure information registration number. In the slave SVP4, the master SVP
After notifying the failure occurrence to 50, detailed failure contents are collected from the node 7, and the collected detailed failure information is recorded in the HDD 40a attached to the slave SVP4.

In the above description, the procedure for collecting fault information when a single fault has occurred has been described. However, even when a plurality of devices have successive faults, the same procedure can be taken. Can be.

[Case where a Plurality of Faults Occur Simultaneously] For example, a case where a fault occurs continuously between the node 7 and the crossbar data switch 2 will be described.

First, the slave SVP corresponding to the node 7
4 is the slave S of the reporting source with respect to the master SVP 50.
The device number “014” of the VP4, the number “027” of the failed device, and the failure registration number of the reporting source are transmitted.
The collection of detailed fault information in the node 7 is started. The master SVP 50 adds the integrated fault registration number and the time at which the notification was received to the notified fault information, and adds
D50a is recorded as failure registration information.

Next, a failure occurs in the crossbar data switch 2, and the corresponding slave SVP2 becomes the master SVP.
For 50, the device number “011” of the reporting slave SVP1 and the number “033” of the failed device
And the fault registration number of the report source. Master SVP
50 adds the integrated fault registration number in the master SVP 50 and the time at which the notification was received to the received fault information, and records the information on the HDD 50a.

The fault information recorded by the master SVP 50 is very simple as shown in FIG. 3, and has a small data size. Therefore, the failure information is transmitted to the master SVP 50 in a short time,
The VP 50 can handle one failure in a short time. Even when the node 7 and the crossbar data switch 2 successively cause a failure and the time interval between the two failures is very small, the possibility that processing batting occurs on the master SVP 50 is very small, and the failure occurrence time Is performed almost simultaneously with the occurrence of a failure.

FIG. 4 is a block diagram showing details of the fault monitoring system according to FIG. As shown in FIG. 1, a node 10 includes a failure detection circuit 10a that outputs a 1-bit signal when a failure is detected in the node 10, a register 10b,
10d and an OR circuit 10c. The slave SVP 40 includes registers 40b and 40e, and a decoder 40c that outputs a code set according to the node number.
And a selector 40d. Master SVP
Reference numeral 50 denotes a register 50b, a selector 50c,
O (First In First Out) buffer 50d.

The operation of this fault monitoring system is as follows. As soon as the failure detection circuit 10a detects a failure in the node 10, a 1-bit signal is output via the register 10b, the OR circuit 10c, and the register 10d. The output signal is input to the decoder 40c via the register 40b of the slave SVP 40.
The decoder 40c is set so that a unique code is output for each connected node.

The output of the register 40b is input not only to the decoder 40c but also to the control terminal of the selector 40d.
0e. Therefore, only the signal that first arrives at the slave SVP 40 is input to the master SVP 50. The signal output from the slave SVP 40 is stored in the FIFO buffer 50d via the register 50b and the selector 50c of the master SVP 50. The operation of the selector 50c is the same as that of the above-described selector 40d. The buffer 50d is a storage device such as a RAM, and can read and write at a higher speed than the HDD 50a. Therefore, the failure notification signal (failure information) intensively sent in a short time is temporarily stored in the buffer 50d.
Is written to the HDD 50a.

The above-described configuration and procedure for failure detection are the same in the crossbar data switch 20, the crossbar control device 30, and other circuits in the system. The clock skew of each line (metallic cable or optical fiber cable) connecting the slave SVP and the master SVP is adjusted to be equal.

[0039]

As described above, the present invention is provided in a node and a crossbar data switch and a crossbar control device, monitors a fault and outputs fault information immediately upon finding the fault, and a slave service processor. A master service processor that monitors failure occurrence time series by receiving failure information from the processor.

With this configuration, the present invention provides:
Since the registration of all the faults that have occurred in the computer system is finally performed by one device, the master SVP,
The fault occurrence order is stored in a state where it can be uniquely determined.

Further, since collection and accumulation of detailed fault information are entrusted to fault processing devices connected to the respective devices constituting the system, even when a plurality of faults occur in close proximity, a specific fault processing device can be used. The possibility of concentration of load and information is reduced, and load distribution is performed.

[Brief description of the drawings]

FIG. 1 is a block diagram showing one embodiment of the present invention.

FIG. 2 is an explanatory diagram showing fault information transmitted from a slave SVP to a master SVP.

FIG. 3 is an explanatory diagram showing data registered in a master SVP and a slave SVP.

FIG. 4 is a block diagram showing details of a fault monitoring system according to FIG. 1;

[Explanation of symbols]

10 ... node, 20 ... crossbar data switch, 30 ...
Crossbar control device, 40: slave service processor (slave SVP), 50: master service processor (master SVP), 10a: fault detection circuit, 10
b, 10d: register, 10c: OR circuit, 40b, 4
0e: register, 40c: decoder, 40d: selector, 50b: register, 50c: selector, 50d: buffer, 40a, 50a: hard disk drive.

Claims (6)

[Claims] A plurality of nodes each being an independent computer; a crossbar data switch for switching a signal path between the nodes; a crossbar control device for controlling driving of the crossbar data switch; and the node and the crossbar data. A slave service processor that is connected to the switch and the crossbar control device, monitors a fault, and outputs predetermined fault information immediately upon detection of the fault, and receives the fault information to monitor a time series of the fault occurrence A fault monitoring system comprising a master service processor. 2. The fault monitoring system according to claim 1, wherein the slave service processor has a hard disk drive for recording the fault information. 3. The fault monitoring system according to claim 1, wherein the master service processor has a hard disk drive for recording the fault information. 4. The system according to claim 1, wherein the fault information includes a number of a slave service processor of a report source, a number of a device in which the fault has occurred, and a registration number of the fault at the report source. Characteristic fault monitoring system. 5. The fault monitoring system according to claim 4, wherein the fault information further includes a number indicating a degree of the fault. 6. The fault monitoring system according to claim 1, wherein the fault monitoring system is applied to a large-scale parallel computer system.