JP2716537B2 - Down monitoring processing method in complex system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Summary] It is reported that a certain business or cluster has entered an abnormal state.
Regarding the down monitoring processing method in a complex system that notifies the down monitoring mechanism from a business or cluster that has entered an abnormal state, the down monitoring mechanism is integrated at the OS level.
Each cluster has a down monitoring mechanism to monitor the occurrence of an error in a subsystem in its own cluster. Means for notifying subsystems in the own cluster that an abnormal state has occurred when a failure occurs in a subsystem in the own cluster to a down monitoring mechanism existing in another cluster. When an abnormality notification is received, there is provided a means for calling a sub-system of the same type registered in advance or a processing unit for performing processing relating to the abnormality.

[Industrial applications]

The present invention relates to a down monitoring processing method in a complex system that notifies a down monitoring mechanism from a service or cluster that has entered an abnormal state when a certain task or cluster has entered an abnormal state.

When a complex system in which a plurality of computers are connected by a communication path or the like is constructed, a mechanism for recognizing an abnormality of one computer by another normal computer is required. This mechanism is called "down monitoring mechanism". The computers that make up the complex system
It is called a “cluster”.

In addition, when subsystems that perform the same kind of business are operated in multiple clusters, and a subsystem in one cluster fails, the subsystem in another cluster collects resources of the failed subsystem. If possible, it is necessary to take over the work.

[Conventional technology]

 FIG. 8 shows an example of conventional down monitoring.

Down monitoring mechanism 15A, 15B in conventional complex system
As shown in FIG. 8, the normal clusters 13A and 13B mutually perform a live notification such as so-called "I'm ALIVE" to monitor an abnormal cluster. If the survival notification does not arrive within a predetermined period, a survival inquiry is made several times, and if there is still no response, it is recognized that the other party is in an abnormal state.

Also, even when subsystems of the same type performing a certain task mutually monitor abnormalities, the occurrence of abnormalities is recognized by periodically notifying the existence of each subsystem. Was.

[Problems to be solved by the invention]

In the conventional down monitoring, the liveness notification that informs that the user is normal is made at regular intervals, so that even if an error occurs, other clusters immediately There was a problem in that it was not possible to recognize, and the detection of abnormalities was delayed by the notification interval.

In addition, when subsystems of the same type performing a certain task monitor each other for abnormalities, a down monitoring mechanism must be created for each "task (subsystem)", which increases the development burden. There was a problem. Furthermore, when a plurality of subsystems operate in a single cluster, a down monitoring mechanism is required for each subsystem, which causes a problem that CPU resources and memory resources are pressed.

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above problems, integrate a down monitoring mechanism at the OS level, and efficiently execute the recognition of the occurrence of an abnormality and the processing for the abnormality.

[Means for solving the problem]

 FIG. 1 shows a configuration example of the present invention.

In FIG. 1, reference numeral 10 denotes an inter-cluster external storage device which can be commonly accessed by each cluster; 11, an inter-cluster communication area for storing communication data between clusters;
Represents a cluster monitoring table for managing the state of the cluster (operating, stopped, down, etc.).

13A and 13B are clusters each composed of a processing device having a CPU and a memory, and 14A and 14B are subsystem monitoring tables for managing the status (operating, stopped, down, etc.) of subsystems operating in the own cluster, 15A , 15B is a down monitoring mechanism that integrates at the operating system (OS) level to achieve system consistency, 16 is a monitoring unit, 17 is a notification unit, 18 is an exit schedule unit, S1 to S3 are database management, TSS and others E1 denotes a resource recovery outlet which serves as a window for performing post-failure processing for recovering resources held by the subsystem in which an error has occurred.

In the present invention, each cluster 13A, 13B has a down monitoring mechanism 15A, 15B at the OS level.

Each of the down monitoring mechanisms 15A and 15B has a processing unit including a monitoring unit 16, a notification unit 17, and an exit schedule unit 18. The monitoring unit 16 performs processing for monitoring the occurrence of an abnormality in the subsystems S1 to S3 in the own cluster. The notification unit 17 is configured to perform an operation when an error occurs in any of the subsystems S1 to S3 in the own cluster.
A process of notifying the down monitoring mechanism existing in another cluster that the subsystem has entered an abnormal state via the inter-cluster communication area 11 or the like is performed. Exit schedule section 18
Is a processing unit that performs processing related to subsystems of the same type or that is registered in advance by referring to the subsystem monitoring table 14B, etc., when subsystems are notified of abnormalities related to subsystems from other clusters. Process to call the resource recovery exit E1 etc.

[Action]

When a certain subsystem is in a state where the operation cannot be continued, an error is notified to another cluster prior to a subsystem down process for collecting a subsystem operating environment or the like. In the present invention, such cluster monitoring and subsystem monitoring mechanisms are integrated at the OS level as down monitoring mechanisms 15A and 15B, thereby realizing system consistency.

Each of the subsystems S1 to S3 makes a monitoring request to the down monitoring mechanisms 15A and 15B in advance. The down monitoring mechanisms 15A and 15B register the status of the subsystem to be monitored in the subsystem monitoring tables 14A and 14B.

When each subsystem S1 to S3 detects an abnormality in its own subsystem, it notifies the down monitoring mechanism 15A or the like of the subsystem name in which the abnormality has occurred by self-report. Also, the down monitoring mechanism 15A, etc.
Depending on the schedule of the survival notification exit for each of 1 to S3, etc.
Detect subsystem errors.

For example, when the down monitoring mechanism 15A detects an abnormality in the subsystem S1 in the cluster 13A, the notifying unit 17 notifies the down monitoring mechanism 15B of the cluster 13B of the abnormality using an appropriate communication means. The information to be notified includes the name of the cluster and the name of the subsystem that have entered the abnormal state.

Down monitoring mechanism 15B in the cluster 13B that received the notification
The exit scheduler 18 of the failed subsystem
The subsystem monitoring table 14B is searched by the subsystem name of S1, and the subsystem operating in the local cluster is searched.
Schedule the resource recovery exit E1 of S1. The resource recovery exit E1 recovers resources such as the operating environment used by the failed subsystem S1 in the cluster 13A, and takes over processing such as transactions being processed as necessary.

〔Example〕

FIG. 2 is an example of an application system of the present invention, FIG. 3 is an example of a control table used in the embodiment of the present invention, FIG. 4 is a processing flow at the time of initialization of the down monitoring mechanism according to the embodiment of the present invention,
FIG. 5 is a processing flow of the monitoring unit according to the embodiment of the present invention, and FIG.
FIG. 4 is an explanatory diagram of a down monitoring mechanism process according to the embodiment of the present invention;
FIG. 7 shows an example of processing at the time of a down notification of the down monitoring mechanism according to the embodiment of the present invention.

The present invention can be applied to, for example, a complex system as shown in FIG. The system storage device 21 has an inter-cluster communication function, and is composed of, for example, a semiconductor storage device that can be shared between clusters. It is also possible to replace the inter-cluster communication function in this example with a conventional communication function such as inter-processor communication, and to replace the function as an external storage device that can be shared between clusters with a DASD such as a magnetic disk device. .

In the example shown in FIG. 2, the processing units 23 constituting each cluster
Has a CPU 24 and a locally used memory 25, and has a memory controller (MCU) 2 for the system storage device 21.
It can be accessed via 2.

Intercluster communication area 11 and cluster monitoring table 12 shown in FIG.
Are created in the system storage device 21, and the subsystem monitoring tables 14A and 14B shown in FIG. 1 are created in the memory 25 of each cluster.

The cluster monitoring table 12 is for monitoring the operation state of each cluster, and includes, for example, an information storage area as shown in FIG.

The cluster identifier is an identification name or an identification number that uniquely determines a cluster in a complex system. In the status display area, a display indicating that the cluster has started operation and has become a monitoring target, alive display, and the like are performed. The number of subsystems operating in the cluster is stored in the area of the number of operating subsystems. The pointer information of the resource recovery outlet is information for correctly grasping the virtual space address of the resource recovery outlet for recovering the resource of the abnormal cluster even if the cluster is different.

The subsystem monitoring table 14 is for monitoring the operation state of the subsystem for each cluster, and is composed of, for example, an information storage area as shown in FIG.

The subsystem identifier is information for uniquely identifying the subsystem. In the status display area, a display indicating that the subsystem has become a monitoring target, an alive display, and the like are performed.
The alive notification exit address is an address of an exit called by the down monitoring mechanism to periodically perform a live notification at a certain interval. The resource recovery exit address is used to recover resources of subsystems of the same type that are operating in another cluster, and to be called by the down monitoring mechanism when another cluster goes down or a subsystem in another cluster goes down. Address of the routine.

The processing at the time of initialization of the down monitoring mechanisms 15A and 15B shown in FIG. 1 is, for example, as shown in FIG.

It is determined whether initialization of the system storage shared by each cluster is necessary. If another cluster has already been initialized, there is no need to initialize, so the next process is skipped.

 Initialize the cluster monitoring table 12.

It is determined whether or not the cluster monitoring table 12 indicates that the own cluster is alive. If there is a display indicating that the own cluster is alive, it means that an error has occurred in the own cluster before, and that the cluster is currently being restarted after the down. Therefore, the following processing is performed. If there is no live indication, skip the next process.

Calls the resource recovery exit for post-processing of the failure and returns the previously used system storage resources.

Register information about the local cluster in the cluster monitoring table 12.

In the cluster monitoring table 12, the existence of the local cluster is displayed.

A trigger for activation is given to the monitoring unit 16 that operates periodically. Thereafter, the initialization processing ends.

The started monitoring unit 16 executes processes such as the processes shown in FIG.

Set the time interval of "I'm ALIVE" communication for notifying the existence of the own cluster to the other cluster.

Set the monitoring time interval for "I'm ALIVE" communication sent from another cluster.

Set the monitoring time interval for scheduling the survival notification exit of the subsystem in the own cluster. afterwards,
Start monitoring.

When the monitoring time of the subsystem has come, the existence notification exit of the monitored subsystem is scheduled and called. This processing is performed at predetermined monitoring time intervals. Although not shown in the drawings, the notification of the existence of the own cluster to another cluster and the monitoring of the non-existence of the existence notification from the other cluster are also performed at predetermined time intervals. The processing related to the existence notification may be the same as the conventional processing.

The interface between each subsystem S1 and the like and the down monitoring mechanism 15 is, for example, as shown in FIG.

Each subsystem S1 requests the down monitoring mechanism 15 to monitor the subsystem by using a macro interface. On the other hand, the down monitoring mechanism 15 registers information required for monitoring the subsystem identifier, the survival notification exit, the resource recovery exit, and the like in the subsystem monitoring table. Hereafter, the fifth
Monitoring of this subsystem is performed by the processing of the monitoring unit shown in the figure.

The monitoring unit of the down monitoring mechanism 15 periodically schedules the existence notification exit registered in the subsystem monitoring table. At this survival notification exit, if the own subsystem is operating normally, a survival claim is made by the macro interface. The down monitoring mechanism 15 displays that the subsystem is alive in the status display area of the subsystem monitoring table. If there is no notification of alive even after waiting for a certain time, it is considered that an error has occurred in this subsystem.

If the subsystem S1 or the like detects an abnormality by itself, the down monitoring mechanism 15
Self-declaration of abnormalities. In addition, since various methods are conventionally known as the technology of the subsystem detecting its own operation abnormality, detailed description thereof is omitted here.

When detecting that the subsystem S1 or the like has entered an abnormal state, the down monitoring mechanism 15 notifies the down monitoring mechanism 15 of another cluster of the down state. The down monitoring mechanism 15
Depending on the type of subsystem monitoring request for
After notifying the down monitoring mechanism 15 of the other cluster, a cluster stop is scheduled.

When terminating the processing, the subsystem S1 etc. uses the macro interface to leave the subsystem monitoring.
Request the down monitoring mechanism 15. On the other hand, the down monitoring mechanism 15 deletes the information from the subsystem monitoring table. If necessary, the resource collection routine (resource collection exit) for the own cluster is deleted from the cluster monitoring table.

When a down notification of a cluster or a subsystem is received from another cluster, the down monitoring mechanism 15 performs the processing shown in FIG.

The down monitoring mechanism 15 refers to the subsystem monitoring table in the cluster that has received the notification, schedules the resource recovery exit E1 of the down monitoring target subsystem registered in this table, and sends a shutdown to the same type of subsystem. To the effect. As a result, the subsystem S1 can recover the resources of the subsystem that has become abnormal, and take over the business as needed.

In the case of the cluster stop type, the inter-cluster shared resources (system resources) are collected prior to the schedule of the resource collection exit.

〔The invention's effect〕

As described above, according to the present invention, the following effects are obtained as compared with the conventional monitoring method using only the survival notification.

(A) The abnormality of another cluster can be instantaneously recognized accurately without being affected by the load state of the system and self-reported from the place where the abnormality occurs. This is particularly useful in a hot standby system or the like, because the speed of system switching processing can be increased.

(B) By integrating the down monitoring mechanisms, it is not necessary to monitor a cluster error or a certain business error by a plurality of down monitoring mechanisms, and it is possible to eliminate the problem of misalignment between the down monitoring mechanisms.

[Brief description of the drawings]

FIG. 1 is an example of the configuration of the present invention, FIG. 2 is an example of an application system of the present invention, FIG. 3 is an example of a control table used in an embodiment of the present invention, and FIG. FIG. 5 is a flowchart of a process performed by a monitoring unit according to an embodiment of the present invention; FIG. 6 is an explanatory diagram of a process performed by a down monitor according to an embodiment of the present invention; 8 shows an example of processing at the time of a down notification of the down monitoring mechanism according to the embodiment, and FIG. 8 shows an example of conventional down monitoring. In the figure, 10 is an inter-cluster external storage device, 11 is an inter-cluster communication area, 12 is a cluster monitoring table, 13A and 13B are clusters, 14A and 14B are subsystem monitoring tables, 15A and 15B are down monitoring mechanisms, and 16 is monitoring. , 17 is a notification unit, 18 is an exit schedule unit, S1 to S3 are subsystems, and E1 is a resource recovery exit.

Claims (1)

(57) [Claims] A plurality of clusters each having a computer (13A,
In a complex system consisting of 13B), a down monitoring mechanism (15A, 15B) with a unified interface for multiple subsystems for each cluster
And a subsystem monitoring table (14A, 14B) that manages the status of each subsystem. Each of the down monitoring mechanisms responds to a monitoring request from a subsystem in its own cluster. Means for registering the identification information of the subsystem, a first interface for notification of existence, and a second interface for abnormality processing; and referring to the subsystem monitoring table, to monitor the subsystem to be monitored at a predetermined cycle. Means for calling the first interface of the above, and when there is no survival notification from the first interface for a certain period of time or when there is a self-report of abnormality from the subsystem, it is recognized that the subsystem has entered an abnormal state. When a failure monitoring mechanism that exists in another cluster is notified, Serial subsystem monitoring table down monitoring processing method in a complex system characterized by comprising a means for calling the second interface that is registered in.