JP2001331330A - Process abnormality detection and restoration system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically and quickly restore an abnormal process and a process group related to the abnormal process in respect to a process abnormality detection and restoration system, in a process system constituted of plural processes. SOLUTION: The system is provided with a monitoring process l-l for monitoring the operation states of respective processes 1-3 distributed in a server, a client or another system and a data base 1-2 for storing the dependent relation of various patterns concerned with restart processing among respective processes. At detecting of a fault in one of processes 1-3 to be monitored, the monitoring process 1-3 causes plural process groups, having depending relation with the process concerned, to restart the processes.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a process abnormality detection and recovery system in a process system including a plurality of processes, and more particularly to a distributed processing middleware (CORBA: Common Object Request Broker Archi).
The present invention relates to a technology for detecting and recovering from an abnormality of each process in a system having a communication means via a network (such as a computer system) and a process group distributed on a plurality of processors or an operating system. Note that the process group may operate on a single processor or operating system.

As a preferred example of the above-mentioned process system including a distributed process group for transmitting and receiving objects using communication means via distributed processing middleware, a network monitoring system for monitoring a group of exchanges in an STM network is a preferred example. Can be

[0003] In a system composed of a plurality of processes, as the number of processes increases, their interdependencies are complicated. In addition, when the process group of the system is distributed on a plurality of processors or operating systems, in addition to the abnormality monitoring of each process, the abnormality detection and recovery of the communication means between the monitoring process and the monitored process are also performed. I have to deal with it.

Further, when one process system cooperates with another process system, or when it is configured based on another process system, not only the process group of the own process system but also the own process system is used. Process groups of other process systems related to the system are also required to be included in the category of abnormality detection and recovery.

[0005] In a process system including such a process group, it is required to establish a means for automatically detecting, classifying, and recovering from an abnormality without manual intervention when an abnormality occurs. The present invention is based on such a demand, and detects, classifies, and recovers from an operation abnormality in a variety of process systems with minimum human intervention as much as possible.

[0006]

2. Description of the Related Art Conventionally, in a system having a process group, when an operation abnormality occurs in a certain process,
In some cases, it is not always possible to recover by only a single restart of the process. However, in order to recover the system, the process of restarting all the process groups of the system involves temporarily stopping all services of the system due to the operation of full restart. Therefore, this is not necessarily an effective means.

[0007] In view of the above, several methods have been proposed so far for the recovery system of the process system. For example, Japanese Patent Publication No. 2500745 discloses that a service control node that controls the execution of an application service for a subscriber determines the execution range of a restart process by comparing the number of failure occurrences of each process with a threshold value. A method of instructing stop and restart of an application service within a range and restoring the application service is disclosed.

Japanese Unexamined Patent Application Publication No. 11-88471 discloses a test method for isolating which part of a failure isolating agent has an abnormality and restoring a file or restarting a process according to the result. And a test device.

[0009]

A process having communication means via distributed processing middleware (such as CORBA) and comprising a plurality of processes distributed on a plurality of physically separated processors or a plurality of operating systems. In a system including a group, the amount of recovery processing and the number of combinations of recovery processing for process abnormalities and failures become enormous, so that automatic abnormality detection and recovery are required.

In particular, in the case where the above-mentioned process system is a system which needs to monitor a monitoring target 24 hours a day, for example, a system for monitoring a communication network or the like, a process abnormality is automatically detected and promptly detected. Need to recover. Further, in the case where a certain process system operates in cooperation with another process system or in a master-slave relationship, it is necessary to extend the range of the monitoring target to a related process group of another processor system.

According to the present invention, in a system having a group of processes, an abnormality of each process is detected, and the abnormal process and a process associated therewith are automatically and promptly restored.
It is another object of the present invention to extend a monitoring target not only to a group of processes in the own system but also to a group of related processes in another system, and to automatically recover the group of related processes when a process abnormality is detected.

[0012]

SUMMARY OF THE INVENTION In a system composed of a plurality of processes, and the processes have a dependency relationship with each other and operate in cooperation with each other, when an abnormality occurs in a certain process, a plurality of processes are required to recover the abnormality. In some cases, a process must be restarted, but there are various patterns in the interdependency of a plurality of processes, and the process becomes more complicated as the number of processes increases.

According to the present invention, (1) in such a process system, a monitoring process for monitoring the operating state of each process distributed to the own device or another device, and various patterns of a restart process between the processes. A monitoring process that, when detecting a failure in any of the processes to be monitored, refers to the database and restarts a process group for a plurality of processes having a dependency with the process. Is executed.

As one of the process abnormalities, there is a case where the number of process restarts is abnormally large. That is,
If a certain process is restarted frequently, inconsistency may occur in information exchange between the process and a process having a dependency.

Therefore, the present invention provides (2) defining a certain time for each process, defining an upper limit of the number of process restarts per the certain time, and storing the upper limit in a database;
When the process restart exceeds the upper limit, the monitoring process detects it, refers to the database that holds the dependency information of the monitored process, and causes the related monitored process to restart. And This makes it possible to detect the number of occurrences of the process restart abnormality and quickly and automatically recover the inconsistency in the information between the processes caused by the process restart.

One of the process abnormalities is a case where the process size abnormally increases. Here, the process size means the ratio of the time during which the process occupies the CPU (the occupancy of the CPU). Then, the increase in the process size (how many times the size at the start) may reduce the processing capacity of the entire system.

Accordingly, the present invention provides (3) a process size upper limit value defined for each process and stored in a database, and when the process size exceeds the upper limit value, the monitoring process detects it and receives the process size. The present invention is characterized in that a database holding dependency information of a monitoring process is referred to, and a related monitoring target process is restarted. An adverse effect on the system due to an abnormal increase in process size can be quickly and automatically detected and restored.

One of the process abnormalities is an abnormal increase in the memory size used by the process. That is, if the memory area used by a certain process abnormally increases, another process sharing the memory cannot operate normally, and the processing capacity of the entire system may be reduced.

Therefore, according to the present invention, (4) the upper limit value (how many times as large as the start time) of the process use memory area is defined for each process and stored in the database, and the process use memory area exceeds the upper limit value. In this case, the monitoring process detects this, refers to the database holding the dependency information of the monitored process, and restarts the related monitoring target process. An adverse effect on the system due to an abnormal increase in the memory area used by the process can be quickly and automatically detected and restored.

Further, as one of the process abnormalities, an abnormal decrease in the memory work area (swap area) in the process processing can be cited. That is, when the memory work area (swap area) is abnormally reduced, the processing capacity of the entire system may be reduced.

Therefore, according to the present invention, (5) the lower limit of the memory work area (swap area) is defined and stored in the database, and the memory work area (swap area) falls below the lower limit in the processing of a certain process. In this case, the monitoring process detects this, refers to the database holding the dependency information of the monitored process, and restarts the related monitoring target process. An adverse effect on the system due to an abnormal decrease in the memory work area (swap area) can be quickly and automatically detected and restored.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an embodiment of the present invention provided with a database holding a dependency relation between processes for restart. As shown in the figure, the process system of the present embodiment includes a monitoring process 1-1 and a monitoring / failure information database 1-2 in a server.
A plurality of processes 1-3 monitored by the server 1 are distributed in a server or a client or another system connected to the server.

Note that the monitoring process 1-
A “client monitoring process” 1-4, which is a resident communication process having a communication function with
One of the clients is a system that can be a “system administrator client” logged in with system administrator authority.

The monitoring process 1-1 has a function 1-11 for periodically or irregularly checking each process 1-3 to be monitored and a function 1-11 for detecting the resumption of each process 1-3. 12 and a function 1-13 for issuing a restart command to each monitored process 1-3 to execute a restart process.

The monitoring / fault information database 1-2 holds dependency information of a group of processes that must be restarted for system recovery when each process performs restart processing. Then, the monitoring process 1-1 acquires dependency information between processes held in the monitoring / failure information database 1-2, and issues a restart instruction to the corresponding process group.

The monitoring process 1-1 and the monitoring / failure information database 1-2 may be provided in a server in the process system, or provided in another information processing apparatus (for example, a personal computer) or a client. It is good also as a configuration. The same applies to the following embodiments.

In the process system shown in FIG. 1, the process of restarting a process group having a dependency is performed according to the flow shown in FIG. First, the monitoring process monitors the process ID of each process regularly or irregularly (when monitoring is necessary) (step 2-1), and
Is changed (Step 2-
2). When the process is restarted, a different process ID is assigned by the operating system, and if the correspondence between the process name and the process ID changes, the monitoring process detects that the monitored process has been restarted. I do.

Next, when it is determined that there is a restart based on the change of the process ID, the monitoring / failure information database is referred to and the dependency information between the processes is obtained (step 2).
-3). Then, based on the interdependency information between the processes, a restart instruction is issued to the corresponding process group to execute the restart (step 2-4).

FIG. 3 shows an embodiment of the present invention provided with a database holding an upper limit of the number of process restarts per unit time. The process system according to this embodiment includes a monitoring process 3- in the server similarly to the embodiment shown in FIG.
1. A monitoring / fault information database 3-2 is provided, and a plurality of processes 3-3 to be monitored exist in the server or in a client or other system connected to the server. A monitoring process is provided, and one of the clients is a system that can be a system administrator client.

The monitoring / fault information database 3-2 includes:
A restart record database 3-21 (see FIG. 21A) that records the time of the process restart and the restarted process;
A restart management database 3-22 (see FIG. 21 (b)) for recording the unit time and the upper limit of the number of process restarts according to each process type and the number of process restarts per unit time. , A dependency database 3-23 that holds dependency information of a group of processes that must be restarted for process system recovery
(See FIG. 21C).

The monitoring process 3-1 has a function 3-1 for checking each process to be monitored, as in the embodiment of FIG.
1 and a function 3-1 for detecting the restart of each monitored process
2 and a function of issuing a restart instruction to each monitored process 3-
13, a function 3-14 for issuing a process restart recording command to the restart record database 3-21 and the restart management database 3-22, and a restart record database 3-21, a restart management database 3-22, and a process restart. 3-15 for referring to the dependency database 3-23 for

In the process system shown in FIG. 3, process restart processing based on the number of process restarts is performed according to the processing flow shown in FIG. First, the monitoring process monitors the process ID of each process regularly or irregularly,
When the process restart is detected by the change of the process ID (step 4-1), the time of the process restart and the name of the restarted process are recorded in the restart record database in the monitoring / failure information database (step 4-2).

Next, the monitoring process calculates the time unit time before the process restart time and refers to the restart record database to measure the number of process restarts during the period from the latest process restart to the unit time before. Is recorded in the restart management database (step 4-3).

Next, the monitoring process refers to the restart management database (step 4-4), determines whether the number of process restarts per unit time exceeds the upper limit (step 4-5), and checks the upper limit. If the number exceeds the limit, the process refers to the dependency database for process restart (step 4-6), and sends a restart instruction to a group of processes having a dependency (step 4-7).

FIG. 5 shows an embodiment of the present invention provided with a database holding the upper limit value of the process size. The process system of this embodiment includes a monitoring process 5-1 and a monitoring / failure information database 5-2 in a server, as in the embodiment of FIG. A plurality of processes 5-3 to be monitored exist in the system, and a client is provided with a client monitoring process, and one of the clients is a system that can be a system administrator client. .

The monitoring / fault information database 5-2 includes:
The process size upper limit database 5-21 (FIG. 2) holding the process size upper limit corresponding to each process type.
2 (a)) and a dependency database 5-22 (FIG. 22) that holds dependency information of a group of processes that must be restarted for recovery when each process exceeds the size upper limit. (See (b)). The process size is the ratio of the time during which the process occupies the CPU (the occupancy of the CPU).

The monitoring process 5-1 has a process size upper limit database 5-2 in the monitoring / fault information database 5-2 in addition to the functions in the embodiment of FIG. 1 or FIG.
1 and a function 5 for referring to the dependency database 5-22
-11.

In the process system shown in FIG. 5, the process restart process based on the process size is performed according to the process flow shown in FIG. First, the monitoring process monitors each process periodically or irregularly to acquire a process size, and refers to the process size upper limit database in the monitoring / failure information database (step 6-1).

Next, the monitoring process determines whether the process size obtained from the processor exceeds the upper limit (step 6-2).
The process refers to the dependency database relating to the process restart (step 6-3), and sends a restart command to a group of processes having a dependency (step 6-4).

FIG. 7 shows an embodiment of the present invention provided with a database for holding the upper limit value of the process use memory area. The process system of this embodiment includes a monitoring process 7-1 and a monitoring / failure information database 7-2 in a server, as in the embodiment of FIG. There are a plurality of processes 7-3 to be monitored in the system, the client is provided with a client monitoring process, and one of the clients is
It is a system that can be a system administrator client.

The monitoring / fault information database 7-2 includes:
Process use memory area upper limit database 7-21 (see FIG. 22 (c)) holding the process use memory area upper limit value corresponding to each process type, and when each process use memory area exceeds the upper limit value And a dependency database 7-22 (see (d) of FIG. 22) that holds dependency information of a process group that must be restarted for recovery.

The monitoring process 7-1 is the same as that shown in FIGS.
Alternatively, in addition to the function in the embodiment of FIG. 3, a function 7-1 for referring to the upper limit database 7-21 of the process use memory area and the dependency database 7-22 related to restart.
1 is provided.

In the process system shown in FIG. 7, the restart processing exceeding the upper limit value of the process use memory area is performed according to the flow shown in FIG. First, the monitoring process monitors each process periodically or irregularly, acquires the value of the process used memory area, and refers to the upper limit database of the process used memory area (step 8-).
1).

Next, the monitoring process determines whether or not the value of the process memory area exceeds the upper limit (step 8-2). If the value exceeds the upper limit, the monitoring process refers to the dependency database regarding process restart. Then, a restart instruction is sent to a group of processes having a dependency (step 8-4).

FIG. 9 shows an embodiment of the present invention provided with a database holding the lower limit value of the memory work area (swap area). The process system of this embodiment includes a monitoring process 9- in the server, as in the embodiment of FIG.
1. A monitoring / failure information database 9-2 is provided, and a plurality of processes 9-3 to be monitored exist in a server or a client or other system connected to the server. And one of the clients
One is a system that can be a system administrator client.

The monitoring / fault information database 9-2 includes:
Memory work area (swap area) lower limit value database 9-that holds the lower limit value of the memory work area (swap area)
21 (see (e) of FIG. 22) and a dependency database 9 for storing dependency information of a group of processes that must be restarted for recovery when the memory work area (swap area) falls below the lower limit. -22 ((f) of FIG. 22)
Reference).

The monitoring process 9-1 refers to the memory work area (swap area) lower limit value database 9-21 and the dependency relation database 9-22 for restarting, in addition to the functions in the embodiment shown in FIGS. Function 9
-11.

In the process system shown in FIG. 9, the resuming process below the lower limit of the memory work area (swap area) is performed according to the flow shown in FIG. First, the monitoring process monitors each process periodically or irregularly, acquires the value of the swap area, and refers to the memory work area (swap area) lower limit database 9-21 (step 10-1).

Next, the monitoring process determines whether the memory work area (swap area) is below the lower limit (step 10-2).
The process refers to the dependency database relating to the process restart (step 10-3), and sends a restart command to a group of processes having a dependency (step 10-4).

FIG. 11 shows an embodiment of the present invention for performing regular monitoring and irregular monitoring by a system administrator. The process system of this embodiment includes a monitoring process 11-1 and a monitoring / failure information database 11-2 in a server, as in the embodiment of FIG. 1 described above, and a client or a client connected to or in the server. A plurality of processes 11-3 to be monitored exist in another system, and a client monitoring process 11-
4 and one of the clients is
It is a system that can be a system administrator client.

In the present embodiment, there are the following two types of execution of monitoring. a. Irregular monitoring performed by activation of a system administrator authority user b. Periodic monitoring led by the monitoring process

The above-mentioned irregular monitoring is realized in the following order. Using the “System Administrator Client” logged in with system administrator authority, the system administrator sends a monitoring request (A) at irregular intervals (when monitoring is necessary). Alternatively, the user logs in to the “client monitoring process”, which is a communication process resident on the client, with a system administrator account, and sends out a monitoring request (A).

The monitoring request A is passed to the monitoring process 11-1 in the server via the client monitoring process 11-4. The monitoring process 11-1 performs abnormality detection and recovery processing of each process described above according to the monitoring request (A).

In the periodic monitoring performed by the monitoring process initiative b, the monitoring process 11-3 periodically checks the monitoring target process 11-3 at predetermined time intervals according to the setting of a timer or the like by the autonomous operation of the monitoring process 11-1. B)
Abnormality detection and recovery processing of each process described above are performed.

With such a configuration, in addition to periodically monitoring the operation status of each process by starting the timer of the monitoring process, the monitoring process can be performed when the user of the process system wants to check the failure at any time. Can be started to perform monitoring.

Next, an embodiment of the present invention for detecting the root cause of a fault will be described. In a process system constructed using communication means via distributed processing middleware (such as CORBA), information about a failure transmitted from a group of monitored processes includes a plurality of layers (physical layer, TCP / IP layer, middleware layer / It is transmitted via a communication protocol stack that straddles the application layer.

Therefore, when any failure occurs in the communication means, some of the communication failures in the upper protocol layer may be caused by a communication failure in the lower protocol layer. In such a case, in the monitoring process, the root cause of the failure in the process system can be clearly specified by classifying the failure information received from the monitored process group by the protocol layer.

That is, for a number of alarms generated on the monitoring process, a fault is detected for each protocol stack.
Only the failure cause information of the lower layer that caused the failure
A device (for example, a client) that executes each process is notified as an alarm or an event, and alarm information that is easily understood by a monitor is displayed.

FIG. 12 shows an embodiment of the present invention for detecting the underlying cause of a fault. The process system of this embodiment includes a monitoring process 12-1 and a monitoring / fault information database 1 in a server, as in the embodiment of FIG.
2-2, a plurality of processes 12-3 to be monitored exist in the server or a client or other system connected to the server, and the client has a client monitoring process 12-4. One of the clients is a system that can be a system administrator client.

FIG. 13 shows a flow of monitoring execution in the embodiment of FIG. Also in the present embodiment, there are the following two types of execution of monitoring. a. Irregular monitoring performed by system administrator authority user trigger b. Periodic monitoring led by the monitoring process

In the case of the irregular monitoring described in a above, the system administrator sends a monitoring request (A) at irregular intervals by using the “System Administrator Client” logged in with system administrator authority (A). Step 13-
1). The monitoring request (A) is passed to the monitoring process 12-1 in the server via the client resident process 12-4 (step 13-2).

In the following, similarly to the periodic monitoring (B) led by the monitoring process described in b above, the monitoring process 12-1 performs abnormality detection and recovery processing of each process (step 13-3). When the monitoring process 12-1 receives the monitoring request (A) from the client monitoring process 12-4, or when the time for the periodic monitoring (B) is reached, the root cause of the failure is determined by the procedure described in detail below (step). 13-4).

The procedure for determining the root cause of a failure is as follows:
The monitoring process 12-1 accesses the monitoring / failure information database 12-2, and acquires a monitoring method (command) and a monitored process IP address as information for performing the monitoring process. The monitoring method (command) is used command registration database 12-21 (see FIG. 23 (a)).
, And the monitored process IP address is the monitored process name / IP address correspondence database 12-2.
2 (see FIG. 23B). The data type of the acquired data is “monitored process name, monitored process IP address”.

Next, the monitoring process 12-1 starts monitoring based on the acquired data (monitored process name, monitored process IP address), and uses the used command registration database 12- against the monitored process IP address. All the commands registered in 21 are input.

Next, after receiving the response results for all commands, the monitoring process 12-1 determines whether or not a failure has occurred in the monitored process and the failure occurrence layer based on the response results, and Sorting is performed to determine the underlying cause.

FIG. 14 shows the logic for determining the underlying cause.
First, it is determined whether a normal response result to a command of a lower protocol such as a physical layer or a TCP / IP layer has been obtained (whether transmission has been successful) (step 14-1).
When a normal response result is obtained, it is determined whether a normal response result to a command of a higher-order protocol such as the CORBA layer or the application layer is obtained (transmission is successful) (step 14-2).

Next, if the transmission is not successful, it is determined whether the monitored process IP address is a client (step 14-3). With the above determination, the determination results 1 to 4 shown in FIG. 14 are obtained.

Here, the determination results 1 to 4 are as follows. -Result 1: Normal (normal in the low-level protocol and high-level protocol)-Result 2: Normal (because the monitored process is a client machine)-Result 3: Abnormality in the physical layer and TCP / IP layer-Result 4: CORBA layer, application Layer error

When the above result 1 is obtained, the application is checked because it is determined that there is no abnormality in the communication means. When the result 2, the result 3, and the result 4 are obtained, the failed machine is specified by the IP address. In the case of the above results 3 and 4, notification of failure information and a recovery method as shown in FIG. 23C is sent to a process execution device such as a client. By monitoring the failure layer, the observer can easily determine the protocol layer that caused the failure.

FIG. 15 shows an embodiment of the failure information notification according to the present invention. This figure shows an embodiment in which failure information is notified to the client when the root cause of the failure is in an upper layer. In the present embodiment, the monitoring process 1 is installed in the server.
5-1, a monitoring / fault information database 15-2, and a fault notification process 15-5, and a plurality of processes monitored by the monitoring process 15-1 are stored in a server or a client or another system connected to the server. Are distributed.

The monitoring process 15 is provided to the client.
A client monitoring process 15-4, which is a resident communication process having a communication function with the client-1, is provided. One of the clients is a "system administrator client" logged in with system administrator authority. (Administrator Client) ".

After the root cause of the fault is classified by the root cause determination means described above, if the root cause is the uppermost layer (middleware layer / application layer), fault information is notified according to the following procedure. The flow of the procedure is shown in FIG.

First, when the monitoring process accesses the database and holds a notification message and information on the action taken to recover the detected failure,
The information is obtained (step 16-1). The data type of the acquired information is "failure result, monitored process name,
Monitored process IP address, action information (recovery method) "
It is.

Next, the monitoring process notifies the failure notification process of the information obtained as described above (step 1).
6-2). The data type to be notified is “failure result, monitored process name, monitored process IP address, handling information (recovery method)”.

Next, the failure notification process issues a failure notification to each client according to the notification data transmitted from the monitoring process (step 16-3). The failure notification process determines whether the client of the notification destination is in the automatic recovery mode (step 16-4), and when the client is in the automatic recovery mode, sends a request for automatic recovery to the monitoring process (step 16-). 5).

The monitoring process that has received the request for automatic recovery from the failure notification process sends a restart instruction to a group of processes including the monitored process and that has a dependency, as described above, and performs recovery processing automatically ( Step 16-
6).

FIG. 17 shows an embodiment of the failure notification according to the present invention when the underlying cause is a lower layer. In the present embodiment, the server includes a monitoring process 17-1, a monitoring / fault information database 17-2, and a fault notification process 17-5, and a plurality of processes monitored by the monitoring process 17-1 are stored in the server or the server. Distributed in clients or other systems connected to

The monitoring process 17 is provided to the client.
A client monitoring process 17-4, which is a resident communication process having a communication function with the client-1, is provided. One of the clients is a "system administrator client" logged in with system administrator authority. (Administrator Client) ".

After the root cause of the fault is classified by the root cause determining means described above, the root cause is determined by the lower layer (physical layer,
If it is in the TCP / IP layer) or the operating system, the failure information is notified according to the following procedure.
The procedure flow is shown in FIG.

First, when the monitoring process accesses the database and stores a notification message and information on the action to recover the detected failure,
The information is obtained (step 18-1). The data type of the acquired information is “failure result, monitored process name, monitored process 1P address, handling information (recovery method)”.

Next, the monitoring process notifies the failure notification process of the information obtained as described above (step 1).
8-2). The data type to be notified is “failure result, monitored process name, monitored process IP address, handling information (recovery method)”. Next, the failure notification process notifies each client of a failure based on the notification data transmitted from the monitoring process (step 18-3).

FIG. 19 shows an embodiment of the present invention in which a client makes a request for handling a failure. In the present embodiment, a monitoring process 19-1 and a monitoring / failure information database 19-2 are provided in the server.
Are distributed in a server or a client or another system connected to the server.

The monitoring process 19 is provided to the client.
A "client monitoring process" 19-4, which is a resident communication process having a communication function with the "-1", is provided. One of the clients is a "system administrator client" who has logged in with system administrator authority. (Administrator Client) ".

There are the following two types of requests for handling a failure from a client. a. The user logs in to the client monitoring process 19-4 with a system administrator authority account and sends a response request for the root cause of the failure (A1). b. A response request for the root cause of the failure is transmitted by the system administrator authority user trigger (A2).

In the above-described embodiment, when the root cause of the failure is notified to the process execution device such as the client, and a response request by the above a or b is received, the monitoring process 1
9-1 refers to the process dependency data (B) held in the database 19-2 as described above, if the root cause is the middleware layer / application layer of the highest layer, and A restart instruction is sent to a group of processes having a dependency relation including (1), and a countermeasure against the failure (C) is performed.

FIG. 20 shows a processing flow of the present invention when a response request is sent from a client. When started by another client trigger (step 20-1), the client monitoring process logs in with a system administrator account (step 20-2), and the client monitoring process sends a response request to the monitoring process (step 20). -3).

When the processing is started by the system administrator authority user trigger (step 20-4), the system administrator authority user sends a response request (step 20-).
5). The monitoring process refers to the process dependency data stored in the database in response to the response request sent in the above step 20-3 or step 20-5, and executes a process group having a dependency including the monitored process. Then, a restart instruction is sent to deal with the failure (step 20-6).

(Supplementary Note 1) In a process system composed of a plurality of processes, a monitoring process for monitoring the operating state of each process distributed to the own device or another device, and a dependency relationship between processes for restart processing are described. The monitoring process, when detecting a failure in any of the processes to be monitored, refers to the database and executes a process restart for a plurality of process groups having a dependency with the process A system for detecting and recovering from a process abnormality, comprising: (Supplementary Note 2) The database holds an upper limit value of the number of process restarts per unit time according to each process type. When the number of restarts of the process to be monitored exceeds the upper limit value, the monitoring process performs the process. The process abnormality detection and recovery system according to claim 1, further comprising a configuration for executing a process restart for a plurality of process groups having a dependency relationship with the monitoring target process. (Supplementary Note 3) The database holds an upper limit value of the process size according to each process type, and the monitoring process determines a dependency with the process when the process size of the process to be monitored exceeds the upper limit value. The process abnormality detection and recovery system according to claim 1, further comprising a configuration for causing the plurality of process groups to execute the process restart. (Supplementary Note 4) The database holds an upper limit value of a process use memory area according to each process type, and the monitoring process performs the process when the process use memory area of the process to be monitored exceeds the upper limit value. 3. The process abnormality detection and recovery system according to claim 1, further comprising a configuration for executing a process restart for a plurality of process groups having a dependency relationship with the process. (Supplementary Note 5) The database holds a lower limit value of a work memory area for each device that executes each process, and the monitoring process determines that a work memory area of a device that executes a process to be monitored falls below the lower limit value. The process abnormality detection and recovery system according to claim 1, wherein the system is configured to execute a process restart for a plurality of process groups having a dependency relationship with the process. (Supplementary Note 6) The monitoring process periodically monitors the operation state of each process, and when a monitoring request is sent to the monitoring process through a communication process by a login of a system administrator, the monitoring process is activated. Supplementary notes 1 to 5 characterized by comprising a configuration for monitoring a state.
A process abnormality detection and recovery system according to any one of the above. (Supplementary Note 7) The process group monitored by the monitoring process includes communication means via distributed processing middleware, is distributed and arranged on a plurality of processors or operating systems, and the monitoring process includes a plurality of protocols in the communication means. The database has fault detecting means that spans the stack, the database holds fault classification information and fault isolation information, and the monitoring process identifies a fault across protocol layers and stores the fault in the database. Means for extracting alarm information of the protocol layer that caused the failure for a large number of pieces of alarm information generated at the time of occurrence of the failure, based on the obtained information, the alarm information that caused the failure 7. The abnormality according to any one of supplementary notes 1 to 6, characterized in that a recovery process and a failure notification are provided based on the Detection and recovery system. (Supplementary Note 8) The database holds coping information corresponding to the cause of the fault, and the monitoring process acquires the coping information of the cause of the fault, and then, by the fault notifying process, classifies the fault, causes the fault, and the like. In addition to reporting the handling information to the device that executes each process, if the root cause of the failure is in a process such as an application or middleware of an upper protocol layer, the process is performed for a plurality of process groups having a dependency with the process. The process according to claim 7, further comprising the step of, when the restart is executed and the underlying cause of the failure exists in the lower protocol layer or the operating system, displaying the failure information including the handling information on the device executing the process. Anomaly detection and recovery system. (Supplementary Note 9) When a restart request is sent to the monitoring process through a communication process by logging in with a system administrator account, the monitoring process executes a process restart for a plurality of processes having a dependency with the process. The process abnormality detection and recovery system according to claim 8, further comprising a configuration for causing the process abnormality to be detected.

[0089]

As described above, according to the present invention,
Refers to the database that holds the process dependencies,
By providing a monitoring process that executes process restart for a plurality of processes that have a dependency relationship with the monitored process, it is possible to automatically and quickly perform complicated recovery processing that would otherwise be performed manually. Can be.

Further, by identifying a failure across a plurality of protocol layers in inter-process communication, only the information that caused the failure is notified to the monitor, and based on the information that caused the failure. By performing the recovery process, the system recovery time can be reduced.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an embodiment of the present invention including a database that holds a dependency relationship regarding restart between processes.

FIG. 2 is a flowchart of a restart process of a process group having a dependency according to the present invention.

FIG. 3 is a diagram showing an embodiment of the present invention including a database holding an upper limit value of the number of process restarts per unit time.

FIG. 4 is a flowchart of a restart process in which the number of process restarts exceeds an upper limit value according to the present invention.

FIG. 5 is a diagram showing an embodiment of the present invention including a database holding an upper limit value of a process size.

FIG. 6 is a flowchart of a restart process that exceeds the upper limit of the process size according to the present invention.

FIG. 7 is a diagram showing an embodiment of the present invention including a database holding an upper limit value of a process use memory area.

FIG. 8 is a flowchart of a restart process in which a process used memory area exceeds an upper limit value according to the present invention.

FIG. 9 is a diagram showing an embodiment of the present invention including a database holding a lower limit value of a memory work area (swap area).

FIG. 10 is a flow chart of a resuming process in which the memory work area (swap area) falls below a lower limit according to the present invention.

FIG. 11 is a diagram showing an embodiment of the present invention that performs regular monitoring and irregular monitoring by a system administrator.

FIG. 12 illustrates an embodiment of the present invention for detecting the underlying cause of a failure.

FIG. 13 is a diagram showing a flow of monitoring execution for detecting an underlying cause of the present invention.

FIG. 14 is a diagram showing an underlying cause determination logic of the present invention.

FIG. 15 is a diagram showing an embodiment of a failure information notification and recovery process according to the present invention.

FIG. 16 is a diagram showing a flow of failure information notification and recovery processing according to the present invention.

FIG. 17 is a diagram showing an embodiment of a failure notification according to the present invention when the underlying cause is a lower layer.

FIG. 18 is a diagram showing a flow of failure information notification according to the present invention when the underlying cause is a lower layer.

FIG. 19 is a diagram illustrating an embodiment of the present invention in which a client makes a request for handling a failure.

FIG. 20 is a processing flowchart of the present invention when a response request is sent from a client.

FIG. 21 is a diagram illustrating an example of a database that holds an upper limit value of the number of process restarts per unit time.

FIG. 22 is a diagram illustrating an example of a database that holds a process size, an upper limit value of a used memory area, and the like.

FIG. 23 is a diagram illustrating an example of a database of detection of a root cause of a failure and a notification message thereof.

[Explanation of symbols]

 1-1 monitoring process 1-2 monitoring / failure information database 1-3 monitoring target process 1-4 client monitoring process

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B027 AA04 BB01 CC04 5B045 DD18 GG06 JJ02 JJ14 JJ16 JJ42 5B098 AA10 GA04 JJ02 JJ08

Claims (5)

[Claims] In a process system composed of a plurality of processes, a monitoring process for monitoring an operation state of each process distributed to its own device or another device, and a dependency related to a restart process between each process are held. And a database, wherein the monitoring process refers to the database when detecting a failure in any of the processes to be monitored,
A process abnormality detection and recovery system, comprising: a process for executing a process restart for a plurality of process groups having a dependency relationship with the process. 2. The method according to claim 1, wherein the database holds an upper limit value of the number of process restarts per unit time according to each process type, and the monitoring process performs a process when the number of restarts of the process to be monitored exceeds the upper limit value. 2. The system according to claim 1, wherein a process restart is executed for a plurality of process groups having a dependency relationship with the monitoring target process.
Process abnormality detection and recovery system described. 3. The database holds an upper limit value of a process size according to each process type, and the monitoring process determines whether the monitoring process has a dependency when the process size of the monitoring target process exceeds the upper limit value. 2. The process abnormality detection and recovery system according to claim 1, further comprising a configuration for causing a plurality of process groups having a process to execute a process restart. 4. The database holds an upper limit value of a process use memory area according to each process type, and the monitoring process, when the process use memory area of the process to be monitored exceeds the upper limit value, 2. The process abnormality detection and recovery system according to claim 1, further comprising a configuration for executing a process restart for a plurality of process groups having a dependency relationship with the process. 5. The database holds a lower limit value of a work memory area for each device that executes each process, and the monitoring process determines that a work memory area of a device that executes a process to be monitored falls below the lower limit value. 2. The process abnormality detection and recovery system according to claim 1, further comprising a step of executing a process restart for a plurality of processes having a dependency relationship with the process.