JP2003345629A - System monitor device, system monitoring method used for the same, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system monitor device which can provide detailed materials needed to prevent a system fault to possibly be caused by process illegality and investigate the basic cause of the process illegality. <P>SOLUTION: An operation information acquiring means 11 acquires system operation state data at fixed intervals and outputs them to an operation data file 12 in time series. An operation style recognizing means 13 initially generates and updates a comparison value table 14 according to the operation data file 12 and informs a system fault preventing means 15 that the current system resource rate exceeds reference data 141 in such a case. The system fault preventing means 15 specifies a process showing symptoms of illegal processing according to the operation data 142, collects detailed data regarding the object process, and takes emergency measures for preventing the system fault while outputting the detailed data to a detailed data file 16. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system monitoring apparatus, a system monitoring method used therefor, and a program therefor. More particularly, the present invention relates to a method for preventing a system failure and investigating a root cause in a computer system of a communication industry.

[0002]

2. Description of the Related Art In recent years, in information processing systems, 24 hours / 3
The number of systems that require stable operation on the 65th is rapidly increasing. Usually, a technology for monitoring the operating status of the system in real time is applied to such important systems in order to maintain stable operation. As this real-time monitoring technique, there is a technique disclosed in, for example, JP-A-9-293004 and JP-A-4-344544.

The above-mentioned technology generally has the following functions, ie, system operation status data at regular intervals [system resource usage rate, process execution information, error correction count of H / W (hardware), etc.]. And edit / visualize in real time (graphs, etc.), alert / report alerts (to system administrator) when abnormalities (set thresholds) are detected for each item, and collect It is configured with a function of saving the operation status data as a log (for future prediction / improvement proposal data).

[0004]

The conventional real-time monitoring technology described above is currently focused on centralized management (reduction of operators) and provision of real-time visualization data to system administrators. There are the following problems.

[0005] The first problem is that the burden on the system administrator is considerably large. This means that if a warning message is received (even if an outline of the countermeasures is given), the system administrator must take prompt and accurate actions (collecting detailed information / emergency measures, etc.) In some cases, the condition may deteriorate significantly, or the system may hang and become inoperable.) This is because, for a period of time, or for a certain period after the occurrence of a serious failure, it may be required to check the system operation status graph at all times in order to catch signs of abnormality.

[0006] The second problem is that a threshold value for abnormality detection is set in advance. Threshold is lower (safe side)
If it is set to, too many warnings will be reported, so it will naturally be set to an upper value (a value closer to the limit value) in order to narrow down the trigger for reporting as much as possible. Increases). In addition, when the operation mode changes, sufficient review work is required.

[0007] The third problem is that it is often difficult to determine the root cause that triggered a warning state when a system stabilization action is taken in response to a warning message (system operation status). Investigation into the past can identify the time at which the sign was observed / the process that triggered the event, but the root cause could not be determined due to lack of detailed data at that time ) That is. In such a case, a simulation environment of the customer is constructed and a reproducibility test is performed on another computer system (evaluation system of the development department, etc.). ).

Accordingly, an object of the present invention is to solve the above-mentioned problems, prevent a system failure caused by a process illegality from occurring, and obtain detailed materials necessary for investigating the root cause of the process illegality. To provide a system monitoring apparatus, a system monitoring method used therefor, and a program therefor.

[0009]

A system monitoring apparatus according to the present invention is a system monitoring apparatus for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data. Means for detecting a sign of the process leading to an illegal operation based on the data obtained, means for collecting detailed data related to the target process at the time when the process detects the sign of the illegal operation, and an illegal operation of the process. Means for performing emergency precautionary measures to curb the resulting deterioration.

Another system monitoring apparatus according to the present invention is a system monitoring apparatus for monitoring a system failure in a computer system having an acquisition function for acquiring system operation state data indicating an operation state in which a plurality of processes are sequentially executed. The operation data file in which the system operation status data acquired at predetermined intervals set in advance by the acquisition function is sequentially entered in time series, and the system resource usage rate in the latest entry read from the operation data file. Means for determining whether or not the system is operating stably by comparing with reference data set in advance, and determining whether or not the system resource usage rate has a tendency to rapidly increase when receiving a negative notification that the system is operating stably Means for performing the above-described processing when a sudden increase in the system resource usage rate is detected. Means for detecting a sign of unauthorized operation of the process and collecting detailed data on the target process, and means for performing an urgent preventive action for preventing deterioration of the state of the system when detecting the sign of illegal operation of the process. It has.

[0011] A system monitoring method according to the present invention is a system monitoring method for monitoring a system fault in a computer system having an acquisition function for acquiring system operation status data, wherein a process is performed based on the data acquired by the acquisition function. Detecting a sign of an illegal operation, collecting detailed data related to the target process at the time when the process detects the sign of the illegal operation, and determining a state deterioration caused by the illegal operation of the process. Performing an emergency preventive action.

Another system monitoring method according to the present invention is a system monitoring method for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed. Sequentially entering the system operation status data acquired at predetermined intervals by the acquisition function into an operation data file in time series; and a system resource usage rate in the latest entry read from the operation data file. Comparing with a reference data set in advance to determine whether or not the system is operating stably, and whether or not the system resource usage rate has a tendency to rapidly increase when receiving a negative notification that the system is operating stably The step of judging and detecting the tendency of the system resource usage rate to increase rapidly. Detecting a sign of an illegal operation of the process at the time of collecting the detailed data on the target process; and Applying step.

A program for a system monitoring method according to the present invention is a program for a system monitoring method for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data. A process of detecting a sign of a process leading to an illegal operation based on the received data, a process of collecting detailed data relating to a target process at a time when the process detects a sign of a illegal operation, and an illegal operation of the process And a process of executing an emergency preventive measure for minimizing the state deterioration caused by the above.

A program for another system monitoring method according to the present invention is a system monitoring method for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed. A program for causing a computer to sequentially enter the system operation status data acquired at predetermined intervals set in advance by the acquisition function into an operation data file in a time-series manner; A process of comparing the system resource usage rate in the entry with reference data set in advance to determine whether or not the system is operating stably; and when receiving a negative notification that the system is operating stably, the system resource usage rate is reduced. Processing for determining whether there is a rapid increase, and the system A process of detecting a sign of the illegal operation of the process when detecting a sudden increase in the source usage rate and collecting detailed data on the target process, and deteriorating the state of the system when detecting a sign of the illegal operation of the process. And performing an emergency preventive action for prevention.

That is, the system monitoring device of the present invention comprises:
Detailed information required to prevent system failures (extreme performance degradation, hang-ups, downs) caused by unauthorized processing of processes (compression of computer resources) in computer systems, and to investigate the root cause of unauthorized processes Is provided.

Generally, a computer system has a function of acquiring system operation status data (system resource usage rate and execution information of each process).
The feature of the present invention is that, based on the data acquired by this standard function, the process automatically detects a sign of an illegal operation, and obtains detailed data (information for finding the root cause) related to the target process at that time. In addition to collecting the information, it is necessary to prevent the deterioration of the state caused by the incorrect operation of the process.

The present invention focuses on the above-mentioned problems, captures a sign that the stable operation of the system is likely to be impeded, irrespective of the predefined threshold value, collects detailed data at that time, and checks the deterioration of the system state. Take emergency measures to prevent this, and then notify the system administrator (after mitigating the urgency).

As a result, in the present invention, the system administrator can perform a calm operation with a margin (elimination of operation errors). That is, the present invention provides a system failure (extreme performance degradation,
Hang-ups and downs) can be prevented beforehand (reducing the burden on the system administrator), and the detailed data (data at the time of the forewarning) required to investigate the root cause of the process improper process Can be provided.

[0019]

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a system monitoring device according to one embodiment of the present invention. FIG.
In the system monitoring device 1, the operation information acquisition unit 11
, Operation data file 12, operation mode recognition means 13
, Comparison value table 14, and system failure prevention means 15
, A detailed data file 16, a countermeasure reporting means 17,
It comprises a display terminal 18 and a recording medium 19.
The system monitoring device 1 is mainly composed of a computer (not shown), and the computer executes the program on the recording medium 19 to realize the above-described units.

The operation information obtaining means 11 obtains system operation status data (system resource usage rate and execution information of each process) indicating the operation status of the system in which a plurality of processes are sequentially executed, and operates in time series. Output to the data file 12.

The comparison value table 14 includes reference data 141 and operation data 142. The reference data 141 is a reference value of the system resource usage rate (index value of stable operation), and the operation data 142 is the latest system operation state data and the previous four times of system operation state data (data of five generations). Hold.

The operation mode recognition means 13 initially creates / updates the comparison value table 14 based on the operation data file 12. When the current system resource usage rate exceeds the reference data 141, the operation mode recognition means 13 notifies the system failure prevention means 15 of the fact.

The system failure prevention means 15 stores the operation data 1
42, a process having a sign of illegal processing is identified, detailed data (processing trace, etc.) relating to the target process is collected and output to the detailed data file 16, and at the same time, a system failure (extreme performance degradation, hang Take emergency preventive measures to prevent ups and downs. Here, the urgent preventive action is to lower the priority of the target process to prevent the deterioration of the system state,
For example, the target process is temporarily stopped.

After the emergency preventive measures have been completed, the system failure preventive means 15 again carefully checks the detailed state of the relevant process. If there is a suspicion of impropriety, the data at the final check, the contents of the emergency measures, etc., are checked. It is added to the detailed data file 16 and the fact is notified to the countermeasure notifying means 17. The response reporting unit 17 outputs the contents of the emergency preventive action, the storage location of the detailed data, and the like to the display terminal 18 to notify the system administrator.

On the other hand, when it is apparently determined that the operation is normal, the system failure prevention means 15 cancels the emergency precautionary action taken, erases the data of the corresponding process from the detailed data file 16, and returns to the operation mode recognition means 13 , An update request for the reference data 141 is issued.

FIGS. 2 and 3 are flowcharts showing the operation of the system monitoring apparatus according to one embodiment of the present invention. The operation of the system monitoring apparatus according to one embodiment of the present invention will be described with reference to FIGS. 2 and 3 are realized by the computer of the system monitoring device 1 executing the program on the recording medium 19.

The operation information acquisition means 11 acquires system operation state data at regular intervals (for example, every two minutes).
Entries are sequentially made in the operation data file 12 in chronological order. Each entry in the operation data file 12 has an acquisition time and a system resource usage rate [CPU (Cen
[tral Processing Unit: central processing unit) usage rate / memory usage rate (for process) / usage rate of each peripheral device], and execution information of each process (CPU usage rate / memory usage / input / output to each peripheral device) , Etc.).

The operation mode recognition means 13 stores the reference data 141
Is appropriately updated (reflecting the operation mode), and the stable operation of the system is determined (when the system resource usage rate in the latest entry read from the operation data file 12 is equal to or less than the reference data 141, the system is in stable operation. Judge that there is).

The operation mode recognition means 13 initializes / updates the reference data 141 based on the system resource usage rate for the past hour accumulated in the operation data file 12 (when 30 minutes have passed since the last update, and The update is performed when an update request is received from the system failure prevention means 15 (steps S1, S3, S5 in FIG. 2). Here, reference data 1
Reference numeral 41 denotes “(average value of past one hour + maximum value) / 2” for each item.

That is, when there is no past data (step S1 in FIG. 2), the operation mode recognition unit 13 waits for one hour (step S2 in FIG. 2), and when there is past data (step S2 in FIG. 2). S1), the reference data 141 is set based on the data of the past hour (step S3 in FIG. 2). Thereafter, the operation mode recognition unit 13 updates the operation data 142 by adding the latest entry (step S4 in FIG. 2).

When receiving the update request from the system failure prevention means 15, the operation form recognition means 13 (step S2 in FIG. 2)
5) Returning to step S3, the reference data 141 is set, and if an update request is not received from the system failure prevention means 15 (step S5 in FIG. 2), it is determined whether the operation is stable (step S6 in FIG. 2).

If it is determined that the operation mode is stable, the operation mode recognizing means 13 waits until the next reference data 141 is acquired (step S7 in FIG. 2). When it is time to update the reference data 141 (step S8 in FIG. 2), step S3 is performed. To set the reference data 141. If it is not time to update the reference data 141 (step S8 in FIG. 2), the operation mode recognition unit 13 returns to step S4 and updates the operation data 142.

On the other hand, if the operation mode recognizing means 13 does not judge that the operation is stable, the fact is notified to the system failure preventing means 15.
(Step S6 in FIG. 2) and the next reference data 141
(Step S7 in FIG. 2), and when it is time to update the reference data 141 (Step S8 in FIG. 2), the flow returns to Step S3 to set the reference data 141. Also,
If it is not time to update the reference data 141 (step S8 in FIG. 2), the operation mode recognition unit 13 returns to step S4 and updates the operation data 142.

When the system failure prevention means 15 receives the notification (denial of stable operation) from the operation form recognition means 13 (step S11 in FIG. 3), it determines whether or not the system resource usage rate is increasing rapidly (short). Spacing, eg 3
At several second intervals, the system resource usage rate is acquired several times, and the difference between the acquired values is determined several times (step S1 in FIG. 3).
2).

Here, in the vicinity where the reference data 141 is exceeded, there is usually a flat increase tendency. Therefore, when a sudden increase in the system resource usage rate is detected, it is concluded that a sign of process irregularity is apparent. And operating data 14
2, illegal signs of a specific process (CPU monopoly, infinite loop, memory leak, start of a large number of processes, etc.) are captured (step S13 in FIG. 3).

The system failure prevention means 15 collects data (processing trace, stack trace) related to the target process (step S14 in FIG. 3), stores it in the detailed data file 6, and at the same time, urgently prevents the system state from deteriorating. Action (low priority of target process,
(Eg, suspension of the target process) (step S1 in FIG. 3).
5). Thereafter, the system failure prevention means 15 determines whether or not the target process has gone out of control based on the collected data and the like (step S16 in FIG. 3).

As a result, when there is a suspicion of a runaway, the system failure prevention means 15 further collects data (memory image of the main part, etc.) relating to the target process and, together with the process identification name, the emergency preventive action contents, and the operation data 142. ,
After storing in the detailed data file 16 (step S in FIG. 3)
17), and notifies the handling notification means 17 (step S in FIG. 3).
18).

On the other hand, if the system failure preventive means 15 clearly determines that the operation is normal, the emergency preventive action taken is canceled (step S19 in FIG. 3), and the data of the corresponding process is deleted from the detailed data file 6. (Step S20 in FIG. 3), reference data 1
A request for updating 41 is made (step S21 in FIG. 3).

In this case, the system failure prevention means 15 detects in this case that the current business load has reached the limit of the H / W processing capacity (CPU utilization is 90%, memory shortage, etc.) 3 step S22), the generation of a new process is suppressed (step S23 in FIG. 3), and the fact is notified to the countermeasure notification means 17 (step S18 in FIG. 3).

The response reporting means 17 displays the process identification name, the contents of the emergency preventive action, the storage location of the detailed data, and the like on the display terminal 1.
8 to notify the system administrator. If necessary, the system administrator further collects data and performs an operation (eg, terminates, restarts, or restores the priority) on the target process.

As described above, in this embodiment, it is possible to prevent a system failure (extreme performance degradation, hang-up, or down) that would be caused by a process illegality (to greatly reduce the burden on the system administrator). At the same time, it is possible to provide detailed data (data at a time when a sign appears) necessary to investigate the root cause of the process fraud. This generally eliminates the need for a reproducibility test that requires enormous man-hours.

[0042]

As described above, the present invention relates to a system monitoring apparatus for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data, the process based on the data acquired by the acquisition function. Detects signs of unauthorized operation, collects detailed data related to the target process at the time the process detects signs of unauthorized operation, and takes urgent precautionary measures to prevent the process from deteriorating. By executing the process, it is possible to prevent a system failure that may be caused by a process fraud, and to provide an effect that detailed data necessary for investigating a root cause of the process fraud can be provided.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a system monitoring device according to one embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of the system monitoring device according to one embodiment of the present invention.

FIG. 3 is a flowchart showing an operation of the system monitoring apparatus according to one embodiment of the present invention.

[Explanation of symbols]

1 System monitoring device 11 Operation information acquisition means 12 Operation data file 13 Operation form recognition means 14 Comparison value table 15 System failure prevention measures 16 Detailed data file 17 Response reporting means 18 Display terminal 19 Recording medium 141 Reference data 142 Operating data

Claims (15)

[Claims] 1. A system monitoring apparatus for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed, the system monitoring device acquiring the system operation status data. Means for detecting a sign of the process leading to an illegal operation based on the data obtained, means for collecting detailed data relating to the target process at the time when the process detects a sign of the illegal operation, and illegal operation of the process. Means for executing an emergency preventive action for preventing a state deterioration caused by the system monitoring apparatus. 2. A system monitoring device for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed, wherein the system monitoring device sets in advance by the acquisition function An operation data file in which the system operation status data acquired at a given interval is sequentially entered in time series, and a system resource usage rate in the latest entry read from the operation data file is compared with preset reference data. Means for determining whether or not the system is in stable operation, and means for determining whether or not the system resource usage rate is in a tendency to rapidly increase upon receiving a negative notification that the system is in stable operation; and Detecting signs of the above-mentioned process malfunction when detecting a rapid increase in the rate A system for collecting detailed data on an elephant process, and a unit for performing an emergency precautionary measure for preventing a deterioration of the state of the system when a sign of an illegal operation of the process is detected. . 3. A means for canceling the emergency precautionary measure when it is determined that the process is operating normally and issuing a request for updating the reference data, and for inputting the request for updating the reference data and for a predetermined period of time 3. The system monitoring apparatus according to claim 2, further comprising: means for updating the reference data based on past data when the time has elapsed. 4. A means for judging whether or not the target process has reached a runaway state based on the detailed data, and further comprising: collecting detailed data on the target process when it is determined that the target process has reached a runaway state. 4. The system monitoring apparatus according to claim 2, further comprising: means for notifying said detailed data by means of the system monitoring apparatus. 5. A means for inhibiting the creation of a new process and notifying of the occurrence when it is determined that the runaway state has been reached and the current business load has reached the limit of the processing capacity of hardware. 5. The method according to claim 4, wherein
A system monitoring device as described. 6. A system monitoring method for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed, the system monitoring method comprising: Detecting a sign of the process leading to an illegal operation based on the received data; collecting detailed data relating to the target process at the time when the process detects the sign of the illegal operation; and Performing an emergency precautionary measure to prevent the resulting condition deterioration. 7. A system monitoring method for monitoring a system failure in a computer system provided with an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed, wherein the system monitoring method is set in advance by the acquisition function. Sequentially entering the system operation status data acquired at a predetermined interval into an operation data file in a time series, and comparing the system resource usage rate in the latest entry read from the operation data file with reference data set in advance. Determining whether or not the system is operating stably, and determining whether or not the system resource usage rate has a tendency to increase rapidly upon receiving a negative notification that the system is operating stably. Sign of illegal operation of the process when detecting a sudden increase in usage rate Detecting detailed information about the target process by detecting the weather, and performing an emergency precautionary measure for preventing a deterioration of the state of the system when detecting a sign of the illegal operation of the process. System monitoring method. 8. A step of canceling the emergency preventive action and making a request for updating the reference data when it is determined that the process is operating normally, and when inputting the request for updating the reference data and for a preset predetermined time. Updating the reference data based on past data when the time has elapsed. 9. The system monitoring method according to claim 7, further comprising: 9. A step of determining whether or not the target process has reached a runaway state based on the detailed data, and further collecting detailed data regarding the target process when it is determined that the target process has reached a runaway state. 9. The system monitoring method according to claim 7, further comprising a step of notifying the detailed data by using the method. 10. A step of, when it is determined that the runaway state has been reached and the current work load has reached the limit of the processing capacity of hardware, suppressing the generation of a new process and notifying the user of the fact. The system monitoring method according to claim 9, wherein: 11. A program for a system monitoring method for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed, wherein the computer has A process of detecting a sign of the process leading to an illegal operation based on the data acquired by the acquisition function, and a process of collecting detailed data relating to the target process at the time when the process detects the sign of the illegal operation; And a process for executing an emergency preventive measure for preventing a state deterioration caused by an illegal operation of the process. 12. A program for a system monitoring method for monitoring a system failure in a computer system having an acquisition function for acquiring system operation status data indicating an operation status in which a plurality of processes are sequentially executed. A process of sequentially entering the system operation status data acquired at predetermined intervals by an acquisition function in an operation data file in time series, and setting a system resource usage rate in the latest entry read from the operation data file in advance A process of determining whether the system is operating stably by comparing with the reference data obtained, and a process of determining whether the system resource usage rate is increasing rapidly when a negative notification that the system is operating stably is received When the sudden increase in the system resource usage rate is detected, A process of detecting a sign of an illegal operation of the process and collecting detailed data on the target process, and a process of performing an urgent preventive measure for preventing a deterioration of the state of the system when the sign of the illegal operation of the process is detected. A program for executing 13. A process for canceling the emergency preventive action when the computer determines that the process is operating normally and issuing a request for updating the reference data, and for inputting the request for updating the reference data and setting the computer in advance And updating the reference data based on past data when a predetermined time period has elapsed.
The program described. 14. A process for judging whether or not the target process has reached a runaway state based on the detailed data, and detailed data relating to the target process when it is determined that the target process has reached a runaway state. 14. The program according to claim 12, wherein the program further executes a process of notifying the detailed data and notifying the detailed data. 15. When the computer is determined to be in the runaway state and the current workload is determined to have reached the limit of the processing capacity of hardware, the generation of a new process is suppressed and the computer is notified of the determination. The program according to claim 14, wherein the program performs a process of performing the following.