JP2010108266A - Failure detection device, failure detection method, failure detection program, wordbook forming device, and failure occurrence analysis device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method of detecting a failure of a computer without using knowledge or log related to a process. <P>SOLUTION: A process monitoring part 2 monitors process information on an OS 21, and acquires monitoring information such as use states of various resources such as a CPU or memory. A rank-group information creation part 3 ranks, for all processes, the use state of the various resources or groups them according to the degree of use state from the monitoring information. A word creation part 4 creates a wordbook used for extracting feature quantity from the obtained rank-group information and process names. A collection part 11 collects the created words, and a failure time zone estimation part 12 extracts a characteristic keyword from the collected words, and estimates a faulty computer and a time zone of the failure occurrence. A display part 13 displays the faulty computer and the time zone. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a failure detection apparatus, a failure detection method, and a failure detection program, and in particular, a failure detection that includes a series of computers that execute the same job and that are configured from the same hardware and software such as a cluster system. The present invention relates to a device, a failure detection method, a failure detection program, a word collection creation device, and a failure occurrence analysis device.

  Conventionally, several efficient failure detection methods have been proposed that are based on a computer group configured by the same hardware and software and represented by a cluster system and executing the same job.

  In the failure detection device disclosed in Patent Document 1, all-to-all monitoring is performed by configuring a tree structure between computers that are the target of failure detection and monitoring each other for failures between nodes connected in the tree structure. Compared to, the number of computers to be monitored per computer is reduced, thereby efficiently detecting failures.

Further, in the failure detection device disclosed in Non-Patent Document 1, a node indicating an abnormal feature amount is obtained by applying the syslog of each node constituting the cluster to the feature amount algorithm and comparing the feature amount for each node. Presume that a failure has occurred. In this method, since the failure occurrence time zone can be detected only by mechanically processing the syslog, special knowledge about the process such as process behavior and characteristics and prior learning are not required for failure detection.
JP 2003-271471 A Bad Words: Finding Faults In Spirit's Syslogs, Cluster Computing and the Grid, 2008. CCGRID '08. 8th IEEE International Symposium on (2008), pp. 765-770.

  As described above, in the conventional failure detection apparatus described above, a method of detecting a failure between computers connected in a tree structure as in Patent Document 1 or a feature value of a syslog as in Non-Patent Document 1 is used. There was a method to estimate the failed computer and time zone by comparing by time zone.

  According to the method of Patent Document 1, the number of computers that must be monitored by one computer is smaller in the monitoring by the tree structure than in the case where the computers monitor all-to-all. For this reason, the load by fault detection is reduced. However, knowledge of the normal behavior of the process is essential to determine the failure. That is, there is a problem that it takes time and effort to create a knowledge database by learning or the like in advance for fault detection.

  In addition, according to the method of Non-Patent Document 1, since failure detection is performed using only the feature value of the syslog, it is not necessary to create a knowledge database about the process in advance. However, there is a problem that a process that can detect a failure by this method is limited to a process that outputs a log to a syslog such as a service process (daemon).

  The present invention has been made in consideration of such circumstances, and its purpose is to provide a failure detection device, a failure detection method, and a failure detection device capable of detecting a failure of a computer without using knowledge about a process or a log. An object of the present invention is to provide a failure detection program, a word collection creation device, and a failure occurrence analysis device.

  In order to solve the above-described problem, the present invention monitors a process executed in each of a plurality of computers, and obtains a resource usage status for each process and process monitoring information, and the process Rank / group information generating means for ranking or grouping for each process based on the resource usage status and monitoring information acquired by the monitoring means, and for each process obtained by the rank / group information generating means Based on the ranking information indicating the ranking, group information indicating the grouping, and the monitoring information, a word generating means for generating a word used for feature quantity extraction for each process, and the word generating means Collecting means for collecting words for each of the plurality of computers, and the plurality of words collected by the collecting means Based on the word each process computer, a failure detection device, characterized in that it comprises a failure time zone estimating means for estimating the computer and the failure occurrence time slot of a failure.

  In order to solve the above-described problem, the present invention monitors a process executed in each of a plurality of computers, acquires a resource usage status for each process, and monitoring information of the process, and the acquisition A step of ranking or grouping for each process based on the resource usage status and the monitoring information, the ranking information indicating the ranking for each process, the group information indicating the grouping, and the monitoring information Based on the above, a step of generating a word used for feature amount extraction for each process, a step of collecting the generated word for each of the plurality of computers, and a word for each of the collected processes of the plurality of computers And a step of estimating a failure occurrence time zone and a failure occurrence time zone. It is the law.

  Further, in order to solve the above-described problem, the present invention is executed on each of the plurality of computers by the computer of the failure detection apparatus that monitors the process executed on each of the plurality of computers and detects the occurrence of the failure. Monitoring the process, obtaining resource usage status for each process and process monitoring information, and ranking or grouping for each process based on the obtained resource usage status and monitoring information. Generating a word used for feature amount extraction for each process based on rank information indicating ranking for each process, group information indicating grouping, and the monitoring information, and The collected words for each of the plurality of computers, and the collected words for each process of the plurality of computers. Zui by a failure detection program for causing and a step of estimating a fault calculator and failure time zone has occurred.

  In order to solve the above-described problems, the present invention monitors a process executed by a computer, and obtains a resource usage status for each process and process monitoring information, and the process monitoring unit. Ranking / group information generating means for ranking or grouping for each process based on the resource usage and monitoring information obtained by the above, and ranking for each process obtained by the ranking / group information generating means A word collection creation comprising: word generation means for generating words used for feature quantity extraction for each process based on rank information indicating attachment, group information indicating grouping, and the monitoring information Device.

  Further, in order to solve the above-described problems, the present invention is obtained by ranking or grouping for each process based on the resource usage status for each process executed by the computer and the process monitoring information. The word used for the feature quantity extraction for each process generated based on the ranking information indicating the ranking for each process and the group information indicating the grouping is collected for each of a plurality of computers. And a failure time zone estimation means for estimating a failure occurrence computer and a failure occurrence time zone based on words for each process of the plurality of computers collected by the collection device. This is a characteristic failure analysis apparatus.

  According to the present invention, there is an advantage that it is possible to detect a computer failure without using special knowledge about the process and without using a log.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

A. First Embodiment First, a first embodiment of the present invention will be described.
FIG. 1 is a block diagram showing the configuration of the failure detection apparatus according to the first embodiment of the present invention. In the figure, the failure detection apparatus includes a word collection creation unit 1 and a failure occurrence analysis unit 10. The word collection creation unit 1 includes a process monitoring unit 2, a rank / group information generation unit 3, and a word generation unit 4. The failure occurrence analysis unit 10 includes a collection unit 11, a failure time zone estimation unit 12, and a display unit 13.

  The word collection creation unit 1 operates on all the computers 20-1, 20-2, and 20-3 that are subject to failure detection, and displays the operation status of the processes that operate on the computers 20-1 to 20-3. Generate a collection of words to represent. The failure occurrence analysis unit 10 periodically collects word collections from all the computers 20-1 to 20-3, and analyzes the computer in which the failure has occurred and its time zone.

  The process monitoring unit 2 monitors information on processes operating on the OS 21 and acquires monitoring information such as usage statuses of various resources such as a CPU and a memory, and an execution user ID and an execution group ID of the process. The rank / group information generation unit 3 ranks and groups processes for all processes operating on the OS 21 according to the usage status of various resources or monitoring information. The word generation unit 4 generates a word used for feature amount extraction from the obtained rank / group information, execution user ID, execution group ID, and process name.

  The collection unit 11 collects the words generated by the word generation unit 4 from all the computers 20-1 to 20-3. The failure time zone estimation unit 12 extracts characteristic keywords from the collected words and estimates the computer and time zone in which the failure occurred. The display unit 13 displays the estimated time zone to the user.

  FIG. 2 is a diagram illustrating an example of process information monitored by the process monitoring unit 2. The process monitoring unit 2 described above monitors various resource usage statuses of processes operating on the OS 21, process execution user IDs, and execution group IDs through an interface provided by the OS 21. Examples of resources to be monitored include a CPU and a memory usage rate. Examples of interfaces provided by the OS 21 include a / proc file system and a system call.

  FIGS. 3 and 4 are diagrams showing an example of ranking and grouping by the rank / group information generation unit 3. The rank / group information generation unit 3 acquires various resource usage statuses of all processes operating on the OS 21 through the process monitoring unit 2, and ranks or groups each resource. When the resource usage status of the process is as shown in FIG. 2, the ranking of the CPU resource usage status is as shown in FIG. In addition, by appropriately determining thresholds such as “high (30% or more)”, “mid (10% or more and less than 30%)”, “low (less than 10%)” depending on the degree of CPU resource usage rate, They can be grouped as shown in FIG.

  The word generation unit 4 connects the process name to the rank obtained by the rank / group information generation unit 3, the group information, the execution user ID of the process, the execution group ID, and the like by using appropriate delimiters. A word collection representing information of processes operating in 20-1 to 20-3 is generated. For example, when the word collection is created from the CPU usage rate of FIG. 3, since the process “firefox” has the first CPU usage rate, if the delimiter is “:”, the generated word is firefox: cpu1. It is. Similarly, the words emacs: cpu2, xterm: cpu3 are generated, and finally a word collection consisting of three words {firefox: cpu1, emacs: cpu2, xterm: cpu3} is generated.

  Further, when the group information of FIG. 4 is used, since the process “firefox” has the CPU group “high”, the generated word is “firefox: cpuhigh”. Similarly, the words emacs: cpuhigh and xterm: cpulow are generated, and finally a word collection consisting of three words {firefox: cpuhigh, emacs: cpuhigh, xterm: cpulow} is generated.

  Although not shown in the figure, when a word collection is created from the execution user ID, the execution user ID of the process “firefox” is yashitho, and thus the generated word is firefox: yasuito. Similarly, the words emacs: yasuhito and xterm: yutaro are generated, and finally a word collection consisting of three words {firefox: yasuhito, emacs: yasuhito, xterm: yutaro} is generated.

  The collection unit 11 collects a word collection from all the computers 20-1 to 20-3. In addition, the ID (host name or the like) of the computer that generated the word collection and the collected time zone information are added for each word collection. The width of the time zone may be arbitrarily set such as 1 hour or 15 minutes.

  The failure time zone estimation unit 12 uses the existing IDF method or the log. Keyword extraction is performed using the entropy method. In the keyword extraction, for a word i appearing in the collected word collection, an index g (i) indicating how much the word i is characteristic is obtained. Next, using the obtained index g (i), the period for which the failure is to be detected is weighted for each certain time period to obtain a characteristic value for each time period. A time zone with a large feature value is abnormal as compared to other time zones, and it is assumed that a failure has occurred.

  In general, as a feature of a word that tends to increase the feature value, it can be expressed by only a few computers in the whole. For example, a characteristic word is generated from a process that is executed only on a specific computer, a process that shows an abnormal CPU usage rate, or a process that has a different execution user ID from other computers. Easy to show value.

  The failure time zone estimation unit 12 creates the following sparse matrix M × N matrix as preprocessing. If the element of the M × N matrix is x (i, j), x (i, j) is defined as “the number of times word i appears in node time zone j”. Here, the node time zone is an ID that uniquely defines a time zone for each computer. That is, the number of columns N = (number of nodes) × (number of time zones). After generating the M × N matrix, the IDF method, or log. The index g (i) is calculated as follows using the entropy method.

  Hereinafter, a keyword extraction method when the IDF method is used will be described. In the IDF method, the index g (i) is obtained by the following equation (1).

Here, n is the number of all node time zones, and df _i is the number of node time zones in which the word i appears. When the obtained g (i) is used for weighting, the characteristic value | x _j | of the node time zone j is obtained by the following equation (2).

  As a feature of the IDF method, the importance of words appearing in many node time zones decreases, and the importance of words that appear only in specific node time zones increases.

  Next, log. A keyword extraction method when the entropy method is used will be described. log. In the entropy method, g (i) is obtained by the following equation (3).

Here, p _ij is the ratio of the number of times the word i has appeared in the node time zone j out of the total number of times the word i has appeared, and is expressed by the following equation (4).

When the obtained g (i) is used for weighting, the characteristic value | x _j | of the node time zone j is obtained by the following equation (5).

  log. As a feature of the entropy method, the importance of a word that appears the same number of times in all node time zones is “0”, and the importance of a word that appears only in one node time zone is “1”. That is, the weight of a word that appears biased in a specific node time zone approaches “1”.

  When the failure time zone estimation unit 12 detects a feature value equal to or greater than a certain threshold, the display unit 13 displays the computer and the time zone to the user. The display method may be any method that allows the user to recognize the result, such as displaying on a display, outputting to a file, sending an e-mail, or sending the contents to another program.

Next, the operation of the first embodiment will be described.
FIG. 5 is a flowchart for explaining the operation of the word collection creation unit 1 according to the first embodiment. First, in step Sa1, the process monitoring unit 2 monitors and acquires various resource usage statuses of processes operating on the OS 21 through an interface provided by the OS 21 (step Sa1).

  Next, the rank / group information generation unit 3 acquires various resource usage statuses of all processes operating on the OS 21 through the process monitoring unit 2, and ranks or groups each resource (step Sa2). . Next, the word generation unit 4 connects the process name and the rank obtained by the rank / group information generation unit 3 or group information with an appropriate delimiter to represent a word representing the information of the process operating on the computer A collection is generated (step Sa3).

  Next, FIG. 6 is a flowchart for explaining the operation of the failure occurrence analysis unit 10 according to the first embodiment. The collection unit 11 is activated at regular intervals and collects the word collections generated by the word collection creation unit 1 of all computers that are fault detection targets (step Sb1). Next, the failure time estimation unit 12 applies the IDF method or the log. The entropy method is used to estimate the failed computer and the time zone (step Sb2).

  Next, the failure time estimation unit 12 determines whether or not a failure has been found (step Sb3). If a failure is detected, the display unit 13 displays the computer and the time zone in which the failure has occurred to the user. Are displayed (step Sb4). Then, it returns to step Sb1 and repeats the process mentioned above. On the other hand, if no failure is detected in step Sb3, the process returns to step Sb1 and the above-described processing is repeated.

  According to the first embodiment described above, it is possible to detect a failure using only process information obtained from the OS without using a log output by the process. For this reason, it becomes possible to detect a failure for a type of process that does not output a log, such as most user processes. In addition, since a knowledge database for fault detection is not required, it is not necessary to create a knowledge database in advance, and it is possible to deal with unknown faults.

B. Second Embodiment Next, a second embodiment of the present invention will be described.
FIG. 7 is a block diagram showing the configuration of the failure detection apparatus according to the second embodiment of the present invention. In the figure, the same parts as those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted. In the second embodiment, a program path acquisition unit 5 is provided in addition to the configuration of the first embodiment. Further, a word generation unit 6 is provided instead of the word generation unit 4 in the first embodiment.

  The program path acquisition unit 5 acquires a program path for each process executed on the OS 21 through the interface of the OS 21. For example, the program path of the process “firefox” obtains information such as / usr / bin / firefox, the program path of the process “xterm” obtains information such as / usr / bin / xterm, and the like.

  The word generation unit 6 appropriately separates the program path obtained by the program path acquisition unit 5 from the rank obtained by the rank / group information generation unit 3, group information, process execution user ID, execution group ID, and the like. By connecting with letters, a word collection representing information of processes operating on a computer is generated.

  For example, when the word collection is created from the CPU usage rate of FIG. 3, it is understood that the process “firefox” is the program / usr / bin / firefox executed by the program path acquisition unit 5, and the CPU Since the usage rate is first, when the delimiter is “:”, the generated word is / usr / bin / firefox: cpu1. Similarly, the words / usr / bin / emacs: cpu2, / usr / bin / xterm: cpu3 are generated, and finally, {/ usr / bin / firefox: cpu1, / usr / bin / emacs: cpu2, A word collection consisting of three words / usr / bin / xterm: cpu3} is generated.

Next, the operation of the second embodiment will be described.
FIG. 8 is a flowchart for explaining the operation of the word collection creation unit 1 according to the second embodiment. The process monitoring unit 2 monitors and acquires various resource usage statuses of processes operating on the OS 21 through an interface provided by the OS 21 (step Sc1). Next, the rank / group information generation unit 3 acquires various resource usage statuses of all processes operating on the OS 21 through the process monitoring unit 2, and ranks or groups each resource (step Sc2). .

  Next, the program path acquisition unit 5 acquires a program path for each process executed on the OS 21 through the interface of the OS 21 (step Sc3). The word generation unit 6 connects the program path name obtained by the program path acquisition unit 5 and the rank obtained by the rank / group information generation unit 3 or group information by an appropriate delimiter. A word collection representing the information of the operating process is generated (step Sc4). The generated word collection is passed as an input to the collection unit 11, and the failure is estimated in the same procedure as in the first embodiment.

  According to the second embodiment described above, the generation of words is performed using the program path that generated the process, not the process name, so that, for example, the case where the program path is different even with the same process name can be distinguished. it can. For this reason, the case where the program camouflaged by the malicious user is executed can be reflected in the failure detection result. For this reason, it is possible to further improve the accuracy of faults that can be detected in the first embodiment.

  The failure detection apparatus of the present invention can be applied to a computer group such as a personal computer, a workstation, and a server having almost the same software and hardware configuration.

It is a block diagram which shows the structure of the failure detection apparatus by 1st Embodiment of this invention. It is a figure which shows an example of the process information monitored by the process monitoring part. It is a figure which shows an example of the ranking by the rank and group information generation part 3, and grouping. It is a figure which shows an example of the ranking by the rank and group information generation part 3, and grouping. It is a flowchart for demonstrating operation | movement of the word collection preparation part 1 by this 1st Embodiment. It is a flowchart for demonstrating operation | movement of the failure generation analysis part 10 by this 1st Embodiment. It is a block diagram which shows the structure of the failure detection apparatus by 2nd Embodiment of this invention. It is a flowchart for demonstrating operation | movement of the word collection preparation part 1 by this 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Word collection preparation part 2 Process monitoring part 3 Order | ranking / group information generation part 4 Word generation part 5 Program path acquisition part 6 Word generation part 10 Failure occurrence analysis part 11 Collection part 12 Failure time estimation part 13 Display part 20-1-20 -3 Computer 21 OS

Claims (13)

A process monitoring means for monitoring a process executed by each of a plurality of computers, and obtaining a resource usage status for each process and process monitoring information;
Rank / group information generating means for ranking or grouping for each process based on the resource usage and monitoring information acquired by the process monitoring means;
Generates words to be used for feature value extraction for each process based on the rank information indicating the ranking for each process, the group information indicating the grouping, and the monitoring information obtained by the rank / group information generating means. Word generating means for
Collecting means for collecting the words generated by the word generating means for each of the plurality of computers;
Failure detection, comprising: a failure time zone estimation means for estimating a failure occurrence time zone and a failure occurrence time zone based on the words for each process of the plurality of computers collected by the collection means apparatus.   The word generation unit generates a word for each process by connecting a process name and rank information, or a process name and group information, or a process name and monitoring information with a predetermined delimiter. Item 5. The fault detection apparatus according to Item 1. A program path acquisition means for acquiring a program path for each process executed by each of a plurality of computers,
The word generating means
The failure according to claim 1, wherein a word used for feature quantity extraction for each process is generated based on the program path acquired by the program path acquisition unit, rank information, group information, and monitoring information. Detection device.   The word generation unit generates a word for each process by connecting a program path and rank information, or a program path and group information, or a program path and monitoring information with a predetermined delimiter. Item 5. The fault detection apparatus according to Item 3.   The failure time zone estimation means obtains an index indicating how characteristic the words for each process of the plurality of computers collected by the collection means, and for each predetermined time zone based on the index And calculating a failure occurrence computer, a failure occurrence computer, and a failure occurrence time zone based on the feature value in each time zone. The fault detection apparatus in any one.   The failure time zone estimation means obtains a feature value for each predetermined time zone by weighting a period for failure detection for each predetermined time zone using the index. The failure detection apparatus according to claim 5. Monitoring a process executed on each of a plurality of computers, obtaining resource usage for each process, and process monitoring information;
Based on the obtained resource usage status and monitoring information, ranking or grouping for each process;
Generating a word to be used for feature value extraction for each process based on rank information indicating ranking for each process, group information indicating grouping, and the monitoring information;
Collecting the generated words for each of the plurality of computers;
A failure detection method comprising: estimating a computer in which a failure has occurred and a failure occurrence time zone based on the collected words for each process of the plurality of computers. In the computer of the failure detection device that detects the occurrence of a failure by monitoring the processes executed on each of the plurality of computers,
Monitoring a process executed on each of a plurality of computers, obtaining resource usage for each process, and process monitoring information;
Based on the obtained resource usage status and monitoring information, ranking or grouping for each process;
Generating a word to be used for feature value extraction for each process based on rank information indicating ranking for each process, group information indicating grouping, and the monitoring information;
Collecting the generated words for each of the plurality of computers;
A failure detection program that executes a step of estimating a computer in which a failure has occurred and a failure occurrence time zone based on the collected words for each process of the plurality of computers. A process monitoring means for monitoring a process executed by a computer and obtaining resource use status for each process and process monitoring information;
Rank / group information generating means for ranking or grouping for each process based on the resource usage and monitoring information acquired by the process monitoring means;
Generates words to be used for feature value extraction for each process based on the rank information indicating the ranking for each process, the group information indicating the grouping, and the monitoring information obtained by the rank / group information generating means. And a word generation means. A program path acquisition means for acquiring a program path for each process executed by each of a plurality of computers,
The word generating means
10. The word according to claim 9, wherein a word used for feature amount extraction for each process is generated based on the program path acquired by the program path acquisition means, rank information, group information, and monitoring information. Collection device. Rank information indicating the ranking for each process and group obtained by ranking or grouping for each process based on the resource usage status for each process executed by the computer and the process monitoring information, and the group A collection means for collecting, for each of a plurality of computers, a word used for feature amount extraction for each process generated based on the monitoring information and group information indicating division;
A failure occurrence characterized by comprising failure time zone estimation means for estimating a failure occurrence time zone and a failure occurrence time zone based on the words for each process of the plurality of computers collected by the collection means Analysis equipment.   The failure time zone estimation means obtains an index indicating how characteristic the words for each process of the plurality of computers collected by the collection means, and for each predetermined time zone based on the index 12. The failure according to claim 11, wherein a failure occurrence, a computer in which a failure has occurred, and a failure occurrence time zone are estimated based on the feature value of each time zone. Development analysis device.   The failure time zone estimation means obtains a feature value for each predetermined time zone by weighting a period for failure detection for each predetermined time zone using the index. The failure occurrence analysis apparatus according to claim 12.

Images