JP5779548B2 - Information processing system operation management apparatus, operation management method, and operation management program - Google Patents

Description

  More particularly, the present invention relates to an operation management apparatus, an operation management method, and an operation management program for an information processing system that monitors the operation status of the system and detects the occurrence of a system failure. .

  In recent years, as information processing systems have become increasingly important as the foundation of corporate activities and social infrastructure, the demand for information processing systems that combine high processing capacity and high reliability has never been higher. .

  As a state of implementation of such an advanced information processing system, a parallel distributed processing system in which a large number of information processing apparatuses are installed in a data center or the like and cooperatively operate them to achieve the purpose of the system is becoming widespread.

  A problem in the operation and management of such a parallel distributed processing system is detection and response to a failure occurring in the system. Due to the characteristic of operating by the cooperation of a large number of information processing devices, the occurrence of a device failure affects the operation of the entire system. Many systems have tolerance against the occurrence of such a failure, so that the entire system can be prevented from being stopped, but it is still inevitable that performance deteriorates and resource utilization efficiency decreases. Also, as the system becomes larger and the number of devices used increases, the frequency of device failures increases to a level that cannot be overlooked.

  As background art related to detection of such a failure occurrence, for example, in Patent Document 1, “a performance operation management device that monitors the performance of an information processing system in which a plurality of information processing devices operate in cooperation with each other, Based on the monitoring status of the operating status of the apparatus and the data communication status of each communication line connecting the plurality of information processing apparatuses, and the monitoring data by the monitoring means, An apparatus for “detecting a fault” is disclosed.

  Further, Non-Patent Document 1 discloses a method of extracting a correlation of measurement data between a plurality of instances in which a certain application operates in a virtualized system, and detecting the occurrence of a failure by a decrease in the correlation. Yes.

JP 2005-327261 A

Hui Kang, Haifeng Chen, and Guofei Jiang. 2010, PeerWatch: a fault detection and diagnosis tool for virtualized consolidation systems, In Proceeding of the 7th international conference on Autonomic computing (ICAC '10) .ACM, New York, NY, USA, 119-128.

  Now, the reason why the above-mentioned parallel distributed processing system has become widespread is that a large amount of information processing devices can be installed and operated due to the low cost of the information processing devices, and the unit performance of these devices has been remarkably improved. In addition, a programming technique has been developed that makes it easy to describe the software required to make a group of such information processing devices work together as a group (referred to as a cluster) and to use it as a system for any purpose. It can be mentioned.

  One such technique is the MapReduce method. In the MapReduce method, a job is divided into a large number of tasks, and each task is distributed and executed in a large number of information processing apparatuses constituting a cluster. Therefore, a significant reduction in execution time can be expected due to a parallel effect. In addition, the task is largely divided into two types, a Map task and a Reduce task, so that interdependencies of data to be processed are eliminated between a plurality of Map tasks or a plurality of Reduce tasks. It is also characterized by simplifying task scheduling, eliminating the need for explicit programming of synchronization between processes. By utilizing this MapReduce method, a wide variety of application systems utilizing parallel distributed processing can be realized. For example, human flow analysis using data obtained from the use history of electronic tickets in public transportation, transmission and distribution networks, etc. Applications such as power demand analysis using power consumption data obtained from sensors installed above are conceivable.

  As described above, in the parallel distributed processing system, the occurrence of a failure in the information processing apparatus affects the operation of the entire system. The MapReduce method is also effective as a countermeasure against such a situation. Suppose that one of the information processing devices that make up a cluster has failed, and the task that was being executed on that device has not ended normally. Even in that case, the execution of the entire job can be completed without stopping by re-execution of the same task in another apparatus. This is because, in the MapReduce method, the interdependence between tasks is kept to a minimum, and even if one task is re-executed, the influence on other tasks is extremely small.

  However, even in the MapReduce cluster having such characteristics, the occurrence of a failure cannot be completely ignored. Although the interdependence between tasks is minimized, re-execution of tasks delays the execution of the entire job.

  In addition, depending on a certain type of failure, even if the task does not end abnormally, a situation may occur in which the execution is delayed from what is originally expected. This is due to, for example, the occurrence of a phenomenon such as thrashing in the information processing apparatus. In such a case, the MapReduce method recognizes that such a task execution delay has occurred, and starts executing the same task in another device. Such processing is called speculative execution, and a task to be started is called a backup task. Then, the output of the task that has been normally executed earlier is adopted as the processing result, and the task delayed from the task is forcibly terminated.

  Although such a backup task method has a certain effectiveness, the execution of the entire job is still delayed.

  In addition, an information processing apparatus in which such a failure occurs frequently, whether it is an abnormal end of a task or an execution delay, causes a waste of computational resources, and is required to be repaired and replaced as soon as possible.

  Therefore, a failure detection method for a parallel distributed processing system adopting the MapReduce method is required, but the known method is not necessarily sufficient.

  For example, the invention disclosed in Patent Document 1 discloses a technique for detecting the occurrence of a failure by acquiring a plurality of types of monitoring data from a plurality of information processing apparatuses and calculating a correlation therefrom. ing. However, it does not provide guidance on how to select monitoring data from which such correlations should be extracted. The monitoring data exemplified in this document correlates to any person who has a certain knowledge about the configuration and dynamics of the system to be monitored, such as the transaction throughput and disk I / O amount in the DB server. In other words, in order to execute fault monitoring, a priori knowledge of the system is required.

  However, it will be difficult to apply the present invention to the MapReduce parallel distributed processing system as described above. This is because, in the MapReduce method, after dividing a job into tasks, it is not determined which task is to be executed by which information processing apparatus unless it is at the time of execution. Because of this property, the MapReduce method has obtained the advantages of flexibility in task scheduling and improvement in utilization efficiency of computing resources. However, when trying to apply the failure detection technology based on the monitoring data as described above. There is a problem that it is not clear which device the correlation should be calculated.

  In addition, if the advantage of the MapReduce method as a programming technique is that a wide variety of parallel distributed software can be easily constructed, the MapReduce parallel distributed processing system is not limited to a specific number of application systems. It can also be used for various applications. In this case, the jobs executed by the cluster will have different characteristics such as those that use a lot of processor resources and those that use a lot of disk I / O, and the load on the apparatus will also vary. As a result, it is difficult to extract what should be the target of correlation calculation from a large number of monitoring data that can be acquired from the device when trying to apply the failure detection technology by the above approach. There is.

  Furthermore, as the parallel distributed processing system is effectively used, a new information processing apparatus will be additionally introduced in order to expand the scale. As a result, it is considered that the computing resources provided for each of the information processing devices constituting the cluster become non-uniform in an era in which the performance of the device is rapidly advanced as in recent years. This also makes it difficult to apply the failure detection technique based on the above approach.

  That is, in the approach of detecting a failure by analyzing the correlation of monitoring data, it is necessary to answer to the question of which monitoring data of which information processing apparatus should be selected and analyzed as a pair.

  For example, in the technique disclosed in Non-Patent Document 1, a variety of monitoring data is collected as a target of analysis by utilizing a statistical technique called canonical correlation analysis. This method is an example that solves the problem of which one to select from the monitoring data of the apparatus. However, it is not a solution to the problem of determining which devices should be considered to be correlated as described above.

  As described above, the known technology cannot cope with failure detection in a parallel distributed processing system, although it is important. For example, even if failure detection is performed based on monitoring data acquired by each of a plurality of information processing devices, such as a MapReduce cluster, it is not determined in advance which device will have a correlation. It cannot cope with a system in which the combination in which the correlation occurs during system operation changes.

  Therefore, a method, program, apparatus, and system for detecting a failure in a parallel distributed processing system are provided. For example, failure detection is performed regardless of job diversity, non-determinism of job execution scheduling, and changes in the operating system configuration.

  In order to solve the above problems, for example, the configuration described in the claims is adopted.

The present application includes a plurality of means for solving the above-mentioned problems, and an example thereof has the following configuration.
An operation management device of an information processing system that executes a job in cooperation with a plurality of information processing devices, the operation management device including a data collection unit that acquires information from each of the plurality of information processing devices, and a plurality of information processing devices A storage unit that stores data related to the information processing unit, and an evaluation unit that evaluates the states of the plurality of information processing apparatuses using the data stored in the storage unit. Each of the plurality of information processing devices has any one of predetermined characteristics.
The data collection unit acquires performance information from each of the plurality of information processing apparatuses and stores the performance information in the storage unit. The storage unit further stores a threshold value regarding the correlation between the performance information of the two information processing devices for each combination of characteristics that the two information processing devices can take. The evaluation unit evaluates the state of the information processing apparatus as the evaluation target for one information processing apparatus as the evaluation target among the plurality of information processing apparatuses, and each of the plurality of information processing apparatuses other than the information processing apparatus as the evaluation target And calculating a correlation value of performance information with the information processing apparatus to be evaluated, specifying a combination of characteristics with the information processing apparatus to be evaluated, and storing the specified combination of characteristics in the storage unit The threshold is compared with the calculated correlation value, and the state of the information processing apparatus to be evaluated is evaluated based on the comparison result.

  Fault detection is possible in a parallel distributed processing system. For example, failure detection is possible even in a parallel distributed processing system in which various jobs are executed.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a figure which shows an example of a structure of information processing apparatus. It is a figure which shows an example of the whole structure of a parallel distributed processing system. It is a figure which shows an example of a structure of a MapReduce cluster. It is a figure which shows an example of the execution flow of the process of MapReduce system. It is a figure which shows an example of the concept of the slot number of a task tracker. It is a figure which shows an example of a structure of the monitoring agent. It is a figure which shows an example of the process sequence of a monitoring agent. It is a figure which shows an example of a structure of the monitoring manager. It is a figure which shows an example of the process sequence of a monitoring agent. It is a figure which shows an example of the table group which the database of the monitoring manager has stored. It is a figure which shows an example of a management object host list table. It is a figure which shows the example of the processor performance information table which is an example of the table which stores OS performance information. It is a figure which shows the example of the memory performance information table which is an example of the table which stores OS performance information. It is a figure which shows the example of the disk performance information table which is an example of the table which stores OS performance information. It is a figure which shows an example of the job list which is an example of MapReduce scheduling information. It is a figure which shows an example of the task list which is an example of MapReduce scheduling information. It is a figure which shows an example of the attempt list | wrist which is an example of MapReduce scheduling information. It is a figure which shows an example of the data transfer trace which is an example of MapReduce scheduling information. It is a figure which shows an example of an operation condition evaluation processing procedure. It is a figure which shows an example of the process sequence of a virtual group production | generation. It is a figure which shows an example of a virtual group table. It is a figure which shows the example of a virtual group node list table. It is a figure which shows an example of the concept of a virtual group. It is a figure which shows an example of the process sequence of node characteristic determination. It is a figure which shows the example of the table used for node characteristic determination. It is a figure which shows the other example of the table used for node characteristic determination. It is a figure which shows an example of the screen display used for the setting of the table used for node characteristic determination. It is a figure which shows an example of the process sequence of cluster map generation. It is a figure which shows an example of a cluster map table. It is a figure which shows an example of the concept of a cluster map. It is a figure which shows an example of the process sequence of node performance matrix production | generation. It is a figure which shows an example of a node performance matrix table. It is a figure which shows an example of the process sequence of correlation calculation. It is a figure which shows an example of a job profile table. It is a figure which shows an example of the process sequence of an event notification. It is a figure which shows an example of the screen display of an event notification. It is a figure which shows an example of the process sequence of a management object host operating condition display. It is a figure which shows an example of the screen display of a monitoring console. It is a figure which shows another example of the screen display of a monitoring console. It is a figure which shows an example of the screen display of a monitoring console. It is a figure which shows another example of the screen display of a monitoring console. It is a figure which shows an example of the whole structure of the parallel distributed processing system by 2nd Embodiment. It is a figure which shows an example of a structure of the monitoring manager and remote monitor by 2nd Embodiment. It is a figure which shows an example of the process sequence of the node performance matrix production | generation by 3rd Embodiment. It is a figure which shows an example of the management object host list table by 3rd Embodiment. It is a figure which shows an example of the job profile table by 4th Embodiment. It is a figure which shows an example of the analysis algorithm table by 4th Embodiment. It is a figure which shows an example of the process sequence of the correlation calculation by 4th Embodiment. It is a figure which shows an example of the screen display used for the analysis algorithm setting by 4th Embodiment. It is a figure which shows an example of the concept of the analysis algorithm automatic determination method by 4th Embodiment. It is a figure which shows another example of the concept of the analysis algorithm automatic determination method by 4th Embodiment. It is a figure which shows an example of the process sequence of the analysis algorithm automatic determination by 4th Embodiment. It is a figure which shows an example of the concept of the failure detection method by 4th Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings described below, the same portions are denoted by the same reference numerals, and repeated description thereof is omitted or simplified.

  First, as a first embodiment, an example of a system operation management apparatus having a failure detection function will be described.

  FIG. 1 is a diagram illustrating an example of the configuration of the information processing apparatus.

  The information processing apparatus 100 includes a processor 101, a memory 102, a storage 103, a network I / F 104, and a console 105. The processor 101 is connected to a memory 102, a storage 103, a network I / F 104, and a console 105. The network I / F 104 is connected to the network 106.

  The information processing apparatus 100 is, for example, a rack mount server, a blade server, a personal computer, or the like. The information processing apparatus 100 may include a plurality of processors 101, a memory 102, a storage 103, a network I / F 104, and a console 105. The storage 103 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like, or a combination of a plurality of these. The network 106 is, for example, an Ethernet (registered trademark) or a wireless network based on the IEEE 802.11 standard.

  The storage 103 can record and read data in a nonvolatile manner. The network I / F 104 can communicate with the network I / F 104 included in the other information processing apparatus 100 via the network 106 to which the network I / F 104 is connected. The console 105 can display text information, graphical information, and the like using a display device, and can receive information from a connected human interface device.

  The information processing apparatus 100 has a user process 200 and an operating system (OS) 210 mounted on a memory 102. The user process 200 and the operating system 210 are both programs and are executed by the processor 101 included in the information processing apparatus 100, whereby the information processing apparatus 100 reads and writes data from and to the memory 102 and the storage 103, and the network I / F 104. Via the network 106, it can communicate with the user process 200 and the operating system 210 installed in the memory 200 of the other information processing apparatus 100, and can display and receive information on the console 105.

  The system operation management apparatus or parallel distributed processing system shown in this embodiment has the same configuration as the information processing apparatus 100 shown in FIG.

  FIG. 2 is a diagram illustrating an example of the overall configuration of the parallel distributed processing system.

  The monitoring server 110, the client 120, the master node 130, and the worker node 140 are all implemented with user processes 200 that are characteristic of the information processing apparatus 100. For example, the monitoring server 110 implements the monitoring manager 201 as a user process. The client 120 implements the monitoring console 202 as a user process. The master node 130 implements a job tracker 203, a name node 204, and a monitoring agent 205 as user processes. The worker node 140 includes a task tracker 206, a data node 207, and a monitoring agent 205 as user processes. Further, these information processing apparatuses can communicate with each other via the network 106.

  The MapReduce cluster 300 includes one master node 130 and one or more worker nodes 140.

  The parallel distributed processing system may include a plurality of monitoring servers 110 and clients 120. A parallel distributed processing system may include a plurality of MapReduce clusters 300.

  The system operation management apparatus according to the present exemplary embodiment includes one or both of a client 120, a monitoring console 202, and a monitoring agent 205 in addition to the monitoring server 110 and the monitoring manager 201. The monitoring server 110 may also serve as the client 120. It should be noted that there is a degree of freedom in the correspondence between the information processing apparatus and the user processes mounted thereon, and this embodiment is an example of many combinations thereof.

  A person in charge of operation management of the parallel distributed processing system monitors the parallel distributed processing system based on information displayed by the monitoring console 202 implemented by the client 120 via the console 105. The monitoring console 202 receives information input by the person in charge of operation via the console 105 and transmits it to the monitoring manager, and the monitoring manager changes the operation based on the information. An example of the interaction between the operation manager and the system operation management apparatus via the monitoring console will be described later.

  FIG. 3 is a diagram illustrating an example of the relationship between the master node 130 and the worker node 140 that configure the MapReduce cluster 300.

  The job tracker 203 of the master node 130 executes the job 301. A job 301 is a set of one or more tasks 302. The task 302 is a state of the user process 200, and the worker node 140 can execute one or more tasks 302. The job tracker 203 communicates with the task tracker 206 of the worker node 140. That is, the job tracker 203 instructs the task tracker 206 to execute the task 302. The main point of the MapReduce method is to improve the processing efficiency by distributing a group of tasks 302 constituting one job 301 to a plurality of worker nodes 140 and executing them in parallel.

  The user of the MapReduce cluster instructs the job tracker 203 to execute the job 301. The instruction may be performed using the console 105 of the master node 130 or may be performed by performing communication via the network 106 from another information processing apparatus 100. The job tracker 203 distributes the group of tasks 302 constituting the job 301 to the task tracker 206 under the management, and the task tracker 206 executes the distributed task 302.

  The MapReduce cluster 300 has a distributed file system function for storing data in the storage 103 of the worker node 140. This stores data required by the task 302 for its processing. The name node 204 of the master node 130 has information (metadata) indicating which worker node 140 stores certain data. A task 302 that requires some data communicates with the name node 204. That is, the task 302 obtains metadata corresponding to the required data from the name node 204, and then communicates with the data node 207 operating on the worker node 140 in which the data is stored, and requests the target data. To do. The data node 207 divides the data into groups of data blocks 303 and stores them in the storage 103, and transfers the requested data to the task 302.

  FIG. 4 is a diagram illustrating an execution flow of the MapReduce type job 301.

  As described above, the job 301 is a set of one or more tasks 302. The task 302 consists of two types, a Map task 305 and a Reduce task 307. The Map task 305 reads the split 304, which is an input file placed in the distributed file system, performs some processing, generates an intermediate file 306, and writes this in the storage 103 of the worker node 140. The intermediate file 306 is a key / value format file, and this key determines which Reduce task 307 the data is input to. That is, the result of the processing by the map task 305 is written in the value of the file 306, and the value specifying the Reduce task 307 to which the value written as the value is input is written in the key. The task tracker 206 transfers data having a specific key from the group of intermediate files 306 to the worker node 140 that executes the Reduce task 307. The task tracker 206 sorts the transferred data and inputs the sorted data to the Reduce task 307. The Reduce task 307 performs some processing on the data, and generates the result as an output file 308 on the distributed file system.

  FIG. 5 is a diagram showing the concept of the number of slots of the task tracker 206.

  The task tracker 206 has a “slot number” setting. This is the upper limit value of the number of tasks simultaneously executed by one task tracker 206 in the worker node 140, and can be set for each of the Map task 305 and the Reduce task 307. The figure shows a state where each slot is set to 4. These values are upper limit values, and the same number of tasks are not always executed. The task tracker 206 sends slot number setting information to the job tracker 203, and the job tracker 203 designates a task to be executed by each task tracker 206 based on the information.

  FIG. 6 is a diagram illustrating an example of the configuration of the monitoring agent 205 and an example of a processing procedure thereof.

  FIG. 6A shows an example of the configuration of the monitoring agent 205. The monitoring agent 205 includes a monitoring data acquisition unit 2051 and a monitoring data transmission unit 2054. The monitoring data acquisition unit 2051 includes an OS performance information acquisition unit 2052 and a MapReduce scheduling information acquisition unit 2053. The monitoring data acquisition unit 2051 is configured to use a function for acquiring monitoring data corresponding to the monitoring target as a plug-in so that the monitoring agent 205 can acquire monitoring data from various monitoring targets. In the present embodiment, the monitoring data acquisition unit 2051 acquires the OS performance information acquisition unit 2052 for acquiring the OS performance information from the operating system (OS) 210, the MapReduce scheduling information from the job tracker 203 and the data node 207. The MapReduce scheduling information acquisition unit 2053 is used as a plug-in.

  The monitoring data transmission unit 2054 transmits the monitoring data acquired by the monitoring data acquisition unit 2051 and its plug-in to the monitoring manager 201. The means for transmitting may be unicast or multicast.

  FIG. 6B shows an example of the processing procedure of the monitoring agent 205. The monitoring agent 205 acquires OS performance information (S601), acquires MapReduce scheduling information (S602), transmits the acquired monitoring data to the monitoring manager 201 (S603), and waits for a certain period of time (S604). Step S601 is started again. As described above, the processing procedure of the monitoring agent 205 is one loop processing, and the monitoring data is continuously transmitted to the monitoring manager 201 at regular intervals while it is operating.

  FIG. 7 is a diagram illustrating an example of the configuration of the monitoring manager 201 and an example of a processing procedure thereof. FIG. 7A shows an example of the configuration of the monitoring manager 201. The monitoring manager 201 includes a monitoring data collection unit 2011, a monitoring data storage unit 2012, a database 2013, an operation status evaluation unit 2014, and an event notification unit 2015. The monitoring data collection unit 2011 collects monitoring data transmitted by the monitoring agent 205. The monitoring data storage unit 2012 stores the collected monitoring data in the database 2013. In the present embodiment, MapReduce scheduling information and OS performance information transmitted by the monitoring agent 205 described above are stored. The operation status evaluation unit 2014 evaluates the operation status of the parallel distributed processing system based on the information of the monitoring data stored in the database 2013, and when the occurrence of a failure is detected, the event notification unit 2015 notifies the monitoring console 202. Event notification is sent to the event.

  FIG. 7B shows an example of the processing procedure of the monitoring manager 201. The process of the monitoring manager 201 consists of two loop processes. The first loop receives monitoring data transmitted by the monitoring agent 205 (S701), stores MapReduce scheduling information in a database (S702), and stores OS performance information in the database (S703). The second loop performs an operation status evaluation based on information obtained from the database 2013 (S704), and if an occurrence of a failure is detected (S705), notifies the monitoring console 202 of an event (S706). After that, a certain time is waited (S707). Among these, steps S704 and S706 will be described in more detail later.

  As described above, the monitoring agent 205 and the monitoring manager 201 regularly communicate for exchanging monitoring data. In this embodiment, the monitoring agent 205 controls the trigger of the communication by executing step S603 in which the monitoring data is transmitted. However, the communication mode is not limited to this. For example, the monitoring manager 201 may periodically request the monitoring agent 205 to transmit monitoring data.

  FIG. 8 is a diagram illustrating an example of a table group stored in the database 2013 of the monitoring manager 201. The database 2013 stores a managed host list table 401. Further, the OS performance information table group 402 is stored for each host. Further, a table group 403 of MapReduce scheduling information is stored for each cluster.

  FIG. 9 is a diagram illustrating an example of the management target host list table 401. The management target host is the information processing apparatus 101 that is the target of the operation status determination by the monitoring manager 201. In this embodiment, it is typically a group of worker nodes 140, but other information processing apparatuses 101 can also be managed hosts.

  One record of the table corresponds to one managed host. Possible triggers for adding a management target host to the table include an operation by an operation manager, a notification process from the monitoring agent 205, a discovery process by the monitoring manager 201, and the like.

  In FIG. 9, only the fields necessary for the description are listed out of the fields of the table. The host name field 4011 records the host name of the management target host. The representative IP address field 4012 records an IP address assigned to one of the network I / Fs 104 of the managed host. The cluster name field 4013 records the MapReduce cluster 300 to which the managed host belongs. The failure detection flag field 4014 stores a flag indicating whether or not a failure has been detected on the managed host. It should be noted that there may be other fields in the table for recording information necessary for the monitoring manager processing.

  FIG. 10 is a diagram illustrating an example of a table group 402 of OS performance information. The OS performance information is monitoring data acquired by the monitoring agent 205 from the operating system 210 and transmitted to the monitoring manager 201. The monitoring manager 201 stores the received OS performance information in the table group 402 in the database 2013.

  In this embodiment, a processor performance information table 4021, a memory performance information table 4022, and a disk performance information table 4023 are shown as individual examples of the OS performance information table. What is common to each table is that each record includes a time at which the information is acquired and an interval at which the information is acquired. In the case of a processor or a disk, the interval indicates how many seconds of values are accumulated and calculated.

  In the case of processor performance information, the usage rate is further calculated from the accumulated value and recorded. In the case of disk performance information, the I / O amount per unit time is calculated from the accumulated value and recorded. On the other hand, in the case of memory performance information, the acquired value is a snapshot at the time of acquisition, and the interval literally means the information acquisition interval. Each record includes a plurality of items of detailed performance information. Each of these items is called a metric.

  The information processing apparatus 101 may be equipped with a plurality of disks constituting the processor 101 and the storage 103. The monitoring agent 205 acquires them as separate monitoring data from the operating system 210, and records them as separate records when the monitoring manager 201 records them in the table group 402 of OS performance information. Are identified by

  In the present embodiment, the above three are taken as typical OS performance information. However, the present invention is not limited to this, and other statistical information that can be acquired from the operating system 210 is also OS performance information. Can be one of the table groups 402.

  FIG. 11 is a diagram illustrating an example of a table group 403 of MapReduce scheduling information. The MapReduce scheduling information is monitoring data acquired by the monitoring agent 205 from the job tracker 203 and the data node 207 and transmitted to the monitoring manager 201. The monitoring manager 201 stores the received MapReduce scheduling information in the table group 403 in the database 2013.

  In the present embodiment, there are a job list 404, a task list 405, an attempt list 406, and a data transfer trace 407 as tables of MapReduce scheduling information.

  The relationship between the job 301 and the task 302 in the MapReduce method is as described with reference to FIG. The job list 404 records one record for each job. The job is uniquely identified by the job ID recorded in the job ID field. The task list 405 records one record for each task. The task is uniquely specified by the task ID recorded in the task ID field, and the job to which the task belongs is specified by the job ID recorded in the job ID field.

  The execution of the task 302 in the MapReduce method is called an attempt. The attempt list 406 records one record per one attempt. An attempt is uniquely specified by an attempt ID recorded in the attempt ID field, and a task that is the basis of the attempt is specified by a task ID recorded in the task ID field. Normally, only one attempt is recorded per task, but when the task execution fails, the task is re-executed, and the same task may be executed multiple times. In this case, multiple attempts are recorded in the attempt list 406 for the same task ID. In the execution node field of the attempt list 406, the host name of the worker node 140 that executed the attempt is recorded.

  As described above, the MapReduce cluster 300 includes a distributed file system. The data transfer trace 407 records data transferred by the data node 207. One record is recorded for each data transfer.

  FIG. 12 is a diagram illustrating an example of the operation status evaluation processing procedure (step S704) in the monitoring manager processing procedure. Based on this figure, the processing procedure for operating status evaluation is first outlined, followed by a more detailed procedure.

  First, the operation status evaluation unit 2014 acquires the job list 404, the task list 405, and the attempt list 406 from the database 2013 from the MapReduce scheduling information 403. Of these three tables, the job list 404 and the task list 405 both include a job ID field, and the task list 405 and the attempt list 406 both include a task ID field. The attempt list 406 includes an execution node field. Therefore, by joining the three tables with these fields, the group of worker nodes 140 used for executing a certain job can be determined. The process of extracting the worker node group is virtual group generation (S1201).

  Next, the operation status evaluation unit 2014 acquires the managed host list table 401 from the database 2013. For each managed host, the task type of the task executed on the managed host is obtained using the previous task list 405 and the attempt list 406. Next, the data transfer trace 407 is acquired, and the information of the data transferred from the managed host obtained therefrom is added to determine the node characteristics of each managed host (S1202). Next, the operating status evaluation unit 2014 merges the previously generated virtual group and the node characteristics of each managed host to generate a cluster map (S1203).

  When the cluster map is generated, for each managed host, a node performance matrix is generated from the OS performance information 402 (S1204), and correlation analysis is performed by calculating a canonical correlation coefficient using the node performance matrix. (S1205). The operating status evaluation unit 2014 detects the occurrence of a failure based on the result of the correlation analysis. If the occurrence of a failure is detected, the process proceeds to the event notification processing procedure (step S706) by the event notification unit 2015.

  Hereinafter, a detailed procedure for each of the above steps will be described.

  FIG. 13 is a diagram illustrating an example of a virtual group generation processing procedure (step S1201) in the operational status evaluation processing procedure. First, the operating status evaluation unit obtains the job list 404 (S1301), and extracts a job whose status is RUNNING or a job whose end time is not recorded (S1302). This extracts the job currently being executed. These job groups are called current jobs.

  Next, the task list 405 is acquired (S1303), and a record including the job ID of the current job is extracted therefrom (S1304). Further, a task whose status is RUNNING or whose end time is not recorded is extracted (S1305). Thereby, the task currently being executed is extracted.

  Next, an attempt list 406 is acquired (S1306), a record including the task ID of the task being executed is extracted therefrom (S1307), and an attempt whose status is RUNNING or whose end time is not recorded is extracted (S1308). ).

  Of the records extracted so far, the record extracted from the task list has a split field, and the record extracted from the attempt list has an execution node field. Therefore, for each of the current jobs, these correspondences are described in the virtual group table (S1310, S1311).

  FIG. 14A is a diagram showing an example of this virtual group table 501.

  All nodes including splits or execution nodes are set as a virtual group of a job (S1312). That is, one virtual group is created for each current job. FIG. 14B is a diagram illustrating an example of the virtual group node list table 502. This is a table in which the job ID, job name, and node name are extracted from the virtual group table 501. With this table, nodes belonging to a certain virtual group can be listed.

  The operation status evaluation unit 2014 stores the virtual group table 501 and the virtual group node list table 502 in the memory 102 of the monitoring server 110 for later processing. Alternatively, it may be stored in the database 2013.

  FIG. 15 is an example of a diagram showing the concept of the virtual group. In the virtual group 503, an execution node of the task 302 and a node including a split (Map task input file) 304 belong to a certain job.

  FIG. 16 is a diagram illustrating an example of a processing procedure (step S1202) for determining node characteristics. Node characteristic determination is to determine what role a node plays in a virtual group. This role is called node characteristics. The judgment materials are a table describing correspondences and a data transfer trace created at the time of virtual group generation. By referring to the transfer destination field of the data transfer trace, it is possible to determine where the split included in the node is transferred.

  Node characteristics are determined for each managed host. First, it is determined based on the virtual group node list table 502 whether the managed host belongs to the virtual group (S1601). If it belongs to the virtual group, the process is continued, but if it does not belong, the process is terminated as out of the operation status evaluation target (S1607). Determination is made (S1602). If it is an execution node, the task type is determined (S1603). Next, it is determined whether the split is included using the information of the virtual group table 501 (S1604). If a split is included, information on the data transfer trace is used to determine the transfer destination of the split (S1605). The node characteristics are determined using these processes, particularly information obtained by the task type determination S1603 and the transfer destination determination S1605, and the following table (S1606).

  FIG. 17 is a diagram illustrating an example of a table used for determining node characteristics and an example of a screen display used for setting them.

  FIG. 17A is an example of the simplest node characteristic classification. In this example, 1 node is set to 1 slot (only 1 task can be executed at the same time), the task type is Map, Reduce, or None (not an execution node), the transfer destination of the split is local, remote, no transfer One of them.

  From the information obtained in the previous steps S1603 and S1605, it can be determined which column of the table 504 the managed host corresponds to. Symbols (ML, MR, etc.) written in the column are the node characteristics of the managed host. These symbols written in the table 504 are symbols that are determined for the purpose of distinguishing a plurality of node characteristics, and any symbol system may be used as long as it is used for this purpose. Absent.

  FIG. 17B is an example of a somewhat complicated classification of node characteristics. A managed host may include multiple splits and transfer them to various destinations. From the data transfer trace information, the transfer destination and the number of bytes to which the managed host has transferred the split can be obtained. This is divided into local transfer (L) and remote transfer (R), and further classified into six levels according to the ratio of the transfer amount. Also, when multiple slots are set for one node, the task type of the task being executed is Map task (M) or Reduce task (R), depending on the ratio of the number of steps in five steps. Or, a state where there is no task being executed (None) is added to classify in six stages.

  Various tables can be considered for determining such node characteristics, but what kind of table is appropriate will vary depending on the parallel distributed processing system. Therefore, a person in charge of operation management of the parallel distributed processing system can instruct the monitoring manager what table to use.

  FIG. 17C is a diagram showing an example of a preference screen for setting a table used for determining these node characteristics. The preference screen 601 is a screen that the monitoring console 202 displays on the monitoring console screen 600. The node property use check box 6011, the node property automatic determination check box 6012, the preset method use check box 6013, the method selection drop-down box 6014, A custom method creation check box 6015 and a custom method table 6016 are provided. The custom method table 6016 includes a plurality of entries, and each entry includes a metrics use check box 6017 and a metrics name 6018. The preference screen 601 includes an OK / Cancel button 6019.

  The person in charge of operations designates a table to be used for determination of node characteristics using the monitoring console screen 600 displayed on the console 105 of the client 120 on which the monitoring console 202 is mounted and the human interface device of the console 105. To do. By checking the node property use check box 6011, it is possible to instruct to apply the node property to the operation status evaluation. By checking the node characteristic automatic determination check box 6012, the monitoring manager can be instructed to automatically determine the node characteristic. By checking this node characteristic automatic determination check box 6012, the following operations related to the node characteristic determination method can be performed.

  By checking the preset method use check box 6013, it is possible to instruct to use the node characteristic determination method registered in advance in the monitoring manager. When the preset method use check box 6013 is checked, a method selection drop-down box 6014 can be used. By operating this method selection drop-down box 6014, it is possible to select and instruct which of the node characteristic determination methods registered in advance is to be used. For example, when the management target host is a node constituting the MapReduce cluster, if a determination method named “MapReduce” is registered as an appropriate node characteristic determination method, this is selected.

  If the person in charge of operation determines that the node characteristic determination method registered in advance in the monitoring manager is not appropriate, check the custom method creation check box 6015 and instruct to use the customized node characteristic determination method. can do. When the custom method creation check box 6015 is checked, the custom method table 6016 can be operated.

  The custom method table 6016 lists various types of monitoring data collected from the monitoring agent by the monitoring manager, and instructs which of them is used to determine the node characteristics. The monitoring data is displayed as a list of entries in the custom method table 6016, and when there are a large number, only a part thereof is displayed by the scroll bar. The name of the monitoring data corresponding to each entry is displayed in the metric name 6018. When the metrics use check box 6017 included in each entry is checked, it can be instructed to use the monitoring data corresponding to the entry for node characteristic determination.

  When the operation manager selects and checks some of the metrics use check boxes in the custom method table 6016 that are determined to be appropriate, and presses the OK button of the OK / Cancel button 6019, the monitoring console is selected. The metrics information is sent to the monitoring manager. Based on the information, the monitoring manager builds a table used for determining node characteristics and stores it in the memory 102 of the monitoring server 110. Alternatively, it may be stored in the database 2013.

  FIG. 18 is a diagram illustrating an example of a cluster map generation processing procedure (step S1203).

  The cluster map is obtained by classifying nodes belonging to a virtual group according to node characteristics. First, a node characteristic determination result is acquired (S1801). Next, since there is one virtual group for each current job, sorting is first performed by job ID (S1802). Next, by sorting by node characteristics (S1803), the nodes can be classified for each node characteristic.

  FIG. 19 is a diagram illustrating an example of a cluster map table.

  It can be said that the cluster map table 506 is obtained by adding node characteristic determination results to the virtual group node list table 502 shown in FIG.

  FIG. 20 is an example of a diagram illustrating the concept of the cluster map.

  The cluster map 503 is obtained by classifying nodes related to execution of a job by node type, with a job identified by a job ID and a job name as a unit. A node 140 having a certain node characteristic belongs to the node characteristic group 507 together with nodes having the same node characteristic.

  FIG. 21 is a diagram illustrating an example of a processing procedure (step S1204) for generating a node performance matrix.

  For each managed host, it is first determined whether there is OS performance information of the host in the database (S2101). If not, it is excluded from the operation status evaluation (S2105). If there is, data of a certain time frame is acquired from the OS performance information (S2102), all metrics are concatenated (S2103), and a node performance matrix is generated (S2104).

  As shown in this procedure, the node performance matrix is obtained by concatenating the OS performance information acquired from the node. For the host in which the OS performance information is recorded in the database, the characteristics of the host are expressed in a time frame. It is characterized by the usage status of resources. Here, a simple concatenation of metrics is used as the node performance matrix, but other examples are possible. For example, using the fact that past OS performance information is also recorded in the database, it is possible to connect after calculating an exponential weighted moving average.

  In addition, the time frame is determined for the purpose of characterizing the characteristics of the host. Information acquisition intervals are recorded in the OS performance information. As long as the time frame is common to each host for each operation status evaluation process, it is necessary to select a time frame that includes a plurality of data for each metric, regardless of what is used.

  FIG. 22 is a diagram illustrating an example of the node performance matrix table. The node performance matrix table is a table that stores a node performance matrix, and is generated for each node that is an object of operation status evaluation. As linked in step S2103, each metric of the OS performance information is listed in the column direction of the table, and data in the time frame of each metric is arranged in the order of acquisition time in the vertical direction of the table. In this example, the name of each metric is included in the first row of the table, and the data acquisition time is included in the leftmost column. This is shown for easy understanding of the contents of the table, and is not necessarily included in the actual table. There is no need.

  FIG. 23 is a diagram illustrating an example of a correlation analysis processing procedure (step S1205).

  Correlation analysis is performed for each managed host. First, it is determined whether the managed host belongs to a virtual group (S2301). If it does not belong to the virtual group, it means that the managed host is not involved in the execution of any current job, and is excluded from the operation status evaluation target (S2314). Next, it is determined whether there is a node performance matrix for the managed host (S2302). If the node performance matrix is not generated because the OS performance information has not been acquired and the node performance matrix table of the managed host does not exist, it is excluded from the operation status evaluation target (S2314).

  The following processing is performed paying attention to the virtual group to which the managed host belongs. First, node characteristics existing in the virtual group are extracted (S2303), a node having the node characteristics is extracted for each node characteristic (S2304), and a correlation coefficient calculation process is performed for each node (S2304). S2305). Node and node characteristic information necessary for these processes can be extracted from the cluster map table 506.

  In this way, correlation coefficient calculation processing is performed with all node attributes existing in the virtual group to which the node belongs, regardless of whether the node characteristics are the same as or different from those of the managed host. By doing the above, it is possible to cope with any configuration of node characteristics belonging to the virtual group.

  Now, after selecting one node from the node group having certain node characteristics in step S2305, next, the node performance matrix Vn of the node is acquired (S2306). Then, a canonical correlation coefficient between the node performance matrix Vp of the managed host and the node performance matrix Vn of the node is calculated (S2307). The canonical correlation coefficient is represented as one or more correlation coefficients ρ1 to ρn.

  The number of canonical correlation coefficients higher than a certain threshold is information indicating the relationship between the two nodes of the current job. If this number is smaller than that between the same set of node characteristics when the job is normal, it means that the correlation between the nodes is low. If the phenomenon is observed for all other nodes, it is regarded as a failure detection for the managed host. In order to execute this processing, information on the canonical correlation coefficient between node characteristics at the time of normality of a certain job is necessary, and such information is referred to as predetermined canonical correlation data. A table storing this information is referred to as a job profile table and will be described later. Returning to the explanation of the correlation analysis procedure. After calculating the canonical correlation coefficients ρ1 to ρn, default canonical correlation data between the node characteristics is acquired from the job profile table using the node characteristics of the managed host and the selected node as a key (S2308). Next, among ρ1 to ρn, those higher than a certain threshold are selected (S2309), and when it is determined that the number is smaller than the number of predetermined canonical correlation data (S2310), the counter is incremented (S2311). . Thus, the calculation of the canonical correlation coefficient and the comparison with the predetermined canonical correlation data are performed for all node characteristics in the virtual group and the nodes belonging to the node group, that is, all nodes in the virtual group, and the value of the result counter is When the number of nodes in the virtual group (excluding the currently-managed managed host) is equal (S2312), it is determined that a failure has occurred in the managed host, and the table of managed host list The failure detection flag is set to 1 for the record (S2313).

  Any threshold can be set as long as it is consistent in one cluster or one system operation management apparatus. In addition, the counter used for determining the failure occurrence is not limited to when the number of nodes in the virtual group is equal, for example, when the number of nodes in the virtual group exceeds half of the number of nodes, the determination criteria are Anything can be set. Such a degree of freedom is necessary to adapt to the complexity of the parallel distributed processing system or the variety of failures that occur in the information processing apparatus.

  FIG. 24 is a diagram illustrating an example of a job profile table.

  The job profile table 509 classifies the nodes related to the execution of the job specified by the job name according to the node characteristics, and calculates the canonical correlation coefficient of the node performance matrix between the nodes in the normal state. As a result, a value larger than a predetermined threshold value is recorded, that is, predetermined canonical correlation data is recorded so that a set of job name and node characteristic can be searched as a key. The operating status evaluation unit 2014 records default canonical correlation data in the job profile table. At this time, as described above, only a value larger than a certain threshold value may be recorded, or all the calculated canonical correlation coefficients are recorded, and when executing the correlation calculation process, only a value larger than the threshold value is recorded. You may make it acquire. In addition, there may be a plurality of combinations of nodes having the same node characteristics, but the one with the smallest value among the canonical correlation coefficients calculated from each may be adopted as the default canonical correlation data. Alternatively, an average value or a median value may be calculated and employed. The operation status evaluation unit 2014 stores the job profile table 509 in the memory 102 of the monitoring server 110 for use in later correlation analysis processing. Alternatively, it may be stored in the database 2013.

  FIG. 25 is a diagram illustrating an example of an event notification processing procedure (S706) in the monitoring manager processing procedure.

  The event notification is a process of extracting, from the management target hosts, correlation detection result failure detection flag field 4014 set to 1, and notifying the monitoring console of the host information. The event notification unit first acquires the managed host list table 401 (S2501), and checks the failure detection flag field 4014 for each record in the table (S2502). If the field is 1, the host name field 4011 is extracted from the record and notified to the monitoring console (S2503). Further, other fields such as the cluster name field 4013 can be notified together with the host name field.

  FIG. 26 is a diagram illustrating an example of a screen display of event notification in the monitoring console.

  The monitoring console 202 that has received the notification from the event notification unit 2015 of the monitoring manager 201 displays the monitoring console screen 600 and the event notification screen 602 on the console 105 of the client 120, thereby notifying the person in charge of operation management of the occurrence of a failure. To do.

  The event notification screen 602 includes a cluster name display 6021, a node name display 6022 of a node belonging to the cluster, and a node status display 6023. If the event notification sent by the monitoring manager includes the cluster name field, the operation manager can easily understand the scope of the failure by classifying the nodes by cluster on the monitoring console screen. it can. For this purpose, a cluster name display 6021 is prepared. The host name included in the event notification is displayed on the node name display 6022. The node status display 6023 displays this in a manner that allows the operation manager to recognize that the monitoring manager has detected the occurrence of a failure in the node. For example, there are methods such as display by characters, display by change in color tone, or display by a combination thereof. In addition, information useful for the person in charge of operation when dealing with a failure can be displayed together.

  As shown in this event notification screen, the system operation management apparatus of this embodiment notifies the operation management person in charge of the occurrence of a failure in the managed host by displaying information via the console 105. There are various methods such as notification by sending an e-mail, notification by sounding a buzzer or turning on a rotation warning light.

  FIG. 27 is a diagram illustrating an example of a processing procedure in which the monitoring console displays the operation status of the managed host on the screen. The monitoring console 202 can display the operating status of the managed host on the console 105 of the client 120 without depending on the event notification processing from the monitoring manager 201. Thereby, regardless of whether or not a failure has occurred in the managed host, the person in charge of operation management can monitor the operating status of the managed host.

  First, the monitoring console 202 acquires the managed host list table 401 (S2701). The managed host list table is acquired from the database 2013 of the monitoring manager 201 through communication with the monitoring manager. Next, a display target host is extracted from the acquired management target host list (S2702). The display target host is a subset of the management target host list, and various criteria can be applied to the extraction. For example, it is possible to use information stored as a table in the memory of the monitoring server by the monitoring manager. . Examples thereof will be described later. Thereafter, the list of hosts to be displayed is shaped into a table format suitable for screen display (S2703), and screen display is performed via the console 105 of the client 120 (S2704).

  The monitoring console 202 acquires the managed host list table and the table stored in the memory by the monitoring manager as described above by communicating with the monitoring manager, and displays the operation status of the managed host on the screen. It is not always necessary to execute it every time it is displayed. The monitoring console saves the acquired table in the memory 102 of the client 120, and reduces the data transfer amount with the monitoring manager by reusing the table stored in the memory in a plurality of screen display processes. be able to. In this case, it is necessary to consider that there is no discrepancy between the table stored in the client memory and the table stored in the monitoring server memory or stored in the database. The process is commonly referred to as cache control and will be well known to those skilled in the art.

  FIG. 28 is a diagram illustrating an example of a screen display in which the monitoring console extracts a display target host based on information in the virtual group node list table as an example of a screen display of the operation status of the management target host.

  When the monitoring console displays the virtual group on the screen, for example, it is possible to display the virtual group in a manner similar to that shown in FIG.

  FIG. 28A is an example in which a cluster display screen 603 of a certain MapReduce cluster is displayed on the monitoring console screen 600 based on information in the virtual group node list table 502. The virtual group node list table is generated by the operation status evaluation unit 2014 of the monitoring manager 201, and the monitoring manager stores it in the memory of the monitoring server 110 as a table shown in FIG. 14B. Based on the managed host list table and the information in this table, if the monitoring console displays the operating status of the managed host in units of virtual groups on the monitoring console screen, a cluster name display 6031, a virtual group 6032, Node 6033 is displayed.

  FIG. 28B is another example in which a cluster display screen 603 of a certain MapReduce cluster is displayed on the monitoring console screen 600 based on the information in the virtual group node list table 502. In this example, a cluster name display 6031, a node 6033, and a job name 6034 related to the execution of the node are displayed. The monitoring console displays such a monitoring console screen 600 on the console 105 of the client 120, so that the person in charge of operation management can know the operating status of the managed host.

  FIG. 29 is a diagram showing an example of a screen display in which the monitoring console extracts the display target host based on the information of the cluster map table as another example of displaying the operation status of the management target host on the screen.

  When the monitoring console displays the cluster map on the screen, for example, it is possible to display it by imitating a diagram as shown in FIG. 20, but an example with higher listability is also possible.

  FIG. 29A is an example in which a cluster display screen 603 of a certain MapReduce cluster is displayed on the monitoring console screen 600 based on the information in the cluster map table 506. The cluster map table is generated by the operation status evaluation unit 2014 of the monitoring manager 201, and the monitoring manager stores it in the memory of the monitoring server 110 as a table shown in FIG. Based on the managed host list table and the information in this table, if the monitoring console displays the operating status of the managed host in units of virtual groups on the monitoring console screen, a cluster name display 6031, a virtual group 6032, A node 6033 and a node attribute 6035 are displayed.

  FIG. 29B is another example in which a cluster display screen 603 of a certain MapReduce cluster is displayed on the monitoring console screen 600 based on information in the cluster map table 506. In this example, a cluster name display 6031, a node 6033, a job name 6034 related to the execution of the node, and a node attribute label 6036 of the node are displayed.

  In the above description, the case where the job 301 has two types of tasks, the Map task 305 and the Reduce task 307, has been exemplified. However, even when a job has a plurality of Map tasks and does not include a Reduce task, the present invention can be implemented according to the above description. In this case, the node characteristics are determined by the split transfer destination (that is, the characteristics of the data file) (FIG. 16).

  Next, a second embodiment to which the present invention is applied will be described. The system operation management apparatus having the failure detection function shown in the first embodiment has implemented a monitoring agent on each managed host in order to collect OS performance information and MapReduce scheduling information from the managed host. In many cases, such a monitoring agent is installed on the host by an operation manager. In other words, the more the number of managed hosts, the more complicated the work will be. There may also be concern about the monitoring agent consuming some of the host's memory.

  Therefore, in the present embodiment, an example in which failure detection is performed without using a monitoring agent will be described. Since the basic configuration is the same as that of the first embodiment, only differences will be described.

  FIG. 30 is a diagram illustrating an example of a parallel distributed processing system according to the second embodiment. The difference from the parallel distributed processing system according to the first embodiment shown in FIG. 2 is that the master node 130 and the worker node 140 do not implement the monitoring agent, and the monitoring server 110 implements the remote monitor 208 instead.

  FIG. 31 is an example of a block diagram of the monitoring manager and the remote monitor in the second embodiment.

  The remote monitor 208 includes a remote monitoring data acquisition unit 2081 and a monitoring data transmission unit 2084.

  The remote monitoring data acquisition unit 2081 includes an OS performance information acquisition unit 2082 and a MapReduce scheduling information acquisition unit 2083. The remote monitor 208 is configured such that the remote monitoring data acquisition unit 2081 uses a function for acquiring monitoring data according to the monitoring target as a plug-in so that the monitoring data can be acquired from various monitoring targets. In this embodiment, the remote monitoring data acquisition unit 2081 acquires the MapReduce scheduling information from the OS performance information acquisition unit 2082 for acquiring OS performance information from the operating system (OS) 210, the job tracker 203, and the data node 207. Each of the MapReduce scheduling information acquisition units 2083 is used as a plug-in.

  The monitoring data transmission unit 2084 transmits the monitoring data acquired by the remote monitoring data acquisition unit 2081 and its plug-in to the monitoring manager 201. A monitoring data collection unit 2011 of the monitoring manager 201 collects monitoring data transmitted by the remote monitor 208.

  Plug-ins get information in their own way. As an example, in the case of OS performance information, there is a method of using SSH (registered trademark) and an OS command or SNMP. In the case of MapReduce scheduling information, there is a method of collecting log files of job trackers and data nodes by SSH. In any case, the acquired information is the same as the plug-in implemented by the monitoring agent in the first embodiment.

  As a method for the monitoring data transmission unit 2084 to transmit the monitoring data to the monitoring manager 201, there is an inter-process communication method such as socket, RPC, or HTTP.

  With the method described above, the second embodiment applies the present invention to a parallel distributed processing system without mounting a monitoring agent.

  Next, a third embodiment to which the present invention is applied will be described. In the first embodiment, monitoring data of OS performance information collected from a managed host in node performance matrix generation is used. In the third embodiment, in addition to this, the node performance index of the managed host is used. The node performance index is a numerical expression of individual performance of computing resources such as a processor and a memory included in the information processing apparatus. For example, for a processor, the number of processors and operating frequency of a certain managed host are node performance indicators. The purpose of this embodiment is to use this information to more effectively perform fault detection for a parallel distributed processing system composed of information processing devices having diversity in the computational resources of the information. To do. Hereinafter, since the basic configuration is the same as that of the first embodiment, only the difference will be described.

  FIG. 32 is a diagram illustrating an example of a processing procedure for generating a node performance matrix in the third embodiment. In addition to the processing procedure shown in FIG. 21, step S3203 is added. In step S3203, the operation status evaluation unit 2014 of the monitoring manager 201 acquires the node performance index of the management target host. Typically, the node performance index is recorded in the managed host list table 401, and is acquired. In step S3204, in addition to the metrics included in the data of a certain time frame acquired from the OS performance information, the nodes enumerated numerical values of the node performance indices are connected to generate a node performance matrix (S3205).

  FIG. 33 is a diagram illustrating an example of the management target host list table 401 in the third embodiment. In addition to the contents of the managed host list table 401 shown in FIG. 9, a field 4015 for recording the number of processors and a field 4016 for recording the operating frequency of the processors are added. It is noted that these fields are exemplified as typical node performance indicators, and other information related to computational resources such as the amount of memory installed can also be used as node performance indicators. I want.

  Next, a fourth embodiment to which the present invention is applied will be described. In the first embodiment, a failure is detected by canonical correlation analysis, that is, by calculating the canonical correlation coefficient from two node performance matrices. In the fourth embodiment, failure detection is performed using various statistical methods without being limited to canonical correlation analysis.

  In the first place, the gist of the present invention is that a node performance matrix is generated from monitoring data acquired from an information processing apparatus, and their correlation is analyzed using a statistical method. The statistical method that can be used for such purposes is not limited to canonical correlation analysis. In a field generally known as multivariate analysis among statistical methods, statistical methods for classifying data, compressing dimensions, and extracting features for a data group composed of a plurality of variables have been studied. For example, methods such as principal component analysis and cluster analysis using Euclidean distance as a distance function are known, and canonical correlation analysis is also an example.

  In applying these various methods to fault detection based on monitoring data, there is a certain suitability. For example, regarding a group of nodes that have the same node characteristics in the execution of a job, when the monitoring data is regarded as time-series data, there is little fluctuation on the whole, but locally the nodes do not synchronize with each other There may be slight fluctuations. In such a case, for example, by calculating the Euclidean distance pairwise for the node performance matrix and determining the distance between the nodes by the group average method, abnormal data can be detected from various monitoring data groups. .

  Such a statistical method is realized as a program for implementing an algorithm on the memory of the information processing apparatus. Then, as a method of selecting an appropriate algorithm from these various algorithm groups according to the nature of the job, for example, monitoring data is displayed on the monitoring console screen as a graph of time-series data, and the operation manager observes the behavior. A method of determining and setting an appropriate algorithm can be considered. It is also conceivable to automate this process with a program.

  There are various other methods for determining the suitability of the algorithm according to the nature of the job. The present invention focuses on various algorithms that are applied to failure detection based on analysis of monitoring data. It is a point that a more effective fault detection can be realized by properly using things. Therefore, in this embodiment, a configuration and processing procedure for detecting a failure more effectively by selecting an analysis algorithm to be applied according to a job or a set of node characteristics in the job will be described. Hereinafter, since the basic configuration and the processing procedure are the same as those in the first embodiment, only differences will be described.

  FIG. 34 is a diagram illustrating an example of a table used by the operation status evaluation unit 2014 in the fourth embodiment.

  FIG. 34A is a diagram illustrating an example of a job profile table with an analysis algorithm in the fourth embodiment. In the job profile table 509 shown in FIG. 24, an analysis algorithm applied to a certain node characteristic and its comparison target is implicitly assumed. On the other hand, the job profile table 510 shown in FIG. 34A includes an analysis algorithm field 5101 and a threshold data field 5102. That is, the job profile table 510 records the analysis algorithm and threshold data so that a combination of job name and node characteristic can be searched as a key.

  An analysis algorithm field 5101 for recording an analysis algorithm includes an ID uniquely corresponding to a specific algorithm. A table for recording the algorithm ID will be described later.

  A threshold data field 5102 containing threshold data contains data that the analysis algorithm uses to determine the occurrence of a fault. The job profile table 509 in the first embodiment records predetermined canonical correlation data, which, as the name indicates, is necessary data in the failure detection process based on the canonical correlation analysis. On the other hand, the threshold data field 5102 for recording threshold data in this embodiment includes threshold data corresponding to various analysis algorithms that the analysis algorithm field 5101 can include. It should be noted that the threshold data is a component different from the threshold used in the correlation analysis process (step S2309 in FIG. 23) in the first embodiment.

  FIG. 34B is a diagram illustrating an example of the analysis algorithm table. The analysis algorithm table 511 includes an algorithm ID field 5111, an algorithm name field 5112 that records the name of the analysis algorithm corresponding to the algorithm ID, an analysis function pointer field 5113 that records a pointer to a function that implements the analysis algorithm, and the analysis function A threshold value determination function pointer field 5114 for recording a pointer to a function for comparing the correlation value and the threshold value data. A pointer to a function is an address indicating a program installed in the memory 102 of the monitoring server 110 as in the monitoring manager 201, and is an address on the memory space of the analysis function program 512, for example. That is, the analysis algorithm table 511 is recorded so that a program that implements a certain analysis algorithm that uniquely corresponds to the algorithm ID and a program that compares the correlation value calculated by the analysis algorithm with threshold data can be searched using the algorithm ID as a key. It is a thing. The name of the analysis algorithm recorded in the algorithm name field 5112 is used when presenting information to the person in charge of operation management via the monitoring console 202.

  The function pointer does not necessarily need to be an address in the memory space. For example, the analysis function program 512 is implemented as a user process 200 different from the monitoring manager, and the program and the monitoring manager are used for communication between processes. An endpoint may be considered a function pointer. One suitable for those skilled in the art may be selected from a variety of techniques related to the mutual calling of such programs.

  The analysis algorithm field 5101 of the job profile table 510 can be rewritten at an arbitrary timing during the operation of the system operation management apparatus. Similarly, the fields 5113 and 5114 in which the function pointers of the analysis algorithm table 511 are recorded and the program stored at the address in the memory space indicated by the pointers can be rewritten at an arbitrary timing. Of course, a plurality of programs can be mounted in the memory of the monitoring server 110, and the function pointer recorded in the field of the analysis algorithm table 511 can be switched from one program address to another program address. That is, the analysis algorithm to be applied can be variously changed during the operation of the parallel distributed processing system. Such a degree of freedom is necessary to adapt to the complexity of the parallel distributed processing system or the variety of failures that occur in the information processing apparatus. An example of timing for rewriting such a field will be described later.

  The operating status evaluation unit 2014 stores the job profile table 510 and the analysis algorithm table 511 in the memory 102 of the monitoring server 110 for use in correlation analysis processing. Alternatively, it may be stored in the database 2013.

  FIG. 35 is a diagram illustrating another example of the correlation analysis processing procedure (step S1205). The correlation analysis processing procedure shown in FIG. 23 is based on the assumption that canonical correlation analysis is used as the analysis algorithm, but here, a process for selectively using a plurality of analysis algorithms is shown. For convenience, the term “correlation” is used for explanation, but the meaning in statistics does not limit the statistical method applied in this embodiment, and “similarity” as a superordinate concept of “correlation coefficient”. Or it should be understood as a broader one that includes the concept of “distance” in an arbitrary metric space.

  In the correlation analysis procedure, the process from the determination of belonging to the virtual group to the acquisition of the node performance matrix (steps S3501 to S3506) is the same as in the first embodiment. That is, the correlation analysis is performed for each managed host. First, it is determined whether the managed host belongs to the virtual group (S3501). If the managed host does not belong to the virtual group, the managed host determines which current job to execute. Is not involved, and is excluded from the operation status evaluation (S3513). Next, it is determined whether or not the node performance matrix of the managed host exists (S3502). If the OS performance information is not acquired, the node performance matrix is not generated, and the node performance matrix table of the managed host. Is not included in the operation status evaluation (S3513).

  The following processing is performed paying attention to the virtual group to which the managed host belongs. First, node characteristics existing in the virtual group are extracted (S3503), a node group having the node characteristics is extracted for each node characteristic (S3504), and one node is sequentially selected from these node groups to obtain correlation. Calculation processing is performed (S3505). Node and node characteristic information necessary for these processes can be extracted from the cluster map table 506. Next, for one node selected in the loop of step S3505, the node performance matrix Vn of the node is acquired (S3506). The subsequent processing is the difference of the first embodiment.

  First, the function f1 is executed with the node performance matrix Vp of the managed host and the node performance matrix Vn acquired in step S3506 as arguments, and the solution r is obtained (S3507). The function f1 is obtained by searching the job profile table 510 using a combination of the node characteristics of the managed host and the selected node as a key to acquire an algorithm ID, and further searching the analysis algorithm table 511 using the algorithm ID as a key. This is an analysis function program that can be indicated by an analysis function pointer. Typically, the analysis function f1 takes Vn and Vp as arguments, and its return value is r. This r is the value of “correlation” described above, and means the similarity or distance between two node performance matrices.

  Next, the threshold value t is obtained by searching the job profile table 510 using a set of node characteristics of the managed host and the selected node as a key (S3508). This threshold value t is data intended to be compared with the correlation value r.

  Next, the function f2 is executed with r and t as arguments, and a true / false value is obtained as a solution (S3509). If true, it is considered that the threshold has been exceeded, and the counter is incremented (S3510). If false, it is assumed that the threshold has not been exceeded. Similar to the function f1, the function f2 searches the job profile table 510 using a combination of the node characteristics of the managed host and the selected node as a key to obtain an algorithm ID, and further uses the algorithm ID as a key to obtain an analysis algorithm table 511. This is a threshold value determination function program indicated by a threshold value determination function pointer, which can be acquired by searching. Typically, the threshold value determination function f2 takes r and t as arguments, and uses a true / false value as a return value.

  In this way, the correlation value is calculated and the threshold data is compared with all node characteristics in the virtual group and the nodes belonging to it, that is, all nodes in the virtual group. If it is equal to the number of nodes (excluding managed managed hosts) (S3511), it is determined that a failure has occurred in the managed host, and a failure detection flag is set for the record in the managed host list table. It is set to 1 (S3512). The value of this counter is not limited to when it is equal to the number of nodes in the virtual group. For example, it can be set as a criterion for the determination, such as when a failure occurs when the number of nodes in the virtual group exceeds half of the number. This is the same as in the first embodiment.

  The threshold determination function f2 does not necessarily compare the magnitudes of r and t as scalar values. For example, if canonical correlation analysis is adopted as the function f1, r is an array of canonical correlation coefficients ρ1 to ρn, and t is a canonical value that is a certain value or more when the parallel distributed processing system is normal. This is an array of correlation coefficients, and the function f2 compares the number of elements n1 of the elements of the array r that are greater than or equal to a certain value with the number of elements n2 of the array t, and true if n1 <n2, otherwise In the case of, it will return fake. Similarly, t may be a structure that stores an array of canonical correlation coefficients when the system is normal and a threshold for determining the “value above a certain value”. These examples are substantially the same as the processing based on the canonical correlation analysis in the first embodiment, but by abstracting with the two functions f1, f2 and the threshold data t in this way, Various analysis algorithms can be applied.

  If an algorithm ID is not recorded in the analysis algorithm field 5101 of the job profile table 510, a default analysis algorithm may be used. Such a default analysis algorithm may be fixed, or may be designated by an operation manager.

  As described above, the analysis algorithm applied at an arbitrary timing can be changed during the operation of the parallel distributed processing system. The determination as to which analysis algorithm to apply may be performed at the discretion of the person in charge of operation management, or may be performed by a system operation management apparatus by a program. In the present embodiment, an example of a method for allowing an operation management person to instruct a system operation management apparatus of an analysis algorithm used for failure detection via a monitoring console is shown first. Next, an example of a method in which the system operation management apparatus determines an analysis algorithm used for failure detection by a program will be described.

  FIG. 36 is a diagram illustrating an example of a screen for setting an analysis algorithm used for failure detection. The analysis algorithm setting screen 604 is a screen that the monitoring console 202 displays on the monitoring console screen 600, and includes a default analysis algorithm selection drop-down box 6041, a multi-analysis algorithm use check box 6042, an analysis algorithm automatic determination radio button 6043, an analysis algorithm manual. A setting radio button 6044, a custom analysis algorithm setting table 6045, and a custom analysis algorithm setting button 6048 are provided. The custom analysis algorithm setting table 6045 includes a plurality of entries, and each entry includes fields of an association use check box 60451, a job name 60452, a node characteristic set name 60453, and an analysis algorithm name 60454. The analysis algorithm setting screen 604 includes an analysis algorithm list display link 6046 and an OK / Cancel button 6047.

  The person in charge of operation management uses the monitoring console screen 600 displayed on the console 105 of the client 120 on which the monitoring console 202 is installed and the human interface device of the console 105 to monitor the analysis algorithm used for fault detection. 201 can be specified.

  The default analysis algorithm selection drop-down box 6041 displays the names of analysis algorithms that are installed as programs in the system operation management apparatus and can be applied to the correlation analysis processing as options. An algorithm can be selected and directed to the monitoring manager. The default analysis algorithm is applied when there is no other opportunity to select an analysis algorithm. This is the case, for example, when there is no instruction to use multiple types of analysis algorithms, or when a job is executed for the first time, or information that is a prerequisite for executing automatic determination of analysis algorithms is not yet accumulated. The default analysis algorithm is applied when the analysis algorithm to be applied is not automatically determined for a reason or when the analysis algorithm to be used is not manually set. If there is only one analysis algorithm installed as a program in the system operation management apparatus, the default analysis algorithm selection drop-down box 6041 displays only the name of the analysis algorithm as an option, and as a default analysis algorithm Apply.

  By checking the multi-analysis algorithm use check box 6042, it is possible to instruct to apply a plurality of types of analysis algorithms to failure detection. Checking the use of multi-analysis algorithm check box 6042 enables the following operations related to analysis algorithm setting.

  When the analysis algorithm automatic determination radio button 6043 is selected, it is possible to instruct the monitoring manager to determine the analysis algorithm to be applied without depending on the designation by the person in charge of operation management. On the other hand, when the analysis algorithm manual setting radio button 6044 is selected, the monitoring manager can be instructed to change the analysis algorithm to be applied based on the setting by the person in charge of operation management. These two radio buttons are in an exclusive relationship and cannot be selected at the same time. Hereinafter, a mode in which the monitoring manager performs processing according to the former designation is referred to as an automatic determination mode, and a mode in which processing according to the latter designation is referred to as a manual setting mode.

  When the monitoring manager is set to the automatic determination mode, the monitoring manager determines an analysis algorithm to be applied based on the collected monitoring data and information of the cluster map table. An example of this process will be described later.

  On the other hand, when the monitoring manager is set to the manual setting mode, the analysis algorithm is applied based on an instruction from the person in charge of operation management. That is, when the analysis algorithm manual setting radio button 6044 is selected, the custom analysis algorithm setting table 6045 can be operated. The custom analysis algorithm setting table 6045 enumerates combinations of job names and node characteristics recorded in the job profile table 510 and instructs association of which analysis algorithm is applied to them. These pieces of association information are listed as entries in the custom analysis algorithm setting table, and in the case of a large number, only a part of them is displayed by a scroll bar.

  When an association use check box 60451 at the beginning of an entry is checked, association based on the contents of each field belonging to the entry is validated. By operating this check box, operations such as temporarily inhibiting application of the association and enabling it can be performed.

  For the job name field 60452 and the node property set name field 60453, an analysis algorithm to be applied is selected in the analysis algorithm name field 60454. There can be one or more sets of node characteristics for one job name, but the default analysis algorithm is applied if there is no setting. This association is set and displayed without an explicit instruction from the person in charge of operation management. For example, “Default” is set as the set name of the node characteristic, the name of the default analysis algorithm is set as the analysis algorithm name, Both are displayed in italics.

  The analysis algorithm name field also serves as a drop-down box, and the analysis algorithm for which association is to be set can be selected. When the name of the displayed analysis algorithm is the default analysis algorithm, it is displayed so that it can be distinguished. For example, the analysis algorithm name is displayed in italics.

  When an algorithm list display link 6046 is selected, a list of analysis algorithm names that are installed as programs in the system operation management apparatus and are applicable to correlation analysis processing are displayed on the monitoring console screen 600. Typically, this list is displayed as a screen different from the analysis algorithm setting screen 604, and the name, characteristics, and past of the analysis algorithm are provided so that the operation manager can refer to the custom analysis algorithm setting table. Displays information such as the usage record of. Similar information may be displayed by selecting a more suitable means from the viewpoint of operability of the monitoring console 202, such as a help window and a tool tip.

  When a custom analysis algorithm setting button 6048 is pressed, a screen for adding an association with the analysis algorithm for the job name and the set of node characteristics is displayed. This screen displays the combination of job name and node characteristics as in the custom analysis algorithm setting table, but all the jobs known to the monitoring manager recorded in the cluster map table 506 and the combination of the node characteristics are entered as entries. indicate. When an entry for which an association with an analysis algorithm is to be set is selected from the entry group, the entry is added to the custom analysis algorithm setting table, and an association with the analysis algorithm can be set. In the entry added by this operation, the selected node characteristic pair is displayed instead of “Default” in the node characteristic pair name field, and the analysis algorithm name field also functions as a drop-down box so that the analysis algorithm to be set can be selected. To do.

  The monitoring manager may display information for reference by the person in charge of operation management when setting the association. This includes, for example, displaying an automatic determination result obtained from a method used for automatic determination of an analysis algorithm to be described later, displaying information of representative monitoring data for each node characteristic as a graph of time series data, Including various methods.

  When the person in charge of operation management presses the OK button of the OK / Cancel button 6047, the monitoring console transmits information related to the association of the analysis algorithm to the monitoring manager. This is typically the name of the default analysis algorithm, the availability of multiple types of analysis algorithms, the distinction between automatic judgment mode and manual setting mode, the name of the job name and its node characteristics, and the associated analysis algorithm. Although it is information such as the name, a suitable method may be selected in view of processing efficiency, such as transmitting only the difference between the information changed before and after the operation of the analysis algorithm setting screen.

  The monitoring manager records the analysis algorithm ID in the analysis algorithm field 5101 of the job profile table 510 based on the information received from the monitoring console. In addition, threshold data corresponding to the analysis algorithm is recorded in the threshold data field 5102. Depending on the analysis algorithm to be applied, corresponding threshold data is required, but the threshold data used in the past may be reused. Therefore, the threshold data is stored in the memory or stored in the database, and appropriately copied to the threshold data field of the job profile table, or the threshold data field with the address in the memory space of the threshold data and the search key on the database. Or may be recorded contents.

  As described above, when the monitoring manager is set to the automatic determination mode, the analysis algorithm to be applied is determined based on the collected monitoring data and the information of the cluster map table. An example of the automatic determination process will be described first, followed by a processing procedure.

  FIG. 37 shows an example in which OS performance information collected from a certain managed host is drawn on a graph as time series data. The horizontal axis represents the transition of time, and the vertical axis represents one of the metrics included in the OS performance information, for example, the change in the usage rate of the processor.

  FIG. 37A is an example of a graph showing a change in the metric of a node having a certain node characteristic A. FIG. When the change of the metric is observed in the section 701, for example, the maximum value in the section is 702 and the minimum value is 703. On the other hand, FIG. 37B is another example showing a change in the metric of a node having another node characteristic B. Similarly, when observing in the section 701, the maximum value in that section is 704 and the minimum value is 705. If attention is paid to the difference α between 702 and 703 and the difference β between 704 and 705, α >> β. In any graph, the metric fluctuates at a fine level, but there is a significant difference in the global fluctuation.

  This means that the analysis algorithm to be applied differs between the node characteristic A characterized by metric fluctuation as shown in FIG. 37A and the node characteristic B characterized by metric fluctuation as shown in FIG. 37B. That's what it means. This is because nodes having a metric variation feature such as node characteristic A can appropriately detect the global variation as a correlation, while nodes having a metric variation feature such as node characteristic B. In the case of mutual correlation, even if a correlation is not detected by a typical correlation analysis algorithm, or if a correlation is detected, it may not be detected due to a change in the correlation at the time of failure, typically a decrease in the correlation. This is because it cannot be ignored.

  For this reason, it is necessary to automatically determine the analysis algorithm using the characteristics of fluctuations in OS performance information. An analysis algorithm based on canonical correlation analysis is effective for node characteristics such as the former, and an analysis algorithm based on relative comparison of metrics between nodes is effective for node characteristics such as the latter. is there.

  For example, a method based on autocorrelation analysis can be considered as an analysis algorithm automatic determination method focusing on the feature of metric fluctuation. Here, as a simpler example, a method using a maximum value and a minimum value is shown.

  FIG. 38 is a diagram illustrating an example of a processing procedure for automatic analysis algorithm determination. This process is executed at an arbitrary timing when the monitoring manager is set to the automatic determination mode.

  The automatic determination of the analysis algorithm is typically performed for each virtual group. The monitoring manager in the automatic determination mode first selects a certain virtual group recorded in the cluster map table 506, and further extracts a node group having a certain node characteristic C from the virtual group (S3801). Next, one node is randomly extracted from the node group (S3802). The OS performance information of the extracted node is acquired (S3803), and for each of a plurality of metrics included in the OS performance information (S3804), data of a certain time frame is acquired, and the maximum value and minimum value in the section are acquired. A value is obtained (S3805). Next, the difference between the maximum value and the minimum value, that is, the fluctuation range is compared with a certain threshold value (S3806). If the fluctuation range exceeds the threshold value, the counter A is incremented (S3807). Is within the threshold value, the counter B is incremented (S3808).

  After comparing the fluctuation range with the threshold value for all metrics, the value of the counter A is compared with the value of the counter B (S3809). As a result of the comparison, if the counter A is larger, the application of the analysis algorithm A is determined (S3810), and if the counter B is larger, the application of the analysis algorithm B is determined (S3811). Typically, the analysis algorithm A is an example of canonical correlation analysis, and the analysis algorithm B is an example of an algorithm that focuses on an average value of metrics as described later.

  When the analysis algorithm is determined, the monitoring manager searches the job profile table 510 for a record of a combination of the job name and node characteristics C involved in the execution of the virtual group, and uses the determined analysis algorithm as the analysis algorithm for the record. The data is recorded in the field 5101 and applied to the subsequent correlation analysis processing.

  In addition, after determining the analysis algorithm to be applied to a certain node characteristic of a job, if the automatic determination processing has not yet been performed for another node characteristic, the analysis algorithm is applied to another set of node characteristics. You may record together as an algorithm. Thus, if an analysis algorithm to be applied to one node characteristic is determined, it can be applied as an analysis algorithm for the job instead of the default analysis algorithm.

  The automatic determination method is a method of paying attention to the maximum value and the minimum value of a certain metric. These two values can also be used for analysis for fault detection. That is, an example of an analysis algorithm applied when only a metric with little fluctuation can be obtained is shown.

  FIG. 39 shows an example in which OS performance information collected from three managed hosts having certain node characteristics is drawn on a graph as time series data. The horizontal axis represents the transition of time, and the vertical axis represents one of the metrics included in the OS performance information, for example, the change in the usage rate of the processor.

  In FIG. 39, the maximum value of the processor usage rate in the managed host of a certain node characteristic C obtained by executing the above-described analysis algorithm automatic determination processing is indicated by 706, and the minimum value is indicated by 707. Further, the average values of the processor usage rates of two managed hosts having node characteristics C are indicated by 708 and 709, respectively. It can be seen that the average values 708 and 709 are within the range spanned by 706 and 707.

  On the other hand, suppose that the average value of the processor usage rate of another managed host having the node characteristic C is 710. This average value 710 is out of the range of 706 and 707. With this, it is determined that a failure has occurred in the managed host exhibiting the average value 710.

  This analysis algorithm only calculates the average value and threshold value determination for each node, and is simpler than the above-described correlation analysis processing procedure. However, depending on the job, failure detection is possible at a practical level.

  As described above, during the operation of the parallel distributed processing system, the analysis algorithm applied to the failure detection by the monitoring manager's operation status evaluation unit 201 can be manually set by the person in charge of operation management or automatically determined by the monitoring manager. Various changes can be made.

  The monitoring manager can also switch between the automatic determination mode and the manual setting mode during operation. In this case, it is desirable to maintain the analysis algorithm and threshold data recorded in the job profile table 510 before the mode transition. However, after the mode transition, it is prohibited to change these data based on each determined process. Absent.

  By the method described above, the fourth embodiment realizes more effective fault detection by selecting and applying one from a plurality of analysis algorithms at various occasions.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a memory, a storage device such as HDD or SSD, or a storage medium such as an SD card or DVD-ROM.

  Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 100 Information processing apparatus 110 Monitoring server 120 Client 130 Master node 140 Worker node 200 User process 201 Monitoring manager 202 Monitoring console 203 Job tracker 204 Name node 205 Monitoring agent 206 Task tracker 207 Data node 208 Remote monitor 300 MapReduce cluster 301 Job 302 Task 303 Data Block 304 Split 305 Map task 306 Intermediate file 307 Reduce task 308 Output file 309 Map slot 310 Reduce slot 401 Managed host list table 402 OS performance information 403 MapReduce scheduling information 404 Job list 405 Task list 406 Temp List 407 Data transfer trace 501 Virtual group table 502 Virtual group node list table 503 Virtual group 504 Node characteristic table 505 Multiphase node characteristic table 506 Cluster map table 507 Node characteristic group 508 Node performance matrix table 509 Job profile table 510 Job with analysis algorithm Profile table 511 Analysis algorithm table 600 Monitoring console screen 601 Preferences setting screen 602 Event notification screen 603 Cluster display screen 604 Analysis algorithm setting screen

Claims (13)

An operation management device of an information processing system that executes a job in cooperation with a plurality of information processing devices,
A data collection unit for obtaining information from each of the plurality of information processing devices;
A storage unit for storing data relating to the plurality of information processing devices;
An evaluation unit that evaluates the state of the plurality of information processing devices using data stored in the storage unit;
Each of the plurality of information processing devices has any one of predetermined characteristics,
The data collection unit acquires performance information from each of the plurality of information processing devices and stores the performance information in the storage unit,
The storage unit further stores, for each combination of characteristics that can be taken by the two information processing devices, a threshold value regarding the correlation between the performance information of the two information processing devices,
The evaluation unit is
When evaluating the state of the information processing apparatus as the evaluation target for one information processing apparatus as the evaluation target among the plurality of information processing apparatuses,
For each of the plurality of information processing apparatuses other than the information processing apparatus to be evaluated, a correlation value of performance information with the information processing apparatus to be evaluated is calculated and a combination of characteristics with the information processing apparatus to be evaluated is specified. And comparing the threshold value stored in the storage unit with the calculated correlation value for the specified combination of characteristics,
An operation management apparatus that evaluates a state of the information processing apparatus to be evaluated based on a result of the comparison. The operation management apparatus according to claim 1, wherein the job includes a plurality of tasks and a plurality of data files.
The operation management apparatus, wherein the plurality of predetermined characteristics are characteristics determined based on a type of a task executed by the information processing apparatus and characteristics of a data file input / output by the information processing apparatus. The operation management apparatus according to claim 2, wherein the job is a MapReduce-type job, and the job includes a Map task and a Reduce task,
The plurality of predetermined characteristics are characteristics determined based on whether a task executed by the information processing apparatus is a Map task or a Reduce task, and a type of data file transfer accompanying the execution of the task. Operation management device. The operation management device according to claim 2,
For each job executed in the information processing system, the data collection unit inputs and outputs the type of task executed by the information processing apparatus and the information processing apparatus for each of a plurality of information processing apparatuses that execute the job. Get data file characteristics,
The operation management apparatus, wherein the evaluation unit specifies characteristics of each of the plurality of information processing apparatuses based on a task type and data file characteristics collected by the data collection unit. The operation management device according to claim 1,
The threshold value stored in the storage unit is a value calculated by the evaluation unit using performance information collected from the plurality of information processing devices by the data collection unit. The operation management device according to claim 5,
The operation management apparatus characterized in that the performance information includes a performance index of a computing resource constituting the information processing apparatus. The operation management device according to claim 1,
The operation management apparatus, wherein the evaluation unit determines occurrence of an abnormality in the information processing apparatus based on a comparison between a threshold value stored in the storage unit and the calculated correlation value. The operation management apparatus according to claim 1, wherein the job includes a plurality of tasks and a plurality of data files.
The operation management apparatus, wherein the plurality of predetermined characteristics are characteristics determined based on characteristics of a data file input / output by the information processing apparatus. The operation management device according to claim 1,
The storage unit is calculated by using a correlation value calculating unit that calculates a correlation value of performance information of the second information processing device and a correlation value calculating unit for each combination of characteristics that the two information processing devices can take. A correlation value and a threshold value determination means for comparing the threshold value,
The said evaluation part calculates the correlation value of the said performance information using the said correlation value calculation means, and compares the said correlation value with the said threshold value using the said threshold value determination means, The operation management apparatus characterized by the above-mentioned. The operation management device according to claim 9,
The storage unit stores a plurality of the correlation value calculation means and a plurality of the threshold determination means,
In the case where the evaluation unit evaluates the state of the evaluation target information processing apparatus for one evaluation target information processing apparatus among the plurality of information processing apparatuses,
Calculate the maximum and minimum values of the performance information collected from the information processing device to be evaluated,
An operation management apparatus that switches between the correlation value calculation means and the threshold value determination means based on the difference between the maximum value and the minimum value. The operation management apparatus according to claim 10,
The storage unit stores a maximum value and a minimum value of the performance information,
The evaluation unit is
When evaluating the state of the evaluation target information processing apparatus for one evaluation target information processing apparatus among the plurality of information processing apparatuses, an average value of performance information collected from the evaluation target information processing apparatus is calculated. ,
An operation management apparatus, wherein occurrence of an abnormality is determined based on a comparison between the average value and the maximum value and the minimum value. An operation management method for an information processing system for executing a job in cooperation with a plurality of information processing apparatuses having any one of a plurality of predetermined characteristics,
An operation management device communicably connected to the plurality of information processing devices,
For each combination of characteristics that can be taken by two information processing devices among the plurality of information processing devices, store a threshold value regarding the correlation of the performance information of the two information processing devices,
Obtaining performance information from each of the plurality of information processing devices;
When evaluating the state of the information processing apparatus as the evaluation target for one information processing apparatus as the evaluation target among the plurality of information processing apparatuses,
For each of the plurality of information processing devices other than the information processing device to be evaluated, a correlation value of performance information with the information processing device to be evaluated is calculated,
Identify a combination of characteristics with the information processing apparatus to be evaluated, compare the threshold value with the calculated correlation value for the identified combination of characteristics,
An operation management method comprising evaluating a state of the information processing apparatus to be evaluated based on a result of the comparison. An operation management apparatus communicably connected to the information processing apparatus of the information processing system that executes a job in cooperation with a plurality of information processing apparatuses having any one of a plurality of predetermined characteristics.
For each combination of characteristics that can be taken by two information processing devices among the plurality of information processing devices, a procedure for storing a threshold value for correlation of performance information of the two information processing devices;
A procedure for acquiring performance information from each of the plurality of information processing devices;
When evaluating the state of the information processing apparatus as an evaluation target for one information processing apparatus as the evaluation target among the plurality of information processing apparatuses,
For each of the plurality of information processing devices other than the information processing device to be evaluated, a procedure for calculating a correlation value of performance information with the information processing device to be evaluated;
A procedure for specifying a combination of characteristics with the information processing apparatus to be evaluated, and comparing the calculated threshold value with the calculated correlation value for the specified combination of characteristics;
An operation management program for executing a procedure for evaluating the state of the information processing apparatus to be evaluated based on the result of the comparison.

Images