JP5747765B2 - Failure analysis apparatus, failure analysis method, and program - Google Patents

Description

  The present invention relates to a failure analysis apparatus, a failure analysis method, and a program used for operation management of a data center or the like that centrally manages a large number of virtual servers using virtualization technology.

  In recent years, computer systems using virtualization technology have been operated. In such a system, for example, a plurality of virtual machines are operated on one server computer (see, for example, Patent Document 1). The virtual computer system described in Patent Literature 1 sets a virtual device using a logical device on a plurality of physical devices, and operates a virtual machine using the virtual device.

  The virtual computer system has a virtual environment operation support system. The virtualization environment operation support system includes first related information that identifies a physical device in which a failure has occurred and a logical device that the physical device affects, a logical device in which a failure has occurred, and a virtual device that has an influence on the logical device. An influence range specifying table unit is provided that stores the specified second related information, and the third related information specifying the virtual device in which the failure has occurred and the virtual machine affected by the virtual device.

  The virtual environment operation support system includes a control unit. The control unit receives failure occurrence site information that identifies a physical device, logical device, or virtual device in which a failure has occurred from the virtual computer system, and refers to the influence range identification table unit based on the failure occurrence site information. Identify the virtual machine affected by the failure. In addition, when the failure occurrence site information is a virtual device, the control unit identifies a virtual machine that is affected by the virtual device with reference to the third related information.

JP 2010-009411 A (summary, claim 1)

  By the way, when performing failure monitoring of a server computer (cloud system) that makes heavy use of virtualization technology represented by VMWare (registered trademark), when a failure occurs, a plurality of failures may be detected despite originally having one cause. Often appear to have occurred at the same time. In such a case, it is not known from which trouble to deal with, and it takes time to identify the cause of the trouble and the affected range. Also, in a virtual environment, there are frequent changes in usage resources, and it is difficult to identify the affected range.

  In the configuration described in Patent Document 1, it is necessary to be able to identify the cause device from the failure event as a precondition for performing the process of identifying the affected range of the failure that has occurred in the virtual machine. That is, it is necessary to register in advance the failure event and the relationship with the causative device. Therefore, it is not possible to analyze the cause of a failure that has not been registered in advance.

  An example of an object of the present invention is to provide a failure analysis apparatus, a failure analysis method, and a program capable of analyzing a failure cause even if the relationship between the error content and the error cause cannot be specified in advance. It is in.

In order to achieve the above object, in one aspect of the present invention, a failure analysis apparatus includes:
Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit; and An error presence / absence determining unit that determines whether an error has occurred in another host that shares the identified physical device and the logical device;
It is characterized by having.

In order to achieve the above object, a failure analysis method according to one aspect of the present invention includes:
(A) obtaining information for specifying a physical device and a logical device used by each of a plurality of hosts including a virtual host;
(B) Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit And steps to
(C) determining whether an error has occurred in another host that shares the identified physical device and logical device;
It is characterized by including.

Furthermore, in order to achieve the above object, a program according to one aspect of the present invention is a program for analyzing failures occurring in a plurality of hosts including a virtual host by a computer,
(A) obtaining information for identifying a physical device and a logical device used by each of the plurality of hosts including the virtual host;
(B) Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit And steps to
(C) determining whether an error has occurred in another host that shares the identified physical device and logical device;
Is executed.

  As described above, according to the present invention, it is possible to analyze the cause of a failure even if the relationship between the error content and the cause of the error cannot be specified in advance.

FIG. 1 is a block diagram showing a schematic configuration of a failure analysis system including a failure analysis apparatus according to an embodiment of the present invention. FIG. 2 is a diagram showing a data table included in the configuration information storage unit and necessary for the operation in the present embodiment. FIG. 3 shows a host management table. FIG. 4 is a diagram showing a physical disk management table, a logical disk management table, and a NIC management table. FIG. 5 is a diagram showing a resource allocation table. FIG. 6 is a diagram illustrating an example of a message structure. FIG. 7 is a flowchart for explaining the flow of processing for narrowing down the cause of error. FIG. 8A is a flowchart for explaining the process performed by the generation node determination unit, and shows details of the process in step S2 of FIG. FIG. 8B is a flowchart for explaining the process performed by the message search range calculation unit, and shows details of the process in step S3 in FIG. FIG. 9 is a flowchart for explaining the process performed by the error presence / absence determination unit, and shows details of the process in step S4 of FIG. FIG. 10 is a flowchart for explaining the detailed processing flow of steps S301 to S303 in FIG. FIG. 11A is a flowchart showing the flow of processing performed by the suspicious object narrowing unit, and FIG. 11B is a flowchart showing the flow of processing performed by the sort processing unit. FIG. 12 is a flowchart for explaining steps S9 to S12.

  Hereinafter, a failure analysis apparatus according to an embodiment of the present invention will be described with reference to the drawings.

[Device configuration]
FIG. 1 is a block diagram showing a schematic configuration of a failure analysis system 1 including a failure analysis apparatus 100 according to an embodiment of the present invention. In the present embodiment, the failure analysis system 1 includes a failure analysis device 100, a monitoring target server 200, and a monitoring terminal 300 operated by an operator.

  The failure analysis device 100 can be constructed by, for example, a server computer, as will be described later. The monitoring target server 200 can be constructed by a server computer, for example. The monitoring target server 200 has a plurality of physical devices such as hard disks and logical devices. Moreover, the monitoring target server 200 has a plurality of hosts that use these physical devices and logical devices. The plurality of hosts includes a physical OS (Operating System), a hypervisor, and a virtual OS.

  The monitoring terminal 300 includes a computer. The failure analysis apparatus 100 and the monitoring target server 200 are connected via a network 400. In addition, the failure analysis apparatus 100 and the monitoring terminal 300 are connected via a network 400.

  The failure analysis apparatus 100 includes an acquisition unit 131 and an error presence / absence determination unit 153. The obtaining unit 131 obtains, from the monitoring target server 200, information on the physical devices used by each host of the monitoring target server 200 and information (device information) for specifying a logical device.

  The error presence / absence determination unit 153 uses the host name of the error-occurring host in which an error has occurred among the plurality of hosts of the monitoring target server 200 and the device used by the error-occurring host from the device information obtained by the obtaining unit 131. Identify devices and logical devices. The error presence / absence determination unit 153 further determines whether an error has occurred in another host that shares the specified physical device and logical device.

  As described above, in the present embodiment, it is possible to analyze the cause of the failure even if the relationship between the error content and the cause of the error cannot be specified in advance by the failure analysis apparatus 100. More specifically, if an error has occurred in the host where the error occurred and another host that shares at least one of the physical device and the logical device, the shared device may be analyzed as the cause of the error. it can. On the other hand, when an error has occurred in the host in which the error occurred, but no error has occurred in the other host, it can be analyzed that the error host itself is the cause of the error. As a result, the operator can know the cause of the error accurately, and can quickly start the error elimination work.

  The above is a schematic description of the failure analysis apparatus 100. Next, a more specific configuration of the failure cause location narrowing system 1 will be described with reference to FIGS. 2 to 6 in addition to FIG.

  As shown in FIG. 1, in the present embodiment, the monitoring target server 200 includes a configuration information acquisition unit 201 and an event storage unit 210. The configuration information acquisition unit 201 constantly monitors hardware configuration information and software configuration information configuring the monitoring target server 200, and acquires the configuration information including configuration change information. In general, a physical OS (Operating System) and a hypervisor manage physical device information and logical device information used by the physical OS. As an example of a specific method for acquiring the configuration information, using the API (Application Program Interface) provided on the monitoring target server 200 side, or operating the agent program to acquire the configuration information, Can be mentioned. In the present embodiment, the configuration information acquired by the configuration information acquisition unit 201 is stored in the configuration information storage unit 140 via the data analysis unit 132 and the table update unit 133. In the present embodiment, the event accumulation unit 210 of the monitoring target server 200 stores event information generated in the monitoring target server 200.

  In the present embodiment, monitoring terminal 300 is operated by an operator. The monitoring terminal 300 includes a message search unit 301, a request unit 310, a display control unit 313, and a display unit 314.

  In the present embodiment, the message search unit 301 searches, for example, one message from the messages stored in the message storage unit 110 described later according to the operation of the operator.

  In the present embodiment, request unit 310 includes a cause narrowing request unit 311 and a message list request unit 312. In the present embodiment, the cause narrowing request unit 311 is operated by an operator who viewed the message searched by the message search unit 301. The cause narrowing request unit 311 gives an instruction to narrow down the cause of the error message to the failure analysis apparatus 100. In the present embodiment, the message list request unit 312 is operated by an operator who has seen the result of the cause narrowing returned from the failure analysis apparatus 100 in response to the operation of the cause narrowing request unit 311. The message list request unit 312 gives the failure analysis apparatus 100 a request to create a list of results of searching for error messages.

  In the present embodiment, the display control unit 313 displays an image based on given data on the display screen of the display unit 314 such as a liquid crystal display.

  In the present embodiment, the failure analysis apparatus 100 includes a message monitoring unit 101, a message storage unit 110, a device information organizing unit 120, a processing unit 130, and a result output unit 111.

  In the present embodiment, the message monitoring unit 101 monitors an event stored in the event storage unit 201 of the monitoring target server 200, acquires the event as a message, and stores the acquired message in the message storage unit 110. .

  In the present embodiment, the device information organizing unit 120 includes an obtaining unit 131, a data analyzing unit 132, a table updating unit 133, and a configuration information accumulating unit 140.

  In the present embodiment, the configuration information monitoring unit 131 acquires configuration information such as hardware information and software information configuring the monitoring target server 200 from the configuration information acquisition unit 201.

  In the present embodiment, the information obtained by the configuration information monitoring unit 131 is collected for each information by the data analysis unit 132 and analyzed and classified. The analyzed data is registered in the configuration information storage unit 140 by the table update unit 133.

  FIG. 2 is a diagram illustrating data tables 141 to 145 included in the configuration information storage unit 140 and necessary for operations in the present embodiment. As shown in FIG. 2, in the present embodiment, the configuration information storage unit 140 stores a table indicating the relationship between physical devices, logical devices, and hosts. More specifically, the configuration information storage unit 140 includes a host management table 141, a physical disk management table 142, a logical disk management table 143, a NIC (Network Interface Card) management table 144, a resource allocation table 145, Is stored.

  FIG. 3 is a diagram showing the host management table 141. As shown in FIG. 3, in this embodiment, the host management table 141 includes a table 1411 for managing all host names, a table 1412 for managing host names that are hypervisors, and a virtual configured on the hypervisor. And a table 1413 for managing host names. Information in these tables 1411, 1412, and 1413 is related to each other.

  In this embodiment, the table 1411 stores a host ID and a host name corresponding to the host ID. The table 1411 stores, for example, the host name of the host ID 1 as VM1 and the host name of the host D2 as Guest2. Further, the table 1411 stores host names of host IDs 3, 4, 5, 6,... As Guest 2, VM 2, Guest 3, host 1,. The table 1412 stores a hypervisor ID and a host name corresponding to the hypervisor ID. The table 1412 stores, for example, the host name of the hypervisor ID 1 as VM1. The table 1413 stores a virtual host ID and a host name corresponding to the host ID. For example, the table 1413 stores the host name of the virtual host ID 1 as Guest 1 and stores the host name of the virtual host ID 2 as Guest 2.

  FIG. 4 is a diagram showing the physical disk management table 142, the logical disk management table 143, and the NIC management table 144. As shown in FIG. 4, the disk management table 142 includes a table 1421 for managing all the physical disks to be monitored and a table 1422 for managing the use host for each physical disk. Identify the physical disk

  In this embodiment, the table 1421 stores a physical disk ID and a disk name corresponding to the physical disk ID. The table 1421 stores, for example, the disk name of the physical disk ID1 as DiskA, the disk name of the physical disk ID2 as DiskB, and the disk name of the physical disk ID3 as DiskC. In this embodiment, the table 1422 stores a host ID that uses the physical Disk A and a host name corresponding to the host ID. For example, the table 1422 stores the host name of the host ID 1 in the physical disk A as VM1 and stores the host name of the host ID 4 in the physical disk A as VM2. 4 illustrates the host ID and host name corresponding to the physical disk A, the table 1422 also stores the host ID and host name corresponding to each of the physical disks B, C,. doing.

  In this embodiment, the logical disk management table 143 includes a table 1431 that stores logical disk names corresponding to the physical disks in the physical disk management table 1421. This table 1431 is linked to a table 1432 that stores the names of hosts using logical disks. This makes it possible to specify the logical disk being used from the host name. In this embodiment, the table 1431 stores a logical disk ID and a logical disk name corresponding to the logical disk ID. For example, the table 1431 stores the logical disk name of the logical disk ID 1 as logical A, stores the logical disk name of the logical disk ID 2 as logical B, and stores the logical disk name of the logical disk ID 3 as logical C. In FIG. 4, the logical disk corresponding to the physical disk A is shown as a table 1431. However, the table 1431 also stores information on the logical disks corresponding to the physical disks B, C,. doing.

  In the present embodiment, the NIC management table 144 is a table for managing all the physical NICs to be monitored, and includes a table 1441 and a table 1442 for managing the use host for each NIC. It is possible to identify the NIC being used from the name. In this embodiment, the table 1441 stores NICIDs and MAC (Media Access Control) address names corresponding to the NICIDs. For example, the table 1441 stores the MAC address of NICID1 as MAC1, stores the MAC address of NICID2 as MAC2, and stores the MAC address of NICID3 as MAC3. Although FIG. 4 illustrates the host ID and host name for MAC1, the table 1442 also stores host information corresponding to the MAC addresses of MAC2 and MAC3.

  FIG. 5 is a diagram showing the resource allocation table 145. In the present embodiment, as shown in FIG. 5, the resource allocation table 145 includes a hypervisor table 1451, a NIC table 1452, a logical disk table 1453, and a virtual host table 1454.

  In the present embodiment, the hypervisor table 1451 stores a hypervisor ID and a host name corresponding to the hypervisor ID. The hypervisor table 1451 stores, for example, the host name of the hypervisor ID1 as VM1. In the present embodiment, the NIC table 1452 stores NICID and a MAC address corresponding to the NICID. The NIC table 1452 stores, for example, MACID names of NICID1, NICID2, and NICID3 as MAC1, MAC2, and MAC3, respectively. In this embodiment, the logical disk table 1453 stores a logical disk ID and a logical disk name corresponding to the logical disk ID. The logical disk table 1453 stores, for example, logical disk names of logical disks ID1, ID2, and ID3 as logical A, logical B, and logical C, respectively. In this embodiment, the virtual host table 1454 stores a virtual host ID and a host name corresponding to the virtual host ID. The virtual host table 1454 stores, for example, the host name of the virtual host ID as Guest1.

  In the present embodiment, in the resource allocation table 145, a virtual host table 1454 including resource information allocated to each virtual host by the hypervisor, a NIC table 1452, and a logical disk table 1453 are associated with each other. Thereby, it is possible to specify the assigned device information from the virtual host name. Each of the NIC table 1452 and the logical disk table 1453 is related to the hypervisor table 1451. Thereby, the hypervisor used by the host can be specified from the host name.

  Next, as illustrated in FIG. 1, a processing unit 130 that performs cause narrowing in this embodiment will be described. In the present embodiment, the processing unit 130 includes a generation node determination unit 151, a message extraction range calculation unit 152, an error presence / absence determination unit 153, a suspicious object narrowing unit 154, and a sort processing unit 155. . In the present embodiment, processing unit 130 is connected to each of configuration information storage unit 140 and message storage unit 110.

  In the present embodiment, the generation node determination unit 151 determines whether or not the message generation source is a virtual host. The message search range calculation unit 152 determines, based on the time of occurrence of the error selected by the operator, how long a message before and after the event occurs is to be searched. The error presence / absence determination unit 153 determines whether or not an error message has occurred in the generation node or the like within the extracted period. In the present embodiment, the suspected object narrowing unit 154 narrows down suspected devices from the ratio of the number of hosts that have issued error messages. The sort processing unit 155 rearranges the output data from the suspected object narrowing unit 154.

  In the present embodiment, the processing unit 150 outputs the data sorted by the sort processing unit 155 to the display control unit 313 of the monitoring terminal 300 via the network 400 by the result output unit 111. The display control unit 313 displays the received data on the display unit 314 of the monitoring terminal 300. As a result, the failure analysis apparatus 100 returns the result of the operation of the request unit 310 to the operator who uses the monitoring terminal 300. In the present embodiment, in response to a request from the cause narrowing request unit 311, processing of the occurrence node determination unit 151, the message extraction range calculation unit 152, the error presence / absence determination unit 153, the suspected target narrowing unit 154, and the sort processing unit 155 is executed. Is done. The request of the cause narrowing request unit 311 is given to the generation node determination unit 151.

  In the present embodiment, the processing unit 130 further includes a request target determination unit 161 and a message search unit 162. The request target determination unit 161 specifies a message request target host. The message search unit 162 searches the message storage unit 110 for a message about a specific host. The sort processing unit 155 sorts the retrieved messages according to a predetermined rule. In the present embodiment, processing related to the message is executed by a request from the message list request unit 312. The request from the message list request unit 312 is given to the request target determination unit 161.

  As described above, the monitoring terminal 300 includes the message search unit 301. The message search unit 301 is connected to the message storage unit 110 via the network 400. The message search unit 301 can refer to an arbitrary message by searching for a message stored in the message storage unit 110. An example of the structure of messages stored in the message storage unit 110 is as shown in FIG.

  FIG. 6 is a diagram illustrating an example of a message structure. In the present embodiment, the message stored in the message storage unit 110 includes the generation node 1101, the message ID 1102, the message content 1103, the alert level 1104, the generation date 1105, and the generation time 1106.

  The generation node 1101 indicates the name of the host where the event has occurred. The message content 1103 indicates specific event content (error content). The alert level 1104 indicates “Error” for an event in which an error has occurred. The occurrence date 1105 indicates the date of occurrence of the event. The occurrence time 1106 indicates the time when the event occurred.

[Description of operation in this embodiment]
[Overview of operation]
As shown in FIG. 1, in the present embodiment, when an error occurs in the monitoring target server 200, the operator operates the cause narrowing request unit 311 of the monitoring terminal 300 to display a message that is determined to be related to the error. Select one. Based on this, the failure analysis apparatus 100 associates the error occurrence host that generated the message with the device information. Then, the failure analysis apparatus 100 extracts one or a plurality of hosts sharing the device with the host that has generated the error message.

  Furthermore, the failure analysis apparatus 100 determines whether an error has occurred in the shared host near the time of the error occurrence. Then, the failure analysis apparatus 100 creates a list of suspected devices suspected of causing the error and hosts that use the suspected device based on the determination result of determining whether or not an error has occurred. Then, the failure analysis apparatus 100 returns the created host list to the monitoring terminal 300. When there is a message list request for the suspected device and the host using the suspected device, based on the operation of the message list requesting unit 312, the failure analysis apparatus 100 searches the message storage unit 110. Then, the failure analysis apparatus 100 searches for a suspected device or a message of a host that uses the device, and outputs the search result to the monitoring terminal 300.

[Detailed description of operation in this embodiment]
Next, details of the operation of the failure analysis apparatus 100 according to the present embodiment will be described with reference to FIGS. FIG. 7 is a flowchart for explaining the flow of processing for narrowing down the cause of error. Moreover, in the following description, FIGS. 1-6 is suitably referred. In the present embodiment, the failure analysis method is performed by operating the failure analysis apparatus 100. Therefore, the description of the failure analysis method in the present embodiment is replaced with the following description of the operation of the failure analysis apparatus 100.

  As shown in FIG. 7, in the present embodiment, the failure analysis apparatus 100 determines whether there is a cause narrowing request for one arbitrary error message (step S1). For example, when a cause narrowing request for one arbitrary error message is issued by operating the cause narrowing request unit 311 (YES in step S1), the occurrence node determination unit 151 performs processing (step S2). Next, the message search range calculation unit 152 performs processing (step S3), and thereafter, the error presence / absence determination unit 153, the suspicious object narrowing unit 154, and the sort processing unit 155 perform processing in order (steps S4, S5, and S6). . The result output unit 111 outputs the estimation result of the suspected cause of failure sorted by the sort processing unit 155 to the display control unit 313 of the monitoring terminal 300 via the network 400 (step S7). Thereby, the estimation result of the suspected object is displayed on the display unit 314.

  Next, when a search request for the suspected object is issued to the failure analysis apparatus 100 by operating the message list request unit 312 by the operator who has seen the estimation result of the suspected object device (YES in step S8), the request is made. The object determination unit 161, the message search unit 162, and the sort processing unit 155 perform processing in order (steps S9, S10, and S11). The result output unit 111 outputs the list of messages related to the suspected object sorted by the sort processing unit 155 to the display control unit 313 of the monitoring terminal 300 via the network 400 (step S12). Thereby, the message list is displayed on the display unit 314.

  As described above, in the present embodiment, failure cause narrowing-down processing is roughly divided into two stages. The first-stage process is a process (steps S2 to S6) for enumerating a possible device and a related host name from one error message. The second stage process is a process (steps S9 to S11) for performing a message search for the listed objects. First, the former process (steps S2 to S6) will be described with reference to FIGS.

  FIG. 8A is a flowchart for explaining the process performed by the generation node determination unit 151, and shows details of the process in step S2 of FIG. FIG. 8B is a flowchart for explaining the process performed by the message search range calculation unit 152, and shows details of the process in step S3 of FIG. FIG. 9 is a flowchart for explaining the process performed by the error presence / absence determination unit 153, and shows details of the process in step S4 of FIG.

  As illustrated in FIG. 7, for example, when the operator operates the cause narrowing request unit 311 of the monitoring terminal 300, the request from the cause narrowing request unit 311 to the fault analysis apparatus 100 to process one error message. 8a, as shown in FIG. 8A, the generation node determination unit 151 includes the host name of the generation node 1101 of the designated message stored in the message storage unit 110, and the configuration information storage unit 140. The host management table 141 is checked to obtain a host name that matches the host name of the source node 1101 (step S101). Next, the generation node determination unit 151 determines whether or not the acquired host name includes the host name of the virtual host table 1413 (step S102). When the acquired host name includes a virtual host ID (YES in step S102), the generation node determination unit 151 determines that the message generation node is a virtual host (step S103). On the other hand, if the acquired host name does not include a virtual host ID (NO in step S102), it is determined that the message generation node is a physical host or a hypervisor (step S104).

  As shown in FIG. 8B, the message search range calculation unit 152 sets the error based on the occurrence date 1105 and the occurrence time 1106 in the error message for which the cause narrowing request is made, that is, based on the error occurrence status. The presence / absence determination unit 153 calculates a message acquisition range as an error determination target (step S201). Although the method for specifying the message acquisition range is not particularly limited, in this embodiment, messages that occur within a period of several seconds or several tens of seconds before and after the occurrence time of the error message that is narrowing down the cause are targeted for the search range. To do. As a method for specifying the message acquisition range, dozens of messages generated before and after the error message generation time may be set as the search target range.

  Next, as shown in FIG. 9, the error presence / absence determination unit 153 searches the configuration information storage unit 140 to extract device information used by the host of the specified generation node 1101 (step S301). .

  Next, the error presence / absence determination unit 153 extracts all the hosts that share the logical device with the host of the designated generation node 1101 (step S302). Next, the error presence / absence determination unit 153 sends messages corresponding to the message search range calculated in step S201 to the message storage unit 110 for all hosts associated with the logical device used by the host of the specified generation node 1101. Search from. Then, the error presence / absence determination unit 153 determines whether or not an error has occurred during the period, that is, whether or not a message with an alert level 1104 has occurred (step S303).

  Next, the error presence / absence determining unit 153 performs the processes of steps S304 and S305. The processes in step S304 and step S305 are the same as the search process for logical devices (steps S302 and S303), and the search process for physical devices is performed.

  Specifically, the error presence / absence determination unit 153 extracts all the hosts that share the physical device with the host of the designated generation node 1101 (step S304). Next, the error presence / absence determination unit 153 sends messages corresponding to the message search range calculated in step S201 for all hosts associated with the physical device used by the host of the specified generation node 1101 to the message storage unit 110. Search from. Then, the error presence / absence determination unit 153 determines whether or not an error has occurred during the period, that is, whether or not a message with an alert level 1104 has occurred (step S305).

  Next, the error presence / absence determination unit 153 confirms whether or not the host of the specified generation node 1101 is a virtual host (step S306). Whether or not the generation node 1101 is a virtual host is determined in advance in the generation node determination unit 151. If the error presence / absence determination unit 153 confirms that the generation node 1101 is a virtual host (YES in step S306), the process proceeds to step S307. In step S307, the error presence / absence determination unit 153 identifies the hypervisor serving as the base of the virtual host from the configuration information storage unit 140, and regards the hypervisor as an error occurrence node. Next, the error presence / absence determination unit 153 extracts the device information used by the hypervisor by searching the configuration information storage unit 140 in the same manner as the processing in step S301 (step S308). Next, the error presence / absence determination unit 153 determines the error occurrence status by performing the processing of steps S309 to S312.

  Note that the processes in steps S309 and S310 are the same as the processes in steps S302 and S303, respectively. Further, the processes in steps S311 and S312 are the same as the processes in steps S302 and S303, respectively.

  Specifically, the error presence / absence determination unit 153 extracts all the hosts that share the logical device with the hypervisor that is the basis of the virtual host (step S309). Next, the error presence / absence determination unit 153 searches the message storage unit 110 for messages corresponding to the message search range calculated in step S201 for all the hosts associated with the logical devices used by the hypervisor. Then, the error presence / absence determination unit 153 determines whether or not an error has occurred during the period, that is, whether or not a message with an alert level 1104 has occurred (step S310).

  Next, the error presence / absence determination unit 153 extracts all hosts sharing the physical device with the hypervisor (step S311). Next, the error presence / absence determination unit 153 searches the message storage unit 110 for messages corresponding to the message search range calculated in step S201 for all hosts associated with the physical device used by the hypervisor. Then, the error presence / absence determination unit 153 determines whether or not an error has occurred during the period, that is, whether or not a message having an alert level 1104 is generated (step S312).

  Next, a detailed processing flow for steps S301 to S303 illustrated in FIG. 10 will be described. FIG. 10 is a flowchart for explaining the detailed processing flow of steps S301 to S303 in FIG.

  Step S3001 is the same process as step S301, and the error presence / absence determination unit 153 searches the configuration information storage unit 140 to search for the logical device and physical device used by the host of the specified generation node 1101. Extract configuration information. Next, in step S3002, the error presence / absence determination unit 153 stores configuration information as to whether there is a host associated with the list of logical devices used by the host of the designated generation node 1101 acquired in step S3001. The determination is made with reference to the unit 140 (step S3002).

  If there is no host that uses a logical device other than the host of the specified generation node 1101 (NO in step S3002), the error presence / absence determination unit 153 proceeds to step S3008. On the other hand, when there is a host using a logical device other than the host of the generation node 1101 (YES in step S3002), the error presence / absence determination unit 153 shares all the logical devices with the specified host of the generation node 110. Are extracted (step S3003). Next, the error presence / absence determination unit 153 searches the message storage unit 110 for a message corresponding to the message search target period calculated in step S201 for one of the extracted hosts (step S3004). Next, the error presence / absence determination unit 153 determines whether or not one or more of the retrieved messages whose alert level 1104 is “Error” exist (step S3005). If one or more alert levels 1104 are “Error” (YES in step S3005), the error presence / absence determination unit 153 adds one count value (step S3006). On the other hand, when there is no alert level 1104 with “Error” (NO in step S305), the error presence / absence determination unit 153 does not add the count value.

  Next, the error presence / absence determination unit 153 determines whether there is a host that has not been subjected to the processing of steps S3004 to S3006 among the hosts that share the logical device with the host of the specified generation node 1101. (Step S3007). Of the hosts sharing the logical device with the host of the specified generation node 1101, if there is a host that has not been subjected to the processing of steps S3004 to S3006 (YES in step S3007), an error presence determination unit 153 Shifts the message search target host to the host (step S3008). Then, the error presence / absence determination unit 153 repeats the processing of steps S3004 to S3006 for the host.

  On the other hand, when the processing of steps S3004 to S3006 has been performed for all of the hosts that share the logical device with the host of the specified generation node 1101 (NO in step S3007), the error presence / absence determination unit 153 determines whether or not step S3009 Proceed to

  In step S3009, the error presence / absence determination unit 153 counts step S3006 for the total number of hosts on the logical device including the host of the specified generation node 1101 and the other host sharing the logical device with the host. The ratio of the value (number of error-occurring hosts) is calculated as the error occurrence ratio. For example, if the number of hosts extracted in step S3003 is 3, and the count value added in step S3006 is 3, the calculated value in step S3009 is 3 / 4≈0.8. After the above ratio calculation, the error presence / absence determination unit 153 proceeds to step S3010. In step S3010, the error presence / absence determination unit 153 determines whether there is a logical device that has not been processed in steps S3002 to S3009 among the logical devices extracted in step S3001.

  If there is a logical device for which the processes in steps S3002 to S3009 have not been performed, the error presence / absence determination unit 153 shifts another logical device for which the above process has not been performed to a processing target (step S3011), The processes of S3002 to S3009 are repeated. On the other hand, when there is no logical device for which the processes of steps S3002 to S3009 are not performed (NO in step S3010), the error presence / absence determining unit 153 ends the process.

  The processing in steps S304 to S305 shown in FIG. 9 is the same as that in the case where “logical device” in the processing in steps S3002 to S3011 shown in FIG. 10 is replaced with “physical device”. Is omitted.

  Further, the processing of steps S308 to S310 shown in FIG. 9 is the same as the processing when “host of the generation node 1101” in the processing of steps S30001 to S3011 shown in FIG. 10 is replaced with “hypervisor”. Description is omitted. Further, in the processing of steps S311 to S312 shown in FIG. 9, “host of the generation node 1101” in the processing of steps S3002 to S3011 shown in FIG. 10 is replaced with “hypervisor”, and “logical device” is replaced with “physical device”. Since the processing is the same as that in the case of replacing with, detailed description is omitted.

  Next, the flow of processing in the suspicious object narrowing unit 154 and the sort processing unit 155 shown in FIGS. 11A and 11B will be described. FIG. 11A is a flowchart illustrating a flow of processing performed by the suspicious object narrowing unit 154, and FIG. 11B is a flowchart illustrating a flow of processing performed by the sort processing unit 155.

  As shown in FIG. 11A, the suspicious object narrowing unit 154 determines the error cause based on the error occurrence ratio in each logical device and each physical device calculated in step S3009 (see FIG. 10). A criterion for narrowing down the suspected device is determined (step S401). In the present embodiment, the narrowing criteria is, for example, that each of the logical device and each physical device has the highest error occurrence ratio value as the top five cases, or the error occurrence ratio is 50%. Anything beyond that could be considered a suspected device.

  After determining the criteria for narrowing down, the suspected subject narrowing-down unit 154 collates the error occurrence rate and the narrowing-down criteria for each logical device and each physical device. Then, the suspected object narrowing unit 154 extracts devices that satisfy the criteria as suspected devices (step S402). Thereafter, the suspected object narrowing unit 154 obtains a list of all host names that use the suspected device by searching the configuration information accumulating unit 140 (step S403), and ends the process.

  Next, as shown in FIG. 11B, the sort processing unit 155 returns the information on all the host names of the suspected device obtained by the suspected target narrowing unit 154 to the operator who uses the monitoring terminal 300. Then, the information is organized (step S501). The information returned to the monitoring terminal 300 is the suspected device, all host names of the suspected device, and the error occurrence rate of each suspected device. The sort processing unit 155 sorts these pieces of information. The sorted results are output to the result output unit 111 as shown in FIG. The result output unit 111 outputs these pieces of information to the display control unit 313 of the monitoring terminal 300 via the network (step S7). Thus, the above information is displayed on the display unit 314, and the operator can confirm the display contents.

  Next, the flow after steps S8 to S12, that is, the process after the suspected target device estimated as the cause of error is extracted will be described with reference to FIG. FIG. 12 is a flowchart for explaining steps S9 to S12.

  As shown in FIG. 7, when a message search request is issued by the operator operating the message list request unit 312 based on the information displayed on the display unit 314 of the monitoring terminal 300 (YES in step S8). ), Step S601 in the request target determination unit 161 is started as shown in FIG. Specifically, the request target determination unit 161 determines whether the target of the message search request is a device name or a host name. The target of the message search request is specified, for example, when the operator operates the message list request unit 312.

  If there is a message search request for the suspect device (the suspect device in step S601), the message search unit 162 searches the message storage unit 110 for all hosts sharing the suspect device, and extracts the corresponding message. The process proceeds to step S801.

  On the other hand, when the target of the message search request by the operator's operation of the message list request unit 312 is a host (host in step S601), the message search unit 162 searches the message storage unit 110 and extracts the corresponding message. Then, the process proceeds to step S801.

  In step S801, the sort processing unit 155 arranges the messages obtained in step S701 or S702 for each occurrence node. Note that step S601 corresponds to step S9 in FIG. 7, steps S701 and S702 correspond to step S10 in FIG. 7, and step S801 corresponds to step S11 in FIG.

  Subsequent to the processing of the sort processing unit 155, step S12 shown in FIG. 7 is executed. That is, the message list sorted in step S801 is output to the result display unit 111, and the result output unit 111 outputs the message list to the display control unit 313 via the network 400 (step S12). Thereby, the operator using the monitoring terminal 300 can check the message list displayed on the display unit 314 by the display control unit 313.

  The program according to the embodiment of the present invention is stored in the computer in steps S1 to S12, S101 to S104, S201, S301 to S312, S3001 to S3011, S401 to S403, S501, S601, S701, and S702 shown in FIGS. , And any program that executes S801. By installing and executing this program on a computer, the failure analysis apparatus 100 according to the present embodiment can be realized. In this case, the CPU (Central Processing Unit) of the computer includes a message monitoring unit 101, a result output unit 111, an acquisition unit 131, a data analysis unit 132, a table update unit 133, a generation node determination unit 151, a message extraction range calculation unit 152, It functions as an error presence / absence determination unit 153, a suspicious object narrowing unit 154, a sort processing unit 155, a request target determination unit 161, and a message search unit 162 to perform processing.

  Further, in the present embodiment, the message storage unit 110 and the configuration information storage unit 140 can be realized by storing data files constituting them in a storage device such as a hard disk provided in the computer. Further, the message storage unit 110 and the configuration information storage unit 140 may be constructed by different computers.

  As described above, according to the present embodiment, the cause of the failure can be analyzed even if the relationship between the error content and the cause of the error cannot be specified in advance by the failure analysis apparatus 100. Specifically, the failure analysis apparatus 100 collects messages such as error messages from the monitoring target server 200, and the device information organizing unit 120 of the failure analysis apparatus 100 has a host (physical OS, hypervisor, virtual OS). Manages device information to be used. Then, the failure analysis apparatus 100 extracts a plurality of hosts sharing the device with the error-occurring host by associating the error-generating host with the device information for one arbitrarily selected error message. . Further, the failure analysis apparatus 100 determines whether an error has occurred in a plurality of hosts sharing the device near the time of occurrence of the focused error. Then, the cause of the error is analyzed based on the error occurrence rate in the plurality of hosts, and the suspected device is specified. This makes it possible to narrow down the cause of errors from the viewpoint of device failure.

  With such a configuration of the present embodiment, it is possible to easily pursue the cause of the occurrence of an error event in a system in which software and physical devices in a virtual environment are configured in a complex manner. More specifically, the device being used is determined from the host name included in the specific error event using the configuration information of the physical device and logical device used by the host. Then, the rate at which a failure occurs at the same time in another host sharing the device is determined. Thereby, it is possible to narrow down whether it is a chain fault based on a specific device or a fault occurring in the host itself.

  Further, in this embodiment, the error presence / absence determination unit 153 is based on the error selected by the monitoring terminal 300, and the other host sharing the device with the host of the error occurrence node within a predetermined range from the reference. It is determined whether an error has occurred. As a result, it is possible to identify other errors that have occurred at the same time as the error selected by the monitoring terminal 300, so that an analysis material suitable for the cause of the error can be obtained, and the cause of the error can be analyzed more accurately. be able to.

  In this embodiment, when the error occurrence host is a virtual host, the other host sharing the physical device and the logical device used by the hypervisor of the error occurrence host is specified, and the specified host is specified. Extract errors. Thereby, even when the error occurrence host is a virtual host, the cause of the error can be specified more accurately.

  In the present embodiment, the suspicious object narrowing unit 154 determines that an error occurs when the number of hosts in which an error has occurred among other hosts sharing the device with the error-occurring host exceeds a predetermined standard. It is determined that an error has occurred in the device used by the generating host. As a result, the device causing the error can be analyzed more accurately.

(Modification)
In the above embodiment, the physical disk, logical disk, and physical NIC of the monitoring target server 200 are illustrated as devices managed by the configuration information storage unit 140. And although it is set as the structure which narrows down the cause of an error by correlating these devices and a host name, it is not limited to this. For example, in addition to the disk and NIC, the configuration information acquisition unit 210 of the monitoring target server 200 is a physical or logical device that can be provided by an API, and is a device shared by a plurality of hosts (partial resource allocation is possible) Can be managed by the configuration information storage unit 140.

  8B, the error message search range may be set by operating the monitoring terminal 300, or the failure analysis apparatus 100 is provided with a device for setting the calculation range determination pattern. Therefore, it may be set as appropriate.

  Further, in step S401 of FIG. 11, the narrowing criteria for determining the suspected device may be set by operating the monitoring terminal 300, or a device for setting the narrowing criteria pattern is provided in the failure analysis apparatus 100. Therefore, it may be set as appropriate.

  Part or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 12) described below, but is not limited to the following description.

(Appendix 1)
An obtaining unit for obtaining information for identifying a physical device and a logical device used by each of a plurality of hosts including a virtual host;
Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit; and An error presence / absence determining unit that determines whether an error has occurred in another host that shares the identified physical device and the logical device;
A failure analysis apparatus comprising:

(Appendix 2)
The failure analysis device according to appendix 1, wherein the error presence / absence determination unit determines whether an error has occurred in another host based on the occurrence state of the error that has occurred in the error-occurring host.

(Appendix 3)
When the error occurrence host is the virtual host, the error presence / absence determination unit identifies the other physical host that shares the physical device and the logical device used by the hypervisor of the error occurrence host, and The failure analysis apparatus according to appendix 1 or appendix 2, which extracts an error of the identified host.

(Appendix 4)
A suspicious object narrowing section for narrowing down the physical device and the logical device in which an error has occurred;
The suspected object narrowing unit is used by the error-occurring host when the number of hosts in which the error has occurred exceeds a predetermined standard among the error-occurring host and the other hosts. The failure analysis apparatus according to any one of appendix 1 to appendix 3, wherein it is determined that a failure has occurred in at least one of the physical device and the logical device.

(Appendix 5)
(A) obtaining information for specifying a physical device and a logical device used by each of a plurality of hosts including a virtual host;
(B) Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit And steps to
(C) determining whether an error has occurred in another host that shares the identified physical device and logical device;
A failure analysis method comprising:

(Appendix 6)
In the step of determining whether or not the error has occurred, it is determined whether or not an error has occurred in another host based on the occurrence state of the error that has occurred in the error-occurring host. Failure analysis method described in 1.

(Appendix 7)
In the step of determining whether or not the error has occurred, if the error-occurring host is the virtual host, the physical device and the logical device used by the hypervisor of the error-occurring host are shared. The failure analysis method according to appendix 5 or appendix 6, wherein the other host is identified and an error of the identified host is extracted.

(Appendix 8)
Further comprising the step of narrowing down the physical device and the logical device in which an error has occurred,
In the narrowing-down step, the physical number used by the error-occurring host when the number of hosts in which an error has occurred exceeds a predetermined standard among the error-occurring host and other hosts. The failure analysis method according to any one of appendix 5 to appendix 7, wherein it is determined that a fault has occurred in at least one of the device and the logical device.

(Appendix 9)
A program for analyzing failures occurring in a plurality of hosts including a virtual host by a computer,
(A) obtaining information for specifying a physical device and a logical device used by each of a plurality of hosts including a virtual host;
(B) Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit And steps to
(C) determining whether an error has occurred in another host that shares the identified physical device and logical device;
A program that executes

(Appendix 10)
In the step of determining whether or not the error has occurred, it is determined whether or not an error has occurred in another host based on the occurrence status of the error that has occurred in the error-occurring host. The program described in.

(Appendix 11)
In the step of determining whether or not the error has occurred, if the error-occurring host is the virtual host, the physical device and the logical device used by the hypervisor of the error-occurring host are shared. The program according to appendix 9 or appendix 10, which identifies another host and extracts an error of the identified host.

(Appendix 12)
Further comprising the step of narrowing down the physical device and the logical device in which an error has occurred,
In the narrowing-down step, the physical number used by the error-occurring host when the number of hosts in which an error has occurred exceeds a predetermined standard among the error-occurring host and other hosts. The program according to any one of appendix 9 to appendix 11, which determines that a failure has occurred in at least one of a device and the logical device.

  The present invention can be applied to a failure analysis apparatus, a failure analysis method, and a program used for operation management of a data center or the like that centrally manages a large number of virtual servers using a virtualization technique.

100 Failure Analyzer 131 Obtaining Unit 153 Error Presence Determination Unit 154 Suspicious Target Narrowing Unit

Claims (6)

An obtaining unit for obtaining information for identifying a physical device and a logical device used by each of a plurality of hosts including a virtual host;
Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit; and An error presence / absence determining unit that determines whether an error has occurred in another host that shares the identified physical device and the logical device;
A failure analysis apparatus comprising:   The failure analysis apparatus according to claim 1, wherein the error presence / absence determination unit determines whether an error has occurred in another host based on the occurrence state of the error that has occurred in the error-occurring host.   When the error occurrence host is the virtual host, the error presence / absence determination unit identifies the other physical host that shares the physical device and the logical device used by the hypervisor of the error occurrence host, and The failure analysis apparatus according to claim 1, wherein an error of the identified host is extracted. A suspicious object narrowing section for narrowing down the physical device and the logical device in which an error has occurred;
The suspected object narrowing unit is used by the error-occurring host when the number of hosts in which the error has occurred exceeds a predetermined standard among the error-occurring host and the other hosts. The failure analysis apparatus according to claim 1, wherein it is determined that a failure has occurred in at least one of the physical device and the logical device. (A) obtaining information for specifying a physical device and a logical device used by each of a plurality of hosts including a virtual host;
(B) Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit And steps to
(C) determining whether an error has occurred in another host that shares the identified physical device and logical device;
A failure analysis method comprising: A program for analyzing failures occurring in a plurality of hosts including a virtual host by a computer,
In the computer,
(A) obtaining information for identifying a physical device and a logical device used by each of the plurality of hosts including the virtual host;
(B) Identifying the physical device and the logical device used by the error-occurring host from the host name of the error-occurring host among the plurality of hosts and the information obtained by the obtaining unit And steps to
(C) determining whether an error has occurred in another host that shares the identified physical device and logical device;
A program that executes

Images