JP2013222313A - Failure contact efficiency system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that since it is necessary to analyze the content of a failure first in order to determine a customer to be contacted with in accordance with the occurrence of a failure in a cloud computing service, a time required until making contact with the customer is increased.SOLUTION: When a failure occurs, a failure which is likely to be spread in the future due to the failure is predicted, and customer contact content is prepared. When the prediction of the failure is correct, the prepared contact content is used to quickly make contact.

Description

The present invention relates to a failure communication efficiency system for customer contact when dealing with a failure in operation management of a computing system.

In cloud computing services, multiple users are affected in the event of a failure due to the high concentration of virtualized physical servers. For this reason, cloud computing service providers need to efficiently communicate faults to their users.
As an efficient method for contacting users, in Patent Document 1, when a problem occurs, a contact address is specified from a problem event registered in advance, and a problem occurrence notification is made.

JP2007-141007

Patent Document 1 is intended for a case where the system is independent for each customer, and when a failure occurs, which customer is to be contacted is unique. In addition, a dedicated operation worker was assigned to each customer, and it was possible to immediately contact the customer once the contact details were determined.
However, the cloud computing service operates a large number of customer systems on a large-scale, virtualized cloud computing system without having a dedicated worker. Therefore, in order to determine the customer to be contacted when a certain failure occurs, it is necessary to analyze the content of the failure first, but this process increases the time required to contact the customer. There is a problem.
In addition, when it is necessary to contact multiple customers as a result of a certain failure, the number of customers who are smaller than the number of customers in the contact information will be handled, resulting in the time required to contact the customer. There is a problem that increases.

In the present invention, when a failure occurs, a failure that may possibly spread in the future due to the failure is predicted, and preparations are made for the contents to be notified to the user. If there is a prediction, we will contact you promptly using the contact details prepared in advance.
In addition, for the occurrence of simultaneous failures, priority is determined by the system based on parameters of user / system characteristics.

Since the content of communication to the user is based on the same obstacles, it is possible to prepare efficiently and to shorten the time to complete the communication to the user.
In addition, the service level in the entire cloud environment can be satisfied more widely by contacting the users in the order of priority.

It is a block diagram of the failure communication efficiency improvement system by Example 1. FIG. 6 shows a flow of processing when a failure occurs, based on the passage of time when the failure occurs according to the first embodiment. 4 is a detailed diagram of data stored in the storage 120 and the storage 121 according to the first embodiment. FIG. 3 is a detailed diagram of data stored in a storage 122, a storage 123, and a storage 124 according to the first embodiment. FIG. FIG. 3 is a detailed view of data exchanged between systems in the first embodiment. 5 is a flowchart showing details of processing procedures of a server monitoring system 110, a root cause analysis system 111, a spread destination prediction system 112, and a customer contact / priority determination system 113 based on a failure occurrence according to the first embodiment. 10 is a flowchart showing details of a processing procedure of the spread destination prediction system 112 based on the occurrence of a failure according to the first embodiment. 10 is a flowchart showing details of a processing procedure for determining a customer contact text in a customer contact / priority determination system 113 based on the occurrence of a failure according to the first embodiment. 6 is a flowchart showing details of a processing procedure for determining a customer contact priority of a customer contact / priority determination system 113 based on a failure occurrence according to the first embodiment. 6 is a flowchart showing details of a processing procedure for determining whether a customer contact / priority determination system 113 determines whether a customer contact message is determined based on the occurrence of a failure according to the first embodiment. It is the figure which showed the specific example of the determination method of the customer contact priority of the customer contact / priority determination system 113 based on the failure occurrence by Example 1. FIG.

  FIG. 1 is a first embodiment showing a system for improving the efficiency of failure communication to users in consideration of a multi-tenant environment to which the present invention is applied. The data center 100 includes a plurality of physical servers, management routers, routers, and storages. Virtual machines are running on multiple physical servers.

  In the present embodiment, the server monitoring system 110, the failure root cause analysis system 111, the spread destination prediction system 112, and the customer contact / priority determination system 113 will be described using examples implemented by virtual servers. Each virtual server and the management router 103 are connected by Ethernet (registered trademark), and communicate Ethernet packets between the servers via the virtual NIC connected to each virtual server. Each virtual server is assigned an IP address, and each virtual server can specify a transmission / reception destination by the IP address. All data sent and received between systems on each virtual server is converted into Ethernet packets and sent to and received from the target system. Here, the IP address of the target system can be defined in advance in the .hosts file, or it can be confirmed by inquiring to the DNS system that converts the IP address from the name of each system. Such a data transmission / reception mechanism using Ethernet is common in data transmission / reception between the following systems, and the above description is omitted unless otherwise specified.

  The server monitoring system 110 collects logs and the like from the devices in the data center 100 (physical servers 130 and 135, hypervisors 131 and 136, virtual servers 132, 133, 134, 137, 138, and 139). This is a monitoring system.

  The server monitoring system 110 installs devices in the data center 100 and performs monitoring. However, the devices in the data center 100 may be monitored remotely from outside the data center. The server monitoring system uses a network and connects to the physical servers 130 and 135, the hypervisors 131 and 136, and the virtual machines 132, 133, 134, 137, 138, and 139 via the management router 103 in the data center 100. Has been. The server monitoring system is connected to the server operation information 120 using a network, and is connected to the failure root cause analysis system 111, the spread destination prediction system 112, the customer contact / priority determination system 113, and the Web portal 114. Has been.

  As a means for the server management system 110 to collect logs from these devices, a mechanism for receiving data transmitted from these devices such as an SNMP trap may be used, or an API of these devices may be used. Thus, a mechanism for actively reading out may be used.

  The data collected from these devices corresponds to information for determining the device status such as a failure. From the physical servers 130 and 135, information such as IP addresses, CPU / memory / network usage rates, the number of IOs per second to storage, and device failures may be collected. From the hypervisors 131 and 136, information on virtual networks and virtual CPUs assigned to virtual machines, and information on failures that have occurred on these virtual devices may be collected. From virtual machines 132, 133, 134, 137, 138, and 139, information such as the usage rate of virtual CPU, virtual memory, and virtual network, the number of IOs per second to virtual storage, and failure of virtual devices are collected. May be.

  Here, the number of physical servers in the data center 100, the number of hypervisors operating on the physical servers, types, presence / absence, the number of virtual servers operating on the hypervisors may be any number, and the server monitoring system 110 Logs may be collected from these devices using multiple subsystems.

  The server monitoring system 110 checks whether there are any abnormalities in the physical servers 130 and 135, the hypervisors 131 and 136, the virtual machines 132, 133, and 134, 137, and 139, and the storage 105 in the cloud computing system 101. Always make sure. Implementation of the server monitoring system 110 may include hardware or hardware and software in addition to software.

  The failure root cause analysis system 111 refers to the server operation information 120 and the configuration management information 121 based on the error message detected by the server monitoring system, and analyzes the location of the root cause of the failure that occurred. System. The parts that can be identified by the root cause analysis system 111 are physical servers 130 and 135, hypervisors 131 and 136, virtual machines 132, 133, 134, 137, 138, and 139, router 104, management router 103, storage 105, and the like. A device and its CPU, memory, storage, IO module, etc.

  The root cause analysis system 111 is installed in the data center 100. However, the root cause analysis system 111 may be remotely connected to an apparatus in the data center 100 from outside the data center. The root cause analysis system 111 uses a network and the physical servers 130 and 135, the hypervisors 131 and 136, and the virtual machines 132, 133, 134, 137, 138, and the like via the management router 103 in the data center 100. 139. Further, the root cause analysis system 111 is connected to server operation information 120 and configuration management information 121 using a network, and also includes a server monitoring system 110, a spread destination prediction system 112, and a customer contact / priority determination system 113. , Connected to the Web portal 114.

  As a means for the root cause analysis system 111 to receive an alert from the server monitoring system 110, a mechanism for transmitting data from the server monitoring system 110 using the API of the root cause analysis system 111 may be used. The root cause analysis system 111 may periodically check whether there is an alert using the API of the server monitoring system 110 and read it. When the server monitoring system 110 detects a failure, the failure root cause analysis system 111 performs a process of determining what the root cause of the detected failure is. Implementation of the root cause analysis system 111 may include hardware or hardware and software in addition to software.

  The spread destination prediction system 112 is a system that refers to the configuration management information 121 and the contract information 122 based on the failure occurrence part specified by the failure root cause analysis system 111 and identifies a customer system that is considered to spread. .

  The spread destination prediction system 112 is installed in the data center 100, but may be connected to a device in the data center 100 remotely from outside the data center.

  The propagation prediction system 112 uses a network and the physical servers 130 and 135, the hypervisors 131 and 136, and the virtual machines 132, 133, 134, 137, 138, and 139 via the management router 103 in the data center 100. It is connected to the. The propagation prediction system 112 uses a network and is connected to the configuration management information 121 and the contract information 122. The server monitoring system 110, the root cause analysis system 111, the customer contact / priority determination system 113, the Web It is connected to the portal 114.

  As a means for the propagation destination prediction system 112 to receive the result from the failure root cause analysis system 111, the failure root cause analysis system 111 transmits data using the API of the propagation destination prediction system 112. Alternatively, the spread destination prediction system 112 may periodically check the presence / absence of an analysis result using the API of the failure root cause analysis system 111 and read it out.

  The spread destination prediction system 112 uses the configuration management information 121 and the contract information 122 when the failure cause root analysis system 111 estimates the part that is the cause of the failure. Process to identify users who are likely to receive. The transmission destination prediction system 112 receives the failure ID and the failure part information 126 (431), and checks whether an unprocessed failure ID exists (432).

  If there is an unprocessed failure ID, it is checked whether the received failure part is a physical server (435). If the physical server is faulty, the customer ID and system name sharing the physical server at the faulty part are searched from the configuration management information 121 (440). If it is not a physical server failure, it is checked whether the received failure part is software (436).

  If there is a software failure, the customer ID and system name using the same software are searched from the configuration management information 121 (441). If it is not a software failure, it is checked whether the received failure part is a network (437). If there is a network failure, the customer ID and system name using the same network device and network line are searched from the configuration management information 121 (442). If it is not a network failure, it is confirmed whether the received failure part is a storage (438).

  If it is a storage failure, the configuration management information 121 is searched for the same storage device, customer ID and system name sharing the disk (443). If it is not a storage failure, it is checked whether the received failure part is a virtual server (439). If a virtual server failure has occurred, the configuration management information 121 is searched for the customer ID and system name using the virtual server.

  If the failure is not a virtual server failure, a message is written in the other chapters as a failure that is difficult to estimate (447). The implementation of the spread destination prediction system 112 may include hardware or hardware and software in addition to software.

  The customer contact / priority determination system 113 refers to the configuration management information 121, the contract information 122, and the contact text template 123 for the customer specified by the spread destination prediction system, and determines the priority of the customer contact and the customer contact. It is a system that creates a contact text.

  The customer contact / priority determination system 113 is installed in the data center 100. However, the customer contact / priority determination system 113 may be remotely connected to devices in the data center 100 from outside the data center.

  The customer contact / priority determination system 113 uses a network and via the management router 103 in the data center 100, physical servers 130, 135, hypervisors 131, 136, virtual machines 132, 133, 134, 137. -It is connected to 138-139. The customer contact / priority determination system 113 is connected to the configuration management information 121, the contract information 122, and the contact message template 123 using a network, and also includes a server monitoring system 110, a root cause analysis system 111, The transmission destination prediction system 112 and the Web portal 114 are connected.

  As a means for the customer contact / priority determination system 113 to receive the result from the spread destination prediction system 112, data is transmitted from the spread destination prediction system 112 using the API of the customer contact / priority determination system 113. It may be a mechanism, or the customer contact / priority determination system 113 may periodically check the presence / absence of a prediction result using the API of the spread destination prediction system 112 and read it out.

The customer contact / priority determination system 113 uses the users predicted to be affected by the transmission destination prediction system 112, the information of the configuration management information 121 and the contract information 122, and determines the order of contact. Do.
The customer contact text creation unit of the customer contact / priority determination system 113 receives the failure ID, the cause ID, the customer ID, and the system name information 127, which is the output of the spread prediction destination system, and receives the received customer ID and contract information. A search is made for a match with the customer ID 280 of the customer management information 136 of 122, the report pattern 295 of the matching record is confirmed, and it is confirmed whether it is a telephone contact or a mail contact (452). In the case of e-mail communication, it is confirmed whether or not an e-mail template corresponding to the cause ID exists from the communication text template 123 (453). If an email template exists, the template is used to prepare an email message (455).

If the mail template does not exist, the operator creates a new mail template (456). The mail address 294 of the customer information management 136 of the contract information 122 is inserted into the mail template, and the log file 124 and the like are attached (459).
In the case of telephone contact, it is confirmed from the contact text template 123 whether a telephone template corresponding to the cause ID exists (454). If a telephone template exists, the template is used to prepare a telephone contact text (457). If the telephone template does not exist, the operator creates a new telephone template (458). The customer information management telephone number 293 of the contract information 122 is inserted into the telephone template (460).

  For 460 and 461, the failure occurrence time 332 and the recovery time of the log 124 matching the failure ID are inserted (461 and 462). The calculation of the recovery time may be calculated by the operator according to the failure content, or the recovery time may be registered in advance according to the failure content.

  The customer contact priority determination unit of the customer contact / priority determination system 113 receives 128 and 129 output from the customer contact text creation unit (471), and stores the received data in the queue (472). It is checked whether priority evaluation is completed for all customers in the queue (473). When the priority evaluation is not completed, the CPU usage rate of the virtual server operation information, the priority based on the number of accesses, the system type 286 of the contract information 122, the SLA 284, the number of claims 285, the operation start date 282, and the operation end date 283 A priority is added based on the usage period calculated from (474, 475, 476).

  When the priority evaluation is completed, the queues are rearranged in order of priority based on the added points (477), and it is confirmed whether there is data to be contacted immediately (478). If there is data to be contacted immediately, the information to be contacted immediately is displayed on the operator's screen from the contact text to the customer with the highest priority (481). When all the data has been displayed, the system waits for a new failure (482) and checks whether a new failure has occurred (479). When a new failure occurs, after a predetermined time has elapsed, the message is displayed on the screen from the customer contact text in descending order of priority (480).

  The Web portal 114 displays the priority of the customer to be contacted and the contact text, which is the output of the customer contact / priority determination system, for the operator 140 to confirm and contact the IT manager 141 or 142 System. However, if necessary, the failure content detected by the server monitoring system 110, the failure occurrence location identified by the failure root cause analysis system 111, the customer system identified by the propagation destination prediction system 112 and the failure expected to spread, the server operation Information such as information 120, configuration management information 121, contract information 122, and a contact text template may be referred to.

Although the Web portal 114 is installed in the data center 100, it may be connected to devices in the data center 100 remotely from outside the data center.
The Web portal 114 is connected to the physical servers 130 and 135, the hypervisors 131 and 136, and the virtual machines 132, 133, 134, 137, 138, and 139 via the management router 103 in the data center 100 using a network. It is connected. The customer contact / priority determination system 113 is connected to the configuration management information 121, the contract information 122, and the contact message template 123 using a network, and also includes a server monitoring system 110, a root cause analysis system 111, The transmission destination prediction system 112 and the customer contact / priority determination system 113 are connected.
The web portal 114 is used by the operator 140 to confirm the output from the server monitoring system 110, the root cause analysis system 111, the spread prediction system 112, and the customer contact / priority determination system 113. Web portal implementations may include hardware or hardware and software in addition to software.

  The data center 100 includes a router 104 that passes communication from the network 102 connecting the virtual server to the customer system (company A) 115 and the customer system (company B) 116 to an appropriate virtual server. Here, as a function of the router 104, a firewall that blocks unauthorized connections to the outside, an Intrusion Detection System or Intrusion Prevention System that monitors and filters unauthorized communications from the outside, and an IP used for communication with the outside of the data center 100 Network Address Translation that converts addresses and IP addresses used for communication within the data center 100, Virtual Routing Forwarding that logically operates the router 104 as a plurality of routers, and VLAN functions that logically divide the network It may be connected to a device having such a function.

  The router 104 selects a connection destination network according to the IP address of the access source, and selects a device that transmits data based on the IP address of the access destination. The router 104 is connected to the virtual servers 132, 133, 134, 137, 138, 139, the physical servers 130, 135, and the hypervisors 131, 136. The customer system (Company A) 115 and the customer system ( B company) has a role of transferring requests from 116 to these virtual servers.

  The virtual servers 132, 133, and 134 are logically divided servers on the physical server 130 and operate on the hypervisor 131. The virtual servers 137, 138, and 139 are logically divided servers on the physical server 135, and operate on the hypervisor 136.

  The hypervisors 131 and 136 logically divide the CPU and memory of the physical server, the network capacity of the network connected to the physical servers 131 and 136, the data area of the storage 105 connected to the physical servers 131 and 135, etc. It has a function to install and use multiple OS on a physical server.

  The management router 103 is connected to a server monitoring system 110, a failure root cause analysis system 111, a spread destination prediction system 112, and a customer contact / priority determination system.

  FIG. 2 shows the flow of processing when a failure occurs, which is a feature of this embodiment. First, the flow from the normal failure occurrence to contact with the customer in the cloud computing system is explained. In the cloud computing system, when a failure occurs (910), the operator 140 performs a failure cause analysis (911) by the root cause analysis system 111 in order to identify the cause of the failure based on the information raised from the server monitoring system 110. . Thereafter, the operator 140 specifies a customer using the propagation destination prediction system 112 in order to specify the target customer based on the specified cause of failure (912).

  Thereafter, the operator 140 creates a customer contact text (913) using the customer contact / priority determination system 113 in order to create a contact content text for each customer. After creating the text, the operator 140 contacts the IT manager (Company A) 141 by telephone or e-mail (914). Usually, the procedure from (910) to (914) is performed every time a failure occurs, and occurs during that time. The failure is processed in the same manner after waiting until the procedure is completed (921, 922, 923, 924). For this reason, the customer waits for the notification of the failure for the time required for this processing until the completion of the unprocessed failure notification.

  The feature of the present embodiment is to use the features of the cloud computing system to improve the efficiency of processing up to the second and subsequent failure notifications. After the failure occurrence (940), the operator 140 performs failure cause analysis using the failure root cause analysis system 111 in order to identify the cause of the failure based on the information raised from the server monitoring system 110 (941). Thereafter, the operator 140 specifies a customer using the propagation destination prediction system 112 in order to specify the target customer based on the specified cause of failure (942). After that, the transmission destination prediction system 112 creates a failure communication message in parallel with a customer who may cause a failure in the future in addition to the customer who is directly subject to the failure (951).

  This is based on the feature of sharing hardware among customers of cloud computing systems. For example, when a failure occurs in a virtual server, the root cause is often due to hardware or hardware settings. In this case, a similar failure is likely to occur in a system used by another customer who shares the same hardware.

  For this reason, not only customers who use a certain virtual server directly but also customers who share the hardware with the virtual server are required to create a trouble report at the same time. It is effective in reducing. When this prediction is true and a failure occurs (950), the operator 140 analyzes the root cause analysis in order to identify the cause of the failure based on the information raised from the server monitoring system 110 after the completion of the previous failure notification. Analysis by the system 111 is performed (952).

  At this time, if the cause of the failure is the same as the previous failure, the content of contacting the customer will be the same for the cause, so the operator 140 uses the contact text created in advance, and the IT administrator (B Company) 142 is contacted by telephone or e-mail (953). As a result, the time until the completion of the communication after the second company can be shortened.

  FIG. 3 shows details of the server operation information 120 and the configuration management information 121.

  The server operation information 120 is data defined for each virtual server, and includes a virtual server name 210, a CPU usage rate log 211, a disk I / O log 212, a network log 213, an access log 214, and a memory usage amount. Data including information such as a log 215 and a disk usage log 216.

  The configuration management information 121 is data including virtual server management information 130, physical server management information 131, network management information 132, storage network management information 133, storage management information 134, software management information 135, and the like.

  The virtual server management information 130 is information unique to each virtual server and used to manage the virtual server. The virtual server name 220, the IP address 221, the connection information 222, the number of cores 223, the frequency 224, the memory capacity 225, and the disk It includes information such as capacity 226, physical server name 227, storage connection information 228, software information 229, and the like.

  The physical server configuration information 131 is information for managing a physical server that is unique for each physical server. The physical server configuration information 131 is a physical server name 230, an IP address 231, a core number 232, a frequency 233, a memory capacity 234, a storage name 235, firmware. Information such as version 236, hypervisor type 237, connection information 238, and setting information 239 is included.

  The network management information 132 is information unique to each network device for managing the network device. Information such as the device name 240, IP address 251, device type 242, firmware version 252, setting information 244, connection information 245, and the like. including.

  The fiber channel switch management information 133 is information for managing the fiber channel switch unique to each fiber channel switch device. The device name 260, the IP address 261, the setting information 262, the firmware version 263, the connection information 264, etc. Contains information.

  The storage management information 134 is information for managing a storage unique to each storage device, and includes information such as a device name 260, an IP address 261, setting information 262, firmware version 263, and connection information 264.

  The software management information 135 is information installed in a virtual server, a physical server, etc., for managing software, and includes information such as a software name 270, setting information 271, version information 272, and license information 273.

  FIG. 4 shows details of the contract information 122, the contact text template 123, and the log 124.

  The contract information 122 is data including customer management information 136, customer system management information 137, and the like.

  Customer management information 136 includes customer ID 280, contract service 281, operation start date 282, operation end date 283, SLA 284, number of complaints 285, system type 286, number of end users 287, system name 288, company name 289, address 290, responsible It includes information such as name 291, title 292, telephone number 293, mail address 294, report pattern 295 and the like.

  The customer system management information 137 includes information such as a system name 310, routing information 311, used virtual server information 312, and used software information 313.

  The contact text template 123 is information that serves as a template when creating a contact text for a customer, and is information that is prepared according to a failure or cause that occurred in the past, the type of countermeasure, and the like. It includes information such as cause ID 320, countermeasure method 321, past text 322, and request 323.

  The log 123 represents data necessary for processing in the server monitoring system 110, the failure root cause analysis system 111, the spread destination prediction system 112, and the customer contact / priority determination system 113, processing results, and the like. Yes. It includes information such as failure ID 330, failure site 331, failure occurrence time 332, related ID 333, cause ID 334, log ID 335, cause correct rate 336, and the like.

  FIG. 5 shows details of the alert 125, the failure root cause 126, the failure spreading customer information 127, the contact message (mail) 128, and the contact message (phone) 129.

  The alert 125 includes information such as a failure ID 340 and a failure occurrence time 341.

  The root cause 126 includes information such as a failure ID 350, a system name 351, a failure part name 352, a failure cause 353, and a message 354.

  The trouble spreading customer information 127 includes information such as a trouble ID 360, a customer ID 361, and a system name 362.

  The contact text (email) 128 includes information such as an email address 370, a subject 371, a failure occurrence time 372, a failure type 373, an estimated failure cause 374, a scheduled recovery time 375, a request 376, a contact address 377, and attached information 378. .

  The contact text (telephone) 129 includes information such as a telephone number 380, a failure occurrence time 381, a failure type 382, an estimated failure cause 383, a scheduled recovery time 384, and a request 385.

  FIG. 6 illustrates a failure root cause analysis system 111 and a transmission destination when a failure occurring in the virtual machine 132, 133, 134, 137, 138, 139, etc. in the data center 100 is detected (400) by the server monitoring system 110. It is the detail of the process of the prediction system 112 and the customer priority / priority determination system 113.

  When a failure occurs (400), the server monitoring system 110 detects the failure, generates an alert (401), and passes the failure ID 340 of the alert 125 to the failure root cause analysis system 111. When the root cause analysis system 111 receives the fault ID 340 of the alert 125 (403), the root cause analysis (404) is performed, the root cause of the fault is identified, the cause ID is identified, and this cause ID is stored in the log 124 for a certain period of time. Hold. It is checked whether the cause ID held in the log 124 for a certain period and the cause ID specified by the failure root cause analysis system 111 match the newly generated failure (405).

  If there is no match, the data group of the failure root cause 126 is notified to the transmission destination prediction system 112 (406).

  Here, failure root cause analysis is a process of estimating an event that may cause a failure from a monitoring result using a computer. As this processing method, FTA (Fault Tree Analysis) may be used. The administrator of the cloud computing system determines the event that causes the accident and the probability that the event is caused based on the result of the monitoring data for a known failure that occurs in the cloud computing system in advance. Create a Fault Tree described by a logical expression.

  When a failure occurs, based on the monitoring result from the server monitoring system indicated by the failure ID, the size of the accident that can be derived from the monitoring result and the failure site that can be the root cause of the monitoring result are derived. There may be a plurality of fault sites, and it may be considered that the fault site has occurred due to multiple faults of a plurality of fault sites. Here, the magnitude of the accident based on the monitoring result is derived by comparing the monitoring result with a predetermined threshold value. For example, in network monitoring, if a Ping response is not returned for 1 minute and it is considered a failure, if the Ping response does not return for more than 1 minute, 1 is set as the value, and the response is returned in less than 1 minute In this case, binarization processing in a state where 0 is set as a value is applicable. Since the Fault Tree is described by a logical expression, as a result, the types of accidents that can occur for certain monitoring results and their probabilities are derived. The root cause derived from the monitoring results is derived from the fault tree structure.

  For example, when a virtual server network IO error is found, this may be caused by a hypervisor failure, a physical server NIC (Network Interface Card) failure, or a cable disconnection from the switch. Define with probability and detect whether there is any abnormality from log data in descending order of occurrence probability. By judging the abnormality in this way, it is possible to identify a faulty part having a high probability of leading to the first network IO error. The root cause analysis system 111 assigns a cause ID defined in advance for each derived root cause and the identified root cause, and stores it in the log data indicated by the fault ID.

  On the other hand, if the cause ID 334 held in the log 124 and the cause ID identified by the failure root cause analysis system 111 for a newly occurring failure coincide (405), the failure root cause 126 Is transmitted to the customer contact / priority determination system 113 (407). When the transmission destination prediction system 112 receives the failure root cause 126 (410), the failure transmission destination customer is specified (411), and the failure transmission destination customer information 127 is output. Details of the spread destination prediction system will be described in detail with reference to FIG. After the customer is specified, a data group including the failure ID 360, the customer ID 361, and the system name 362 of the failure spreading destination customer name 127 is transmitted to the customer contact / priority determination system 113 (412). Details of the spread destination prediction system 111 will be described later. When the customer contact / priority determination system 113 receives the data group including the failure ID 360, the customer ID 361, and the system name 362 of the customer name 127 to which the failure has spread (414), the customer contact priority determination processing, A process for creating a message is performed (415). Details of the customer contact / priority determination system 113 will be described in detail with reference to FIGS. The determined contact priority and contact text are stored in the DB, and the corresponding failure ID is stored in the queue (416). It is confirmed that the failure ID of the newly generated failure has not already been stored in the queue (417). If the failure ID has already been stored, the operator 140 displays a message to the customer with the higher priority in which the failure has occurred. Is displayed on the screen for browsing (418). The failure ID stored in the queue is compared with the failure ID. If they do not match, the processing returns to 414. The operator 140 confirms the display screen (419), confirms the contact text (420), and then contacts the corresponding customer (421). Details of the customer contact procedure will be described later.

  FIG. 7 shows the details of the processing of the spread destination prediction system 112.

  The spread destination prediction system 112 has a buffer that holds the reception data from the failure root cause analysis system 111 and can hold a plurality of reception data. It is assumed that the data is arranged in the order of reception like a queue so that it can be known which data is the oldest data. In addition, a flag indicating whether or not the data has been processed is stored in the memory in correspondence with each received data. The spread destination prediction system 112 receives the failure root cause 126 from the failure root cause analysis system 111 (431).

  The transmission destination prediction system 112 checks whether or not there is an unprocessed fault ID in which the reception in the reception buffer is old and the processed flag is not set (432). If so, it is checked whether there is data in the transmission queue (434). If there is no data in the transmission queue, the processing is terminated (434). If there is data in the transmission queue, the failure transmission destination customer information 127 created by the transmission destination prediction system 112 is sent to the customer contact / priority determination system 113. At the same time, the failure ID and cause ID of the failure spreading customer information 127 are stored in the log 124, and the processing is terminated (438). If there is an unprocessed failure ID, it is checked whether there is a physical server failure (435).

  If there is a physical server failure, the physical server name 230 of the physical server management information 131 of the configuration management information 121 is used as a search key, and the physical server name 227 of the virtual server management information 130 of the configuration management information 121 is matched. Search and first identify the virtual server. Further, using virtual server name 220 as a search key, virtual server information of customer system management information 137 of contract information 137 is acquired, system name 288 of customer management information 136 is searched using system name 310 as a search key, Locate the ID (440).

  If it is not a physical server failure, it is confirmed whether it is a software failure (436). If there is a software failure, the software name in the software management information 135 in the configuration management information 121 is used as a search key, and a search is made for a match with the software information 229 in the virtual server management information 130 in the configuration management information 121. Specify the server. Further, using virtual server name 220 as a search key, virtual server information of customer system management information 137 of contract information 137 is acquired, system name 288 of customer management information 136 is searched using system name 310 as a search key, Find the ID (441).

  If it is not a software failure, it is confirmed whether it is a network failure (437). If there is a network failure, the connection information 245 of the network management information 132 of the configuration management information 121 is used as a search key to search for a match with the connection information 222 of the virtual server management information 130 of the configuration management information 121. Specify the virtual server. Further, using virtual server name 220 as a search key, virtual server information of customer system management information 137 of contract information 137 is acquired, system name 288 of customer management information 136 is searched using system name 310 as a search key, Locate the ID (442).

  If it is not a network failure, it is confirmed whether it is a storage failure (438). In the case of a storage failure, the connection information 253 of the storage management information 134 of the configuration management information 121 is used as a search key to search for a match with the storage connection information 228 of the virtual server management information 130 of the configuration management information 121. Specify the virtual server. Further, using virtual server name 220 as a search key, virtual server information of customer system management information 137 of contract information 137 is acquired, system name 288 of customer management information 136 is searched using system name 310 as a search key, Locate the ID (443).

  If it is not a storage failure, it is confirmed whether it is a virtual server failure (439).

  If there is a virtual server failure, the virtual server name 220 is used as a search key, the virtual server information of the customer system management information 137 of the contract information 137 is acquired, and the system of the customer management information 136 is acquired using the system name 310 as the search key. The name 288 is searched to find the customer ID (444).

  If the failure is not a virtual server failure, a message is written as another failure that is difficult to estimate the propagation destination of the failure, and the processing returns to processing 432 (447).

Next, the spread destination prediction system 112 determines the customer ID and system name found as a result of the processing 440 and 441 and 442 and 443 and 444, the failure ID received from the failure root cause analysis system 111, The cause ID and failure transmission destination customer information 127 assigned as a result of the processing of the transmission destination prediction system 112 are stored in the transmission queue.
FIG. 8 shows the details of the contact text creation process of the customer contact / priority determination system 113. When the customer contact / priority determination system 113 receives the failure transmission destination customer information 127 transmitted from the transmission destination prediction system (451), the customer ID 361 and the system name 362 information of the failure transmission destination customer information 127 are used as search keys. , The report pattern 295 of the matching record is confirmed from the customer management information 136 of the contract information 122. Based on the report pattern 295, it is determined whether it is telephone contact, mail contact, or both (452). The method of judging the contact method will be described in detail later. If it is determined that the contact is an email, the presence or absence of a contact text (email) 128 corresponding to the cause ID is checked (453).

  If the contact text (mail) 128 corresponding to the cause ID exists, the contact text (mail) 128 corresponding to the cause ID is copied, the cause ID is inserted into the subject 371, and the contact text (mail) 128 is prepared. (455). The subject is not a cause ID but may be other information. If the contact text (mail) 128 corresponding to the cause ID does not exist, the operator 140 creates the contact text (mail) 128 (456). In the contact text (mail) 128, the mail address 370 of the customer management information 136 of the contract information 122 is searched based on the customer ID, inserted into the mail, and if there is a file to be attached such as a log, it is attached (459). Thereafter, the failure occurrence time 372 is searched from the failure ID of the alert 125 and inserted into the mail (461).

  If the customer contact / priority determination system 113 determines in step 452 that the contact is a telephone contact, the customer contact / priority determination system 113 checks whether there is a contact text (telephone) 129 corresponding to the cause ID (454). If there is a contact text (telephone) 129 corresponding to the cause ID, the contact text (telephone) 129 corresponding to the cause ID is copied to prepare for contact (457). If the contact text (telephone) 129 corresponding to the cause ID does not exist, the operator 140 creates a contact text (telephone) 129 (458). Based on the customer ID, the telephone number 293 of the customer management information 136 in the contract information 122 is retrieved from the contact text (telephone) 129 and inserted into the contact text (telephone).

  Preparation of a contact text (telephone) 129 is made (457). If the contact text (telephone) 129 corresponding to the cause ID does not exist, the operator 140 creates a contact text (telephone) 129 (458). The customer telephone number is inserted into the contact text (telephone) 129 (460). As with telephone contact and mail contact, the recovery time is calculated and inserted into the text (462), and the failure ID, customer ID, cause ID, system name, and contact text are passed to the priority determination unit (463). Similar to telephone contact and email contact, the recovery time is calculated from the failure occurrence time and the content of the failure, and inserted into the contact text (mail) 128 or contact text (phone) 129 (462), and the failure ID is input to the priority determination unit. , Customer ID, cause ID, system name, contact text (mail) 128, or contact text (phone) 129 is passed (463).

  FIG. 9 shows the details of the customer contact priority determination process of the customer contact / priority determination system 113. When the customer contact / priority determination system 113 receives (471) the contact message (mail) 128 or contact message (phone) 129 transmitted from the contact message creation processing unit, it stores it in the queue (472). As a structure of this queue, in addition to the received contact text (mail) 128 or the contact text (telephone) 129, priority can be stored. It is assumed that the customer contact / priority determination system 113 processes information stored in the queue at regular intervals. First, the customer contact / priority determination system 113 confirms whether contact priority is assigned to all data stored in the queue (473). If there is data for which the contact priority has not yet been assigned, the customer contact / priority determination system 113 uses the virtual server operation information to determine the priority based on a predetermined threshold for each CPU usage rate and number of accesses. Is calculated (474). As an example of this process, the CPU usage rate is determined to be 1 point when the CPU usage rate is 50% or more, and 0 point when the CPU usage rate is less than 50%, and the resulting score is added for each item. Next, the priority based on the system type is calculated based on the configuration management information, and added to the calculation result of the process 474 (475).

  Finally, based on the contract information, the priority based on the SLA (Service Level Agreement), the number of claims, and the usage record is calculated, and in addition to the calculation result of the process 475, the contact text (email) 128 that is the target. Or, it is linked to the contact text (phone) 129 and stored in the queue as the priority (476). However, this parameter calculation method is merely an example, and other parameters may be added. After the process 476, the customer contact / priority determination system 113 executes the process 473 again. When giving priority to all data in the queue is completed in this way, the customer contact / priority determination system 113 rearranges the queue in order of contact priority (477).

  Here, the customer contact / priority determination system 113 checks whether or not a fault directly related to a fault or a fault that is required to be reported by the SLA is included even if the fault is related (478). If there is something to be contacted, it is displayed on the screen from the contact text of the customer with high priority, and the state shifts to a waiting state for a new alert (481). If there is no data to be immediately contacted in the process 478, the state shifts to a new failure waiting state (482). When a new failure occurs in this state, that is, when the root cause 126 of failure is transmitted to the customer contact / priority determination system 113 as a result of the process 407 in FIG. 6, the customer contact / priority determination system 113 , Wait for a certain number of simultaneous failures, check whether there are any matches received in the queue in the order of contact priority, using the cause ID as a key for the failures received during that time. Display on the screen in descending order.

  FIG. 10 shows details of the contact method determination processing of the customer contact / priority determination system 113. The customer contact method needs to respond to exceptional cases such as telephone contact or e-mail contact, depending on the time of failure, details of the failure, and contract with the customer. . After the failure contact customer name 127 transmitted from the transmission destination prediction system 112 is received by the customer contact / priority determination system 113 (491), the customer contact / priority determination system 113 uses the customer ID as a search key to obtain contract information. The report pattern 295 that matches the customer ID 280 of the customer management information 136 is searched, the standard contact method is referred to, and the contact method is confirmed (492). If there is no exception in the report pattern 295, the customer is contacted according to the standard contact method (497, 498, 499).

  If there is an exception, check whether there is a communication method according to the failure (495), and if there is a communication method according to the failure, make a communication according to the communication method according to the failure communication (497, 498, 498). 499). If there is no contact method according to the failure, it is checked whether the contact method differs depending on the time (496), and the contact is made according to the contact method based on the time (497, 498, 499).

  FIG. 11 shows an embodiment of a priority order determination unit for contacting a customer. Reference numeral 508 represents an example of parameters for determining the contact priority. For example, a customer company name 500, a CPU usage rate 501, an access count 502, a system type 503, an SLA (support time) 504, an SLA (contract operation rate remaining time) 505, a complaint count 506, and a usage record 507 are used. Parameters other than 508 may be used. Reference numeral 519 represents a method for determining the communication priority. As in the threshold example 513, the threshold and the score example 513 (weighting) are determined for the parameters 519, and the parameters 508 are assigned to obtain the total score. The one with this high score is the highest priority for communication. The parameter threshold 513 and the score example 514 are merely examples, and may be changed.

DESCRIPTION OF SYMBOLS 100 ... Data center 101 ... Cloud computing system 102 ... Network 103 ... Management router 104 ... Router 105 ... Storage 110 ... Server monitoring system 111 ... Failure root cause analysis system 112 ... Ripple destination prediction system 113 ... Customer contact / priority determination System 114 ... Web portal 115 ... Customer system (Company A)
116 ... Customer system (Company B)
130, 135, physical servers 131, 136, hypervisors 132, 133, 134, 137, 138, 139, virtual machines 140, operators 141, IT manager (Company A)
142 ... IT manager (Company B)

Claims (2)

A server monitoring system having a function of detecting and notifying a failure of the cloud computing system by the system;
A root cause analysis system having a function of estimating and outputting a cause of a failure by receiving a notification from the server monitoring system;
Ripple that has the function to identify the user who is directly affected by the failure and the user who can be affected by the failure due to the failure by using the cause of the failure estimated by the root cause analysis system. A forecasting system,
Using the cause of the failure estimated by the root cause analysis system and the user information derived by the propagation destination prediction system as input, the user who is directly affected by the failure and the failure caused by the failure are affected. Customer contact that generates information to be communicated to the operator by both users and has a function to determine the order of contact based on the priority of the communication set for each user and the system status A priority determination system,
The failure communication efficiency system created in advance a failure contact content for users affected by future failures estimated by the spread destination prediction system in the customer contact / priority determination system, and a failure actually occurred In this case, when the root cause analysis system determines that a fault has occurred due to the same cause, the fault communication efficiency system is characterized in that the fault communication content previously created is presented to the operator. 2. The customer contact / priority determination system according to claim 1, further comprising a data holding unit that temporarily holds a notification that has occurred within a predetermined time, and each notification stored in the data holding unit after a predetermined time has passed. Based on the priority of communication set for the user and the status of the system, the order to be contacted is judged, and the operator is notified after changing the order of notification to the operator based on the order. , Failure communication efficiency system.

Images