JP7082285B2 - Monitoring system, monitoring method and monitoring program - Google Patents

Description

The present invention relates to a monitoring system, a monitoring method and a monitoring program.

Large-scale information processing systems such as data centers contain many parts such as hard disk drives (HDDs) that are expected to be replaced because they are more likely to fail due to continuous use. If parts are replaced after an actual failure occurs, the availability of the information processing system may decrease. Therefore, the parts with a high possibility of failure are determined in advance and replaced before the failure occurs. Is being done. For example, from past failure cases, a model showing the conditions of parts that are likely to fail is learned, and the information collected from the information processing system is compared with the model to select the parts that are likely to fail. Detection is being done.

A monitoring system having a plurality of storage devices and a monitoring server has been proposed. When an abnormality occurs in any one of the storage devices, the monitoring server collects the operation information and the configuration information of the storage device in which the abnormality has occurred, and collects the operation information and the configuration information of the storage device in which the abnormality has not occurred. The monitoring server determines the cause of the abnormality by comparing the former operation information and configuration information with the latter operation information and configuration information.

In addition, a sign monitoring system that detects signs of system failure has been proposed. The proposed sign monitoring system collects event messages from the system, analyzes the relationship between the event type and the failure, and learns the failure sign determination conditions. In addition, a management device that supports the improvement of reliability of a plurality of data centers has been proposed. The proposed management device holds the system configuration information of each data center and collates the latest failure report indicating the failure occurrence condition with the system configuration information. The management device sends a failure report to the data center that meets the failure condition.

Japanese Unexamined Patent Publication No. 2012-174232 Japanese Unexamined Patent Publication No. 2016-10060 Japanese Unexamined Patent Publication No. 2017-58789

The causes of failures include those caused by the use of parts, such as natural deterioration caused by proper use and abnormal deterioration caused by improper use. Failures caused by the use of parts can be predicted by accumulating failure occurrence cases for each information processing system and learning a model.

On the other hand, some of the causes of failure are caused by the manufacture of parts, such as manufacturing defects and firmware defects. For example, the fact that a part was manufactured on a particular date of manufacture or that it has a particular version of firmware can be one of the causes of failure. However, there is a problem that the failure caused by the manufacture of parts is often the cause of the failure experienced for the first time by the method of learning the model for each information processing system, and it is difficult to predict in advance. In addition, it is difficult to directly share detailed failure information between multiple information processing systems such as multiple data centers from the viewpoint of maintenance contracts with parts vendors and confidentiality, and problems caused by parts manufacturing occur. There is a problem that information is difficult to share.

In one aspect, it is an object of the present invention to provide a monitoring system, a monitoring method and a monitoring program that improve the accuracy of failure prediction.

In one aspect, a monitoring system with a first information processing device and a second information processing device is provided. The first information processing apparatus is based on the first configuration information indicating the first component included in the first information processing system and the first failure information indicating the failure generated in the first component. The process of learning the first model showing the relationship between the condition of the component and the failure occurrence probability of the component corresponding to the condition is performed. When the first information processing apparatus receives a certain component condition, the first information processing apparatus calculates the failure occurrence probability corresponding to the received component condition using the first model, and performs a process of transmitting the calculated failure occurrence probability. The second information processing apparatus is based on the second configuration information indicating the second component included in the second information processing system and the second failure information indicating the failure generated in the second component. A process of learning a second model showing the relationship between the condition of the component and the failure occurrence probability of the component corresponding to the condition is performed. The second information processing apparatus calculates the second failure occurrence probability corresponding to the specific component condition using the second model, and transmits the specific component condition to the first information processing apparatus to be the first. The first failure occurrence probability based on the model is received from the first information processing device, and the future failure occurrence of the part corresponding to the specific part condition is based on the first failure occurrence probability and the second failure occurrence probability. Is processed.

Also, in one aspect, a monitoring method is provided. Further, in one aspect, a monitoring program to be executed by a computer is provided.

In one aspect, the accuracy of failure prediction is improved.

It is a figure explaining the example of the monitoring system of 1st Embodiment. It is a figure which shows the example of the information processing system of the 2nd Embodiment. It is a block diagram which shows the hardware example of the management server. It is a block diagram which shows the functional example of a management server. It is a figure which shows the example of a table. It is the first figure which shows the generation example of a decision tree. It is the 2nd figure which shows the generation example of a decision tree. It is a 3rd figure which shows the generation example of a decision tree. It is a flowchart which shows the procedure example of a latent failure determination. It is a flowchart (continued) which shows the procedure example of a latent failure determination. It is a sequence diagram which shows the communication example between data centers.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
The first embodiment will be described.

FIG. 1 is a diagram illustrating an example of a monitoring system according to the first embodiment.
The monitoring system of the first embodiment includes an information processing device 10 (first information processing device) and an information processing device 20 (second information processing device).

The information processing apparatus 10 monitors the information processing system 1 (first information processing system), and among the parts (first parts) included in the information processing system 1, the parts that are likely to fail in the future. Is predicted in advance before a failure occurs. The information processing device 10 monitors the information processing system 2 (second information processing system), and among the parts (second parts) included in the information processing system 2, the parts that are likely to fail in the future. Is predicted in advance before a failure occurs.

The information processing systems 1 and 2 may provide information processing services to a third party such as a data center or a cloud computing system. The information processing systems 1 and 2 may be installed at distant places and operated independently of each other. The information processing device 10 may belong to the information processing system 1, and the information processing device 20 may belong to the information processing system 2. The parts monitored by the information processing devices 10 and 20 are, for example, HDDs and other parts that are more likely to fail due to continuous use, and are electronic devices that can be replaced independently of other parts. be.

For example, when the information processing apparatus 10 detects a component having a high possibility of failure in the information processing system 1, it notifies the administrator of the information processing system 1, a predetermined management apparatus, or the like with a warning. Further, when the information processing apparatus 20 detects a component having a high possibility of failure in the information processing system 2, the information processing apparatus 20 notifies the administrator of the information processing system 2, a predetermined management apparatus, or the like with a warning. However, the information processing device 10 may notify the administrator of the information processing system 2 or a predetermined management device of a warning, and the information processing device 20 may notify the administrator of the information processing system 1 or a predetermined management device. A warning may be notified to the device or the like.

The information processing devices 10 and 20 may be client devices or server devices. The information processing devices 10 and 20 may be general-purpose computers. For example, the information processing devices 10 and 20 include processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor). The information processing devices 10 and 20 may have dedicated hardware such as FPGA (Field-Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Further, for example, the information processing devices 10 and 20 have a volatile semiconductor memory such as a RAM (Random Access Memory) and a non-volatile storage such as an HDD or a flash memory. The processor executes the program stored in the memory. A set of two or more processors may be referred to as a "multiprocessor" or simply a "processor".

Here, the information processing apparatus 10 and the information processing apparatus 20 cooperate with each other to predict a failure in advance.
The information processing apparatus 10 has configuration information 11 (first configuration information) indicating a component included in the information processing system 1 and failure information 12 (first failure) indicating a failure occurring in a component included in the information processing system 1. Information) and get. For example, the information processing apparatus 10 monitors the information processing system 1 and collects the configuration information 11 and the failure information 12 from the information processing system 1. The configuration information 11 may include the vendor that manufactured the component, the date of manufacture of the component, the firmware version of the component, and the like. The failure information 12 indicates, for example, a component in which a failure has occurred within a certain period of time. The information processing apparatus 10 may further acquire sensor information indicating the usage status of parts from the sensor device included in the information processing system 1. The sensor information may include information on the usage environment such as temperature and humidity around the component, and may include log information such as the number of input / output errors.

The information processing apparatus 10 learns a model 13 (first model) showing a relationship between a condition of a component and a failure occurrence probability of a component corresponding to the condition, based on the configuration information 11 and the failure information 12. For example, the information processing apparatus 10 discovers a condition for appropriately grouping parts so that a failed part can be distinguished from other parts, and generates a model 13. The model 13 may be a decision tree that classifies parts into a plurality of groups according to hierarchical conditions. The failure occurrence probability can be defined as, for example, the ratio of the parts that have failed within a certain period of time among the parts that satisfy a certain condition.

The conditions for specifying one group may be a combination of two or more of the conditions for the type of parts and the conditions for the usage status of parts. For example, a failure occurrence probability of 0% is calculated for a group of parts that are vendor X disk drives and whose firmware version is later than A3. Further, a failure occurrence probability of 10% is calculated for a group of parts that are vendor X disk drives and have a firmware version of A3 or earlier and are used under a temperature of 40 ° C. or lower. Further, a failure occurrence probability of 90% is calculated for a group of parts that are vendor X disk drives and have a firmware version of A3 or earlier and are used at a temperature exceeding 40 ° C.

Similarly, the information processing apparatus 20 learns a model 23 (second model) showing the relationship between the condition of the component and the failure occurrence probability of the component corresponding to the condition, based on the configuration information 21 and the failure information 22. .. The model 23 may be a decision tree that classifies parts into a plurality of groups according to hierarchical conditions. The condition of the parts shown by the model 23 may be different from that of the model 13.

For example, a failure occurrence probability of 0% is calculated for a group of parts that are vendor X disk drives and whose firmware version is later than A3. Further, a failure occurrence probability of 20% is calculated for a group of parts that are vendor X disk drives and have a firmware version of A3 or earlier and are used at a temperature of 30 ° C. or lower. Further, a failure occurrence probability of 70% is calculated for a group of parts that are vendor X disk drives and have a firmware version of A3 or earlier and are used at a temperature exceeding 30 ° C.

The information processing apparatus 20 calculates the failure occurrence probability 25 (second failure occurrence probability) corresponding to the specific component condition 24 using the model 23. The information processing apparatus 20 may select a component condition whose failure occurrence probability is equal to or higher than a threshold value among the component conditions shown by the model 23 as the component condition 24. For example, the information processing apparatus 20 selects, as the component condition 24, a condition that the disk drive of the vendor X has a firmware version of A3 or earlier and is used at a temperature exceeding 30 ° C. Further, for example, the information processing apparatus 20 calculates the failure occurrence probability 25 as 70%. The information processing device 20 transmits the component condition 24 to the information processing device 10.

When the information processing apparatus 10 receives the component condition 24, the information processing apparatus 10 calculates the failure occurrence probability 15 (first failure occurrence probability) corresponding to the component condition 24 using the model 13. For example, the information processing apparatus 10 calculates the failure occurrence probability 15 as 90%. The information processing device 10 transmits the calculated failure occurrence probability 15 to the information processing device 20.

The information processing apparatus 20 determines the future failure occurrence of the component corresponding to the component condition 24 in consideration of the failure occurrence probability 15 in addition to the failure occurrence probability 25. For example, the information processing apparatus 20 counts the number of failure occurrence probabilities that are equal to or greater than the threshold value among the collected plurality of failure occurrence probabilities, and when the counted number is equal to or greater than the threshold value, the failure of the component corresponding to the component condition 24 Judge that the possibility of occurrence is high. The information processing device 20 may notify the administrator of the information processing system 2 or a predetermined management device of a warning. Further, the information processing apparatus 20 may notify the administrator of the information processing system 1 or a predetermined management apparatus of a warning. The information processing device 10 may perform the same processing as the information processing device 20 described above.

According to the monitoring system of the first embodiment, the component configuration information and past failure occurrence cases are collected for each information processing system, the model is learned, and the conditions of the parts with a high possibility of failure in the future. Is detected. Therefore, it is possible to predict the failure in advance. In addition, by including conditions related to the usage status of parts such as temperature and the number of input / output errors in the parts conditions, failures caused by natural deterioration or abnormal deterioration due to improper use can be expressed on the model. In addition, by including the conditions related to the manufacture of parts such as the manufacturing date and the firmware version in the parts conditions, it is possible to express a failure caused by a manufacturing defect or a firmware defect on the model.

Further, by sharing the information between the information processing apparatus 10 and the information processing apparatus 20, the accuracy of the failure prediction can be improved. In particular, failures caused by the manufacture of parts such as manufacturing defects and firmware defects can often be predicted in advance as the causes of failures that the information processing system experiences for the first time simply by learning the model for each information processing system. It can be difficult. Therefore, by sharing information among a plurality of information processing systems, it is possible to consider the experience of failures caused by the manufacture of parts in other information processing systems. In addition, it is possible to carefully determine whether or not the failure is caused by the manufacture of parts.

Further, the information processing device 10 and the information processing device 20 do not share the failure information 12 and 22 themselves, but the failure occurrence corresponding to the specific component condition 24 calculated by statistically processing the failure information 12 and 22. Share probabilities 15 and 25. Therefore, even if it is difficult to directly share detailed failure information between information processing systems from the viewpoint of maintenance contracts with parts vendors and confidentiality, information that contributes to advance prediction of failures should be shared. Can be done.

[Second Embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram showing an example of an information processing system according to a second embodiment.

The information processing system of the second embodiment includes data centers 31 to 33. The data centers 31 to 33 are information processing systems installed at locations separated from each other, and provide information processing services to customers. The data centers 31 to 33 are connected to the network 30. The network 30 includes a wide area network such as the Internet.

The data center 31 has a storage server 41, a management server 51, and an administrator terminal 61. The storage server 41, the management server 51, and the administrator terminal 61 are connected to the network 30 via the local network in the data center 31. The storage server 41 is a server computer that manages data using non-volatile storage such as an HDD. The management server 51 is a server computer that monitors electronic devices in the data center 31 such as the storage server 41. The administrator terminal 61 is a client computer used by the administrator of the data center 31. Similarly, the data center 32 has a storage server 42, a management server 52, and an administrator terminal 62. The data center 33 has a storage server 43, a management server 53, and an administrator terminal 63.

The management server 51 monitors the configuration and operation of replaceable parts such as HDDs included in the storage server 41, and predicts the failure in advance before the failure actually occurs. When the management server 51 detects a component that is likely to cause a failure, the management server 51 notifies the administrator terminal 61 of the detection result. The administrator who uses the administrator terminal 61 replaces, for example, a part indicated by the detection result with a new one before the failure occurs. As will be described later, the management server 51 may notify the detection result to the administrator terminals 62 and 63 used by the administrator of another data center.

Similarly, the management server 52 monitors replaceable parts such as HDDs included in the storage server 42 to predict failures in advance. When the management server 52 detects a component that is likely to cause a failure, the management server 52 may notify the administrator terminal 62 of the detection result, and may further notify the administrator terminals 61 and 63 of the detection result. The management server 53 monitors replaceable parts such as HDDs included in the storage server 43 to predict failures in advance. When the management server 53 detects a component that is likely to cause a failure, the management server 53 may notify the administrator terminal 63 of the detection result, and may further notify the administrator terminals 61 and 62 of the detection result.

In the second embodiment, in principle, component monitoring and failure prediction are performed in data center units, and specific failure occurrence cases are not shared between different data centers. This is because communicating details of a failure in one data center to another is often undesired by the vendor that manufactures the part and is often restricted from the perspective of confidential information management. Is. However, as will be described later, the minimum information for improving the accuracy of failure prediction is shared between the data centers 31 to 33.

The management server 51 corresponds to the information processing device 20 of the first embodiment. The management server 52 corresponds to the information processing apparatus 10 of the first embodiment.
FIG. 3 is a block diagram showing a hardware example of the management server.

The management server 51 includes a CPU 101, a RAM 102, an HDD 103, an image signal processing unit 104, an input signal processing unit 105, a medium reader 106, a communication interface 107, and a sensor device 108. The storage servers 41 to 43, the management servers 52 and 53, and the administrator terminals 61 to 63 may also have the same hardware as the management server 51.

The CPU 101 is a processor that executes a program instruction. The CPU 101 loads at least a part of the programs and data stored in the HDD 103 into the RAM 102 and executes the program. The CPU 101 may include a plurality of processor cores, and the management server 51 may include a plurality of processors.

The RAM 102 is a volatile semiconductor memory that temporarily stores a program executed by the CPU 101 and data used by the CPU 101 for calculation. The management server 51 may include a type of memory other than the RAM, or may include a plurality of memories.

The HDD 103 is a non-volatile storage that stores software programs such as an OS (Operating System), middleware, and application software, and data. The management server 51 may include other types of storage such as a flash memory and an SSD (Solid State Drive), or may include a plurality of storages.

The image signal processing unit 104 outputs an image to the display 104a connected to the management server 51 in accordance with a command from the CPU 101. As the display 104a, any kind of display such as a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD: Liquid Crystal Display), and an organic EL (OEL: Organic Electro-Luminescence) display can be used.

The input signal processing unit 105 receives an input signal from the input device 105a connected to the management server 51. As the input device 105a, any kind of input device such as a mouse, a touch panel, a touch pad, and a keyboard can be used. Further, a plurality of types of input devices may be connected to the management server 51.

The medium reader 106 is a reading device that reads programs and data recorded on the recording medium 106a. Examples of the recording medium 106a include magnetic disks such as flexible disks (FDs) and HDDs, optical disks such as CDs (Compact Discs) and DVDs (Digital Versatile Discs), and optical magnetic disks (MOs: Magneto-Optical disks). A semiconductor memory or the like can be used. The medium reader 106 stores, for example, a program or data read from the recording medium 106a in the RAM 102 or the HDD 103.

The communication interface 107 is an interface that is connected to a network and communicates with other information processing devices via the network. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or a router, or may be a wireless communication interface connected to a base station or an access point.

The sensor device 108 generates sensor information indicating the usage status of the component. As the sensor device 108, a temperature sensor that measures the temperature inside or around a certain part, a humidity sensor that measures the humidity inside or around a certain part, and the like can be used. The sensor device 108 may be built into the component or may be located outside the component.

FIG. 4 is a block diagram showing a functional example of the management server.
The management server 51 includes a data storage unit 111, a model storage unit 112, a data collection unit 113, a model learning unit 114, a failure determination unit 115, and a failure information sharing unit 116. The data storage unit 111 and the model storage unit 112 are realized by using the storage area of the RAM 102 or the HDD 103. The data acquisition unit 113, the model learning unit 114, the failure determination unit 115, and the failure information sharing unit 116 are realized by using a program executed by the CPU 101.

The data storage unit 111 stores information collected from electronic devices in the data center 31 such as the storage server 41. The information to be collected includes configuration information indicating the configuration of parts used in electronic devices such as HDDs. Further, the collected information includes sensor information indicating the usage status of each component, such as information measured by the sensor device included in the storage server 41. In addition, the collected information includes failure information indicating the occurrence of a failure in a component. Examples of component failures include HDD data loss, continuous I / O errors, and significant I / O delays. HDD failures can occur, for example, due to deterioration of the board, motors, magnetic heads, damage to the data recording surface, firmware defects, and the like.

The model storage unit 112 stores the model learned based on the configuration information, the sensor information, and the failure information. The model learned by the management servers 51 to 53 is a decision tree in which component conditions for grouping components are hierarchically arranged. The component conditions of the component group with many failures are closely related to the cause of the failure, and appropriate component conditions are found by learning. At the end of the decision tree (leaf node), the number of normal parts that have not failed and the number of abnormal parts that have failed within the most recent fixed period are registered for the component group that meets certain component conditions.

The data collection unit 113 collects configuration information, sensor information, and failure information from electronic devices in the data center 31 such as the storage server 41, and stores them in the data storage unit 111. The data collection unit 113 may collect such information on a regular basis. Further, the data collection unit 113 may collect such information irregularly when updating the model.

The model learning unit 114 learns a determination tree as a model using the configuration information, sensor information, and failure information stored in the data storage unit 111, and stores the learned determination tree in the model storage unit 112. The model learning unit 114 clusters the parts so that the parts groups having similar configurations, usage conditions, and the presence or absence of failures are formed, and determines the parts conditions for forming such parts groups. The component condition to be determined may include a manufacturing date threshold, a firmware version threshold, a temperature threshold, a humidity threshold, and the like.

That is, the model learning unit 114 forms a component group in which the configuration and usage conditions are similar and the failure rate is sufficiently high or the failure rate is sufficiently low by adjusting the component conditions. The failure rate is the ratio of the number of failed parts to the total number of parts. Parts that meet the component conditions with a sufficiently high failure rate may fail in the future even if they are currently normal, and it may be preferable to replace them as soon as possible. The model learning unit 114 may update the decision tree on a regular basis. Further, the model learning unit 114 may update the decision tree when the configuration information changes. Further, the model learning unit 114 may update the decision tree when the sensor information changes significantly. Further, the model learning unit 114 may update the decision tree every time a new failure is detected or when a certain amount of failure information is accumulated.

The failure determination unit 115 detects the component condition of the component group whose failure rate is equal to or higher than the threshold value (for example, 70% or 90%) from the decision tree stored in the model storage unit 112. The failure determination unit 115 determines that a component corresponding to the detected component condition is likely to have a failure in the near future even if it is currently normal. The failure determination unit 115 notifies the administrator terminal 61 of the detected component condition. The notification from the failure determination unit 115 to the administrator terminal 61 has a meaning as a replacement request message requesting early replacement of the component corresponding to the detected component condition.

In addition, the failure determination unit 115 cooperates with other data centers to examine the possibility that the predicted failure is a failure caused by a manufacturing defect of a part or a firmware defect, and if the possibility is high, another failure may occur. Notify the data center administrator. This is because it is difficult to detect failures caused by manufacturing defects or firmware defects at an early stage only by monitoring a single data center. Failures due to manufacturing defects or firmware defects may occur only under certain usage conditions, and the timing of occurrence may vary among data centers using similar components. Therefore, by notifying the data center that has not yet experienced such a failure, it is possible to take proactive measures against the failure caused by the manufacturing defect or the firmware defect.

That is, the failure determination unit 115 notifies the management servers 52 and 53 of the component conditions detected above through the failure information sharing unit 116, and acquires the failure rate corresponding to the detected component conditions in the data centers 32 and 33. .. The failure rate is calculated based on the decision tree learned by each management server. If the failure rate of the data centers 31 to 33, which is equal to or higher than the threshold value (for example, 70% or 90%), accounts for the majority, the failure determination unit 115 may be a failure due to a manufacturing defect or a firmware defect. Judge that the sex is high. Then, the failure determination unit 115 also notifies the administrator terminals 62 and 63 of the detected component conditions. The notification from the failure determination unit 115 to the administrator terminals 62 and 63 has a meaning as a replacement request message requesting early replacement of the component corresponding to the detected component condition.

The failure information sharing unit 116 shares information on component conditions and failure rates with the management servers 52 and 53. However, the information to be shared is the minimum information for improving the accuracy of failure prediction, and specific failure cases are not shared. When the failure information sharing unit 116 receives the component condition from the failure determination unit 115, it notifies the management servers 52 and 53 of the received component condition and requests the calculation of the failure rate corresponding to the component condition. The failure determination unit 115 receives the failure rate corresponding to the notified component condition from the management servers 52 and 53, and reports the received failure rate to the failure determination unit 115. Further, when the failure determination unit 115 receives the request for calculation of the failure rate from the management server 52 or the management server 53, the failure rate corresponding to the notified component condition is used by the decision tree stored in the model storage unit 112. Is calculated and replied.

FIG. 5 is a diagram showing an example of a table.
The data storage unit 111 stores the configuration information registered in the configuration information table 121. The configuration information table 121 includes items of a component ID, a vendor, a manufacturing date, a FW (Firmware) version, and a parent component ID. The component ID is an identifier that identifies a component such as an HDD. Vendor is the name of the manufacturer that manufactured the part. The manufacturing date is the date on which the part was manufactured. The FW version is the version of the firmware in which the component is installed. The parent part ID is an identifier that identifies the other part or product when another part or product containing the part is present. Part or all of the configuration information may be entered by the administrator.

Further, the data storage unit 111 stores the sensor information registered in the sensor information table 122. The sensor information table 122 includes items of component ID, number of IO (Input Output) errors, temperature and humidity. The component ID is an identifier that identifies the component. The number of IO errors is the number of input / output errors that have occurred in the HDD. The number of IO errors can be collected from the HDD firmware, device driver, OS, and the like. Temperature is a measured value measured by a temperature sensor installed inside or outside the component. Humidity is a measured value measured by a humidity sensor installed inside or outside the component.

Further, the data storage unit 111 stores the failure information registered in the failure information table 123. The failure information table 123 includes the component ID and time items. The component ID is an identifier that identifies the component. The time is the time when the failure occurred. The occurrence of a failure can be detected, for example, based on an error log collected from an electronic device such as a storage server 41. Whether or not it is a failure may be determined by the management server 51 or by another information processing device. In addition, part or all of the failure information may be input by the administrator.

Next, a method of generating a decision tree by the management server 51 will be described.
FIG. 6 is a first diagram showing an example of generating a decision tree.
The model learning unit 114 generates the decision tree 124 of the first stage by using the configuration information registered in the configuration information table 121. The decision tree 124 includes nodes 130-134, 140-142. Nodes 130 to 134 are nodes indicating component conditions for classifying components. Nodes 140 to 142 are leaf nodes in which the number of parts satisfying the parts condition corresponding to the path from the root node is registered among the parts possessed by the data center 31.

Since the failure information is not referred to in the first stage, the nodes 140 to 142 do not divide the number of parts into the number of normal parts and the number of abnormal parts. Further, the nodes 130 to 134 use information such as a vendor that does not need to determine a threshold value as a component condition among the configuration information. For each of the nodes 130 to 134, the parts that meet the component conditions are classified into the left subtree, and the parts that do not meet the component conditions are classified into the right subtree. By following the path between the nodes from the root node to the leaf node, each part is classified into one of the leaf nodes.

Specifically, the node 130 is a root node, and it is determined whether or not the node 130 is used in the SAN (Storage Area Network) storage product of the vendor A. The node 131 is a left child node of the node 130, and determines whether or not it is a vendor X disk drive. The node 132 is a right child node of the node 130, and determines whether or not it is used in the SAN storage product of the vendor B. Node 133 is a left child node of node 132, and determines whether or not it is a vendor X disk drive. The node 134 is a right child node of the node 133, and determines whether or not it is a vendor Y disk drive.

Node 140 is a left child node of node 131. Node 140 indicates the number of vendor X disk drives used in vendor A's SAN storage product. Node 141 is a left child node of node 133. Node 141 indicates the number of vendor X disk drives used in vendor B's SAN storage product. Node 142 is a left child node of node 134. Node 142 indicates the number of vendor Y disk drives used in vendor B's SAN storage product.

FIG. 7 is a second diagram showing an example of generating a decision tree.
The model learning unit 114 further uses the failure information registered in the failure information table 123 to update the first-stage decision tree 124 to the second-stage decision tree 125. The decision tree 125 includes nodes 135 to 137 in addition to nodes 130 to 134, and includes nodes 150 to 154 in place of nodes 140 to 142. Nodes 135 to 137 are nodes indicating component conditions for classifying components. Nodes 150 to 154 are leaf nodes in which the number of parts satisfying the parts condition corresponding to the path from the root node is registered.

Since the failure information is referred to in the second stage, the nodes 150 to 154 divide the number of parts into the number of normal parts and the number of abnormal parts. Nodes 135 to 137 use information that requires an appropriate threshold such as a firmware version from the configuration information as a component condition. The model learning unit 114 adjusts the threshold value by clustering so that the failure rate of each leaf node becomes sufficiently large or sufficiently small. The failure rate is the ratio of the number of abnormal parts to the total number of normal parts and abnormal parts.

Specifically, the node 135 is a left child node of the node 131, and it is determined whether or not the firmware version (FW version number) is later than A3. Node 136 is a left child node of node 133, and determines whether or not the FW version number is later than A3. Node 137 is a left child node of node 134, and determines whether or not the FW version number is later than T3.

Node 150 is a left child node of node 135. Node 150 is a vendor X disk drive used in vendor A's SAN storage product, and indicates the number of FW versions after A3. Node 151 is a right child node of node 135. Node 151 is a vendor X disk drive used in vendor A's SAN storage product, and indicates the number of FW versions of A3 or earlier. Node 152 is a left child node of node 136. Node 152 is a vendor X disk drive used in vendor B's SAN storage product, and indicates the number of FW versions after A3.

Node 153 is a right child node of node 136. Node 153 is a vendor X disk drive used in vendor B's SAN storage product, and indicates the number of FW versions of A3 or earlier. Node 154 is a right child node of node 137. Node 154 is a vendor Y disk drive used in vendor B's SAN storage product, and indicates the number of FW versions before T3.

FIG. 8 is a third diagram showing an example of generating a decision tree.
The model learning unit 114 further uses the sensor information registered in the sensor information table 122 to update the second-stage decision tree 125 to the third-stage decision tree 126. The decision tree 126 includes nodes 138 and 139 in addition to nodes 130 to 137, and includes nodes 155 to 158 instead of nodes 151 and 153. Nodes 138 and 139 are nodes indicating component conditions for classifying components. Nodes 155 to 158 are leaf nodes in which the number of parts satisfying the parts condition corresponding to the path from the root node is registered.

The nodes 138 and 139 use information such as temperature in the sensor information as a component condition. The model learning unit 114 adjusts the threshold value by clustering so that the failure rate of each leaf node becomes sufficiently large or sufficiently small. Specifically, the node 138 is a right child node of the node 135, and determines whether or not the temperature is higher than 30 ° C. Node 139 is a right child node of node 136, and determines whether or not the temperature is higher than 30 ° C.

Node 155 is a left child node of node 138. Node 155 is a vendor X disk drive used in vendor A's SAN storage product, and indicates the number of FW plates of A3 or earlier and the temperature at the time of use being higher than 30 ° C. Node 156 is a right child node of node 138. Node 156 is a vendor X disk drive used in vendor A's SAN storage product, and indicates the number of FW plates of A3 or earlier and a temperature of 30 ° C. or lower at the time of use. Node 157 is a left child node of node 139. Node 157 is a vendor X disk drive used in vendor B's SAN storage product, and indicates the number of FW plates of A3 or earlier and the temperature at the time of use being higher than 30 ° C. Node 158 is a vendor X disk drive used in vendor B's SAN storage product, and indicates the number of FW plates of A3 or earlier and a temperature of 30 ° C. or lower at the time of use.

The decision tree 126 is stored in the model storage unit 112. For example, the failure rate of the node 150 is 0%, the failure rate of the node 152 is 0%, the failure rate of the node 154 is 0%, the failure rate of the node 155 is 100%, the failure rate of the node 156 is 0%, and the node. The failure rate of 157 is 67%, and the failure rate of node 158 is 0%. The model learning unit 114 selects the component conditions to be adopted so that the failure rate of each leaf node is sufficiently high or sufficiently small with as few component conditions as possible, that is, the components are properly grouped. do.

The failure determination unit 115 selects a leaf node having a failure rate equal to or higher than a threshold value (for example, 70% or 90%) from the decision tree 126, and determines the component condition indicated by the path from the root node to the selected leaf node. , Detect as a component condition with a high possibility of failure. For example, the failure determination unit 115 selects the node 155 having a failure rate of 100%, and "is a vendor X disk drive used in the vendor A SAN storage product, and the FW version is A3 or earlier. The component condition that "the temperature at the time of use is higher than 30 ° C." is detected.

When the management server 52 or the management server 53 notifies the component condition, the failure information sharing unit 116 traces the decision tree 126 from the root node toward the leaf node and selects the leaf node corresponding to the notified component condition. , Returns the failure rate of the selected leaf node. In addition, the threshold value in the decision tree 126 and the threshold value in the notified component condition may be different. For example, the management server 51 may set the temperature threshold to 30 ° C, while the management server 52 may set the temperature threshold to 40 ° C. In that case, the failure rate is returned as follows.

When the threshold value in the decision tree 126 is smaller than the threshold value in the notified component condition, the failure information sharing unit 116 returns that the failure rate corresponding to the notified component condition is unknown. When the failure determination unit 115 receives an answer that the failure rate is unknown from a certain management server, the failure determination unit 115 excludes the answer and determines whether the failure rate equal to or higher than the threshold value accounts for the majority of the number of other answers. On the other hand, when the threshold value in the decision tree 126 is larger than the threshold value in the notified component condition, the failure information sharing unit 116 selects the left child node of the decision tree 126 and calculates the failure rate. Reply the failure rate.

Next, the processing of the management server 51 will be described.
FIG. 9 is a flowchart showing an example of a procedure for determining a latent failure.
(S10) The model learning unit 114 acquires configuration information such as a vendor, a manufacturing date, and a firmware version for the parts used in the data center 31.

(S11) The model learning unit 114 classifies the parts indicated by the configuration information into a plurality of parts groups based on the vendor and the like, and counts the number of parts for each part group.
(S12) The model learning unit 114 generates a decision tree corresponding to the classification method of step S11, and records the number of parts of each part group in the leaf node of the decision tree.

(S13) The model learning unit 114 acquires failure information indicating the presence or absence of a failure of a component used in the data center 31 in the latest fixed period.
(S14) The model learning unit 114 determines a threshold value such as a manufacturing date and a firmware version so that the correlation with the presence or absence of a failure is as large as possible by clustering. That is, when the component group is subdivided by the threshold value, the threshold value is adjusted so that the component group is divided into a component group having a very high failure rate and a component group having a very low failure rate.

(S15) The model learning unit 114 refines the decision tree in step S12 based on the threshold value determined in step S14. Based on the failure information, the model learning unit 114 counts the number of normal parts and the number of abnormal parts for the parts corresponding to the detailed determination Konoha node, and records the counted number of normal parts and the number of abnormal parts in the leaf node.

(S16) The model learning unit 114 acquires sensor information such as temperature, humidity, and the number of IO errors for the parts used in the data center 31.
(S17) The model learning unit 114 determines threshold values such as temperature, humidity, and the number of IO errors so that the correlation with the presence or absence of a failure is as large as possible by clustering. That is, when the component group is subdivided by the threshold value, the threshold value is adjusted so that the component group is divided into a component group having a very high failure rate and a component group having a very low failure rate.

(S18) The model learning unit 114 refines the decision tree in step S15 based on the threshold value determined in step S17. The model learning unit 114 counts the number of normal parts and the number of abnormal parts for the parts corresponding to the detailed determination Konoha node based on the failure information, and records the counted number of normal parts and the number of abnormal parts in the leaf node.

FIG. 10 is a flowchart (continued) showing an example of a procedure for determining a latent fault.
(S19) The failure determination unit 115 searches the decision tree generated in the data center 31 (own DC) for the component condition whose failure rate is equal to or higher than the threshold value. The component condition for which the failure rate is equal to or higher than the threshold value is to search for a leaf node in which the ratio of the number of abnormal parts to the total number of normal parts and abnormal parts is equal to or higher than the threshold value, and extract the path from the root node to the leaf node. You can search. The threshold value of the failure rate is a predetermined sufficiently large value such as 70% or 90%. In the decision tree, the path from the root node to the leaf node whose failure rate is equal to or higher than the threshold value can be called a Failure Path.

(S20) The failure determination unit 115 determines in step S19 whether the corresponding component condition, that is, the component condition forming a component group having a failure rate equal to or higher than the threshold value is detected. If the corresponding component condition is detected, the process proceeds to step S21, and if the corresponding component condition is not detected, the current latent failure determination is terminated.

(S21) The failure determination unit 115 notifies the administrator terminal 61 of the data center 31 of the component conditions detected in step S19. This notification is a message recommending the administrator to replace the parts that meet the detected parts conditions before the failure occurs.

(S22) The failure information sharing unit 116 notifies the management servers of other data centers (other DCs) such as the management server 52 of the data center 32 and the management server 53 of the data center 33 of the component conditions detected in step S19. , Request the failure rate corresponding to the notified part condition.

(S23) The failure information sharing unit 116 receives the failure rate corresponding to the notified component condition from the management server of another data center. Decision trees are generated in other data centers as in steps S10 to S18, and the received failure rate is calculated by collating the notified component conditions with the decision trees in other data centers. be.

(S24) The failure determination unit 115 compares each of the failure rates received in step S23 with the threshold value. The threshold value of the failure rate used here may be the same as that in step S19, and may be a sufficiently large value such as 70% or 90%. The failure determination unit 115 counts the data centers 31 and other data centers such as the data centers 32 and 33 whose failure rate corresponding to the detected component condition is equal to or higher than the threshold value.

(S25) The failure determination unit 115 determines whether the data center corresponding to step S24 occupies a majority of the total number of data centers including its own DC and other DCs. However, the threshold value of the number of data centers used here may be changed. If the corresponding data center occupies the majority, the process proceeds to step S26, and if the number is less than half, the current latent failure determination is terminated.

(S26) The failure determination unit 115 notifies the administrator terminals of other data centers, such as the administrator terminal 62 of the data center 32 and the administrator terminal 63 of the data center 33, of the component conditions detected in step S19. This notification is a message recommending the administrator to replace the parts that meet the detected parts conditions before the failure occurs.

FIG. 11 is a sequence diagram showing an example of communication between data centers.
The data center 31 generates a decision tree independently of other data centers based on the configuration information, sensor information, and failure information collected in the data center 31 (S30). The data center 32 generates a decision tree independently of other data centers based on the configuration information, sensor information, and failure information collected in the data center 32 (S31). The data center 33 generates a decision tree independently of other data centers based on the configuration information, sensor information, and failure information collected in the data center 33 (S32).

The data center 31 calculates the failure rate of each component group based on the decision tree generated in the data center 31 (S33), and the component belonging to the component group having a high failure rate and currently normal has a latent failure. It is determined that there is (S34).

The data center 31 notifies the data centers 32 and 33 of the component conditions of the component having a potential failure. The data center 32 calculates the failure rate of the component corresponding to the component condition based on the decision tree generated in the data center 32, and responds to the data center 31 (S35). The data center 33 calculates the failure rate of the component corresponding to the component condition based on the decision tree generated in the data center 33, and responds to the data center 31 (S36).

The data center 31 determines that if there are many data centers with a high failure rate, a failure may occur in the future even in a data center having a low failure rate due to manufacturing defects, firmware defects, or the like. Then, the data center 31 notifies the data centers 32 and 33 of the component conditions of the component having the potential failure and requests the replacement of the corresponding component (S37).

According to the information processing system of the second embodiment, component configuration information, sensor information, and failure information are collected for each data center, and the conditions for parts in which many failures occur in the data center are discovered. The decision tree is learned. Therefore, it is possible to detect a potential failure before an actual failure occurs in each component, and it is possible to improve the availability of the data center by replacing the component. Further, in the decision tree, the component conditions related to the use of parts such as temperature and the number of IO errors are used in combination with the component conditions related to the manufacture of parts such as the manufacturing date and the firmware version. Therefore, latent failures of various causes such as natural deterioration, abnormal deterioration due to improper use, manufacturing defects, and firmware defects can be detected.

In addition, when one data center detects a latent failure, the failure rate of another data center is checked, and if it can be said that the failure rate is high in multiple data centers, the other data center is also notified of the latent failure. .. Therefore, even a data center that has little experience of a certain type of failure and has not detected the latent failure can notice the latent failure of the type. In particular, it is possible to grasp potential failures caused by manufacturing defects and firmware defects at an early stage. In addition, by comprehensively examining the failure rates of a plurality of data centers, the accuracy of determining potential failures can be improved. In addition, specific failure information is not shared among a plurality of data centers, and component conditions and failure rates related to latent failures in the learned decision tree are shared. Therefore, even when it is difficult to directly share failure information between data centers from the viewpoint of maintenance contracts with component vendors and confidentiality, it is possible to improve the accuracy of determining potential failures.

Further, the following appendices are disclosed with respect to the embodiments including the first and second embodiments.
(Appendix 1) Conditions of parts based on the first configuration information indicating the first component included in the first information processing system and the first failure information indicating the failure generated in the first component. The process of learning the first model showing the relationship between the above and the failure occurrence probability of the component corresponding to the relevant condition, and the failure occurrence probability corresponding to the received component condition when a certain component condition is received are obtained by the first model. A first information processing device that performs a process of calculating using the above and transmitting the calculated failure occurrence probability, and
Based on the second configuration information indicating the second component included in the second information processing system and the second failure information indicating the failure that occurred in the second component, the conditions of the components and the conditions are set. The process of learning the second model showing the relationship with the failure occurrence probability of the corresponding component and the second failure occurrence probability corresponding to the specific component condition are calculated using the second model, and the specific component is specified. The component condition is transmitted to the first information processing apparatus, the first failure occurrence probability based on the first model is received from the first information processing apparatus, and the first failure occurrence probability and the second failure occurrence probability are received. A second information processing apparatus that performs a process of determining a future failure occurrence of a component corresponding to the specific component condition based on the failure occurrence probability of the above.
Monitoring system with.

(Appendix 2) The first model is a first decision tree that classifies the first part into a plurality of groups according to the conditions of the hierarchical parts, and the second model is a hierarchical part. It is a second decision tree that classifies the second part into a plurality of groups according to conditions.
The monitoring system described in Appendix 1.

(Appendix 3) The first information processing apparatus acquires first sensor information indicating the usage status of the first component from the first sensor device included in the first information processing system.
The second information processing apparatus acquires the second sensor information indicating the usage status of the second component from the second sensor device included in the second information processing system.
The conditions of the parts defined by the first model and the second model include conditions for the type of parts indicated by the configuration information and conditions for the usage status of the parts indicated by the sensor information.
The monitoring system described in Appendix 1.

(Appendix 4) The condition for the type of the part includes at least one of the date of manufacture of the part and the version of the firmware of the part, and the condition for the usage condition of the part is at least one of the measured temperature and the measured humidity. including,
The monitoring system described in Appendix 3.

(Appendix 5) The second information processing apparatus selects, as the specific component condition, a condition of a component whose failure occurrence probability calculated by using the second model is equal to or greater than a threshold value.
The monitoring system described in Appendix 1.

(Appendix 6) The second information processing apparatus has a plurality of failures including the first failure occurrence probability and the second failure occurrence probability in the plurality of first information processing devices for the specific component condition. The occurrence probability is acquired, and it is determined whether or not a failure will occur in the component corresponding to the specific component condition in the future based on the number of failure occurrence probabilities above the threshold value among the plurality of failure occurrence probabilities.
The monitoring system described in Appendix 1.

(Appendix 7) The computer included in the second information processing system is
Based on the configuration information indicating the parts included in the second information processing system and the failure information indicating the failure generated in the parts of the second information processing system, the conditions of the parts and the parts corresponding to the conditions Learn a model that shows the relationship with the failure probability
Using the model, a second failure occurrence probability corresponding to a specific component condition is calculated.
The specific component condition is transmitted to another computer included in the first information processing system, and the first failure occurrence probability corresponding to the specific component condition calculated by the other computer is obtained from the other computer. Receive and
Based on the first failure occurrence probability and the second failure occurrence probability, the future failure occurrence of the component corresponding to the specific component condition is determined.
Monitoring method.

(Appendix 8) The model is a decision tree for classifying the parts of the second information processing system into a plurality of groups according to the conditions of the hierarchical parts.
The monitoring method described in Appendix 7.

(Appendix 9) In the learning of the model, sensor information indicating the usage status of the parts of the second information processing system is acquired from the sensor device included in the second information processing system.
The condition of the component specified by the model includes a condition for the type of component indicated by the configuration information and a condition for the usage status of the component indicated by the sensor information.
The monitoring method described in Appendix 7.

(Appendix 10) For the computer included in the second information processing system,
Based on the configuration information indicating the parts included in the second information processing system and the failure information indicating the failure generated in the parts of the second information processing system, the conditions of the parts and the parts corresponding to the conditions Learn a model that shows the relationship with the failure probability
Using the model, a second failure occurrence probability corresponding to a specific component condition is calculated.
The specific component condition is transmitted to another computer included in the first information processing system, and the first failure occurrence probability corresponding to the specific component condition calculated by the other computer is obtained from the other computer. Receive and
Based on the first failure occurrence probability and the second failure occurrence probability, the future failure occurrence of the component corresponding to the specific component condition is determined.
A monitoring program that executes processing.

(Appendix 11) The model is a decision tree for classifying the parts of the second information processing system into a plurality of groups according to the conditions of the hierarchical parts.
The monitoring program according to Appendix 10.

(Appendix 12) In the learning of the model, sensor information indicating the usage status of the parts of the second information processing system is acquired from the sensor device included in the second information processing system.
The condition of the component defined by the model includes a condition for the type of component indicated by the configuration information and a condition for the usage status of the component indicated by the sensor information.
The monitoring program according to Appendix 10.

1, 20 Information processing system 10, 20 Information processing device 11, 21 Configuration information 12, 22 Failure information 13, 23 Model 24 Part condition 15, 25 Failure occurrence probability

Claims (8)

Based on the first configuration information indicating the first component included in the first information processing system and the first failure information indicating the failure that occurred in the first component, the conditions of the components and the conditions are set. The process of learning the first model showing the relationship with the failure occurrence probability of the corresponding component, and when a certain component condition is received, the failure occurrence probability corresponding to the received component condition is calculated using the first model. Then, the first information processing apparatus that performs the process of transmitting the calculated failure occurrence probability, and
Based on the second configuration information indicating the second component included in the second information processing system and the second failure information indicating the failure that occurred in the second component, the conditions of the components and the conditions are set. The process of learning the second model showing the relationship with the failure occurrence probability of the corresponding component and the second failure occurrence probability corresponding to the specific component condition are calculated using the second model, and the specific component is specified. The component condition is transmitted to the first information processing apparatus, the first failure occurrence probability based on the first model is received from the first information processing apparatus, and the first failure occurrence probability and the second failure occurrence probability are received. A second information processing apparatus that performs a process of determining a future failure occurrence of a component corresponding to the specific component condition based on the failure occurrence probability of the above.
Monitoring system with. The first model is a first decision tree that classifies the first part into a plurality of groups according to the conditions of the hierarchical parts, and the second model is the first decision tree according to the conditions of the hierarchical parts. A second decision tree that classifies two parts into multiple groups,
The monitoring system according to claim 1. The first information processing apparatus acquires the first sensor information indicating the usage status of the first component from the first sensor device included in the first information processing system.
The second information processing apparatus acquires the second sensor information indicating the usage status of the second component from the second sensor device included in the second information processing system.
The conditions of the parts defined by the first model and the second model include conditions for the type of parts indicated by the configuration information and conditions for the usage status of the parts indicated by the sensor information.
The monitoring system according to claim 1. The conditions for the type of part include at least one of the date of manufacture of the part and the version of the firmware the part has, and the conditions for the usage of the part include at least one of the measured temperature and the measured humidity.
The monitoring system according to claim 3. The second information processing apparatus selects, as the specific component condition, a condition of a component whose failure occurrence probability calculated by using the second model is equal to or greater than a threshold value.
The monitoring system according to claim 1. The second information processing device acquires a plurality of failure occurrence probabilities including the first failure occurrence probability and the second failure occurrence probability in the plurality of first information processing devices for the specific component condition. Then, based on the number of failure occurrence probabilities equal to or higher than the threshold value among the plurality of failure occurrence probabilities, it is determined whether or not a failure will occur in the component corresponding to the specific component condition in the future.
The monitoring system according to claim 1. The computer included in the second information processing system
Based on the configuration information indicating the parts included in the second information processing system and the failure information indicating the failure generated in the parts of the second information processing system, the conditions of the parts and the parts corresponding to the conditions Learn a model that shows the relationship with the failure probability
Using the model, a second failure occurrence probability corresponding to a specific component condition is calculated.
The specific component condition is transmitted to another computer included in the first information processing system, and the first failure occurrence probability corresponding to the specific component condition calculated by the other computer is obtained from the other computer. Receive and
Based on the first failure occurrence probability and the second failure occurrence probability, the future failure occurrence of the component corresponding to the specific component condition is determined.
Monitoring method. For the computer included in the second information processing system,
Based on the configuration information indicating the parts included in the second information processing system and the failure information indicating the failure generated in the parts of the second information processing system, the conditions of the parts and the parts corresponding to the conditions Learn a model that shows the relationship with the failure probability
Using the model, a second failure occurrence probability corresponding to a specific component condition is calculated.
The specific component condition is transmitted to another computer included in the first information processing system, and the first failure occurrence probability corresponding to the specific component condition calculated by the other computer is obtained from the other computer. Receive and
Based on the first failure occurrence probability and the second failure occurrence probability, the future failure occurrence of the component corresponding to the specific component condition is determined.
A monitoring program that executes processing.

Images