JP7401764B2 - Control program, control method and control device - Google Patents

Description

The present invention relates to a control program, a control method, and a control device.

There is a computer virtualization technology that forms a virtual processing unit on a physical machine that looks like a physical machine to an application. A portion of the hardware resources of a physical machine, such as a processor and memory, are allocated to the virtual processing unit. A virtual processing unit may execute applications using allocated resources. By using computer virtualization technology, it is easy to add, move, and delete virtual processing units, making it possible to flexibly respond to changes in application demand and the operational status of physical machines.

The virtual processing unit includes a narrowly defined virtual machine that executes a guest operating system (OS) and a lightweight container that does not execute a guest OS. A virtual machine can be created on a physical machine by running a hypervisor or the like on the physical machine. Furthermore, by having the physical machine execute a container engine, a container can be formed on the physical machine. Additionally, a container can be created on a virtual machine by having the virtual machine run a container engine. Therefore, it is possible to stack processing units such as physical machines, virtual machines, containers, etc. in a hierarchical manner.

Note that a fault diagnosis method has been proposed for detecting a fault in a distributed application implemented using two or more cloud systems. The proposed fault diagnosis method collects and monitors information on connections between different cloud systems and detects connection-related faults.

International Publication No. 2012/162171

By the way, one of the system design concepts is to arrange a large number of virtual processing units, distribute and execute subdivided applications among these virtual processing units, and have these applications work together to realize a series of services. There are ways to do this. However, when various applications are executed on a virtual environment including a large number of virtual processing units, there is a problem in that it is not easy to efficiently analyze the cause of a failure.

When a failure occurs in an application, the root cause may not lie in the application itself, but in a failure in the virtual environment that is the infrastructure that supports the application. For example, a hardware failure in a physical machine or a failure in a virtual machine's guest OS may be the root cause of a failure in an application running on a container. However, in a system that includes many processing units and many applications, identifying other failures that are the root cause of one application failure can be difficult and require a lot of effort by the system administrator. .

Accordingly, in one aspect, an object of the present invention is to provide a control program, a control method, and a control device that make analysis of the cause of a failure in a virtualized system more efficient.

In one aspect, a control program is provided that causes a computer to perform the following processing. Each of the processing nodes is a processing node that can execute an application using the allocated resources, and includes multiple processing nodes that can be arranged hierarchically using virtualization software, and each of the multiple applications can execute multiple processing nodes. Regarding the information processing system executed on any of the nodes, first failure information indicating a failure of the first application, second failure information indicating a failure of the second processing node, and first failure information. and a teacher label indicating whether there is a relationship with the second failure information. a first evaluation value indicating a hierarchical relationship in arrangement between a first processing node that executes the first application and a second processing node based on the first failure information and the second failure information; A second evaluation value indicating a dependency relationship between a second application executed on a processing node placed above the second processing node and the first application, and a second evaluation value included in the first failure information. A third evaluation value indicating the degree of similarity between the first error message and the second error message included in the second failure information is calculated. Using training data in which feature information including the first evaluation value, second evaluation value, and third evaluation value is associated with a teacher label, 2. A model is generated to estimate whether two pieces of fault information are related.

Also, in one aspect, a computer-implemented control method is provided. Moreover, in one aspect, a control device having a storage section and a processing section is provided.

In one aspect, analysis of the cause of failure in a virtualized system can be made more efficient.

FIG. 2 is a diagram illustrating an example of a control device according to the first embodiment. FIG. 3 is a diagram illustrating an example of an information processing system according to a second embodiment. FIG. 2 is a block diagram showing an example of hardware of a management server. FIG. 2 is a diagram illustrating an example hierarchy of virtualization infrastructure. FIG. 3 is a diagram showing an example of a system configuration graph. FIG. 3 is a diagram showing an example of a service mesh graph. FIG. 3 is a diagram showing an example of generation of a cause determination model. FIG. 3 is a diagram showing an example of a system management screen. FIG. 2 is a block diagram illustrating a functional example of a management server. FIG. 3 is a diagram showing an example of a failure table. FIG. 3 is a diagram showing an example of a configuration table. It is a figure showing an example of a service distance table and a service arrangement table. FIG. 3 is a diagram showing an example of a training data table. 3 is a flowchart illustrating an example of a model generation procedure. 3 is a flowchart illustrating an example of a procedure for determining the cause of a failure. 3 is a flowchart illustrating an example of a model update procedure.

The present embodiment will be described below with reference to the drawings.
[First embodiment]
A first embodiment will be described.

FIG. 1 is a diagram illustrating an example of a control device according to a first embodiment.
The control device 10 according to the first embodiment is used to analyze a failure that occurs in the information processing system 20. The control device 10 may be a client device or a server device. The control device 10 can also be referred to as a computer, an information processing device, an analysis device, a machine learning device, or the like. The control device 10 and the information processing system 20 may be connected via a network.

The information processing system 20 is a monitored system that executes applications using computer virtualization technology. The information processing system 20 has a plurality of processing nodes including processing nodes 21, 22, and 23 (first, second, and third processing nodes).

Each processing node can execute applications using assigned resources. Resources may include hardware resources such as the computing power of a processor, a storage area of a memory, and a communication band. Each processing node may be a physical machine or a virtual processing node such as a virtual machine or a container. Two or more processing nodes can be arranged hierarchically using virtualization software. For example, by having a processing node run a hypervisor, a virtual machine can be placed on the processing node. Further, by executing a container engine on a certain processing node, a container can be placed on that processing node. One or more virtual machines may be placed on top of a physical machine, and one or more containers may be placed on top of each virtual machine.

In the information processing system 20, each of a plurality of applications including applications 24 and 25 (first and second applications) is executed on one of the processing nodes. An application can also be called application software. Each application may be implemented by a single program or a collection of two or more programs. Each application operates as a single process or a collection of two or more processes. The above program can also be called an application program or a user program. The above process can also be called a user process. Examples of applications include web servers, business logic servers, database servers, and the like.

The plurality of applications may be implemented based on a so-called microservice architecture. In microservices architecture, multiple applications with subdivided functions are implemented, and these multiple applications are distributed and placed on multiple processing nodes. By mutually communicating and coordinating these multiple applications, a series of services such as web services are provided to the user.

As an example, processing node 21 is running application 24 . Further, the processing node 23 is executing the application 25. Processing node 23 exists above processing node 22 . Here, the presence of the processing node 23 above the processing node 22 suffices if there is a hierarchical parent-child relationship between the two. There may be a direct parent-child relationship where the processing node 22 directly controls the processing node 23, or an indirect relationship where another layer of processing nodes exists between the processing node 22 and the processing node 23. It may be a parent-child relationship. The lower tier is closer to the physical machine, and the higher tier is closer to the application.

The processing node 21 may be located above the processing node 22 or may not be located above the processing node 22. Therefore, the processing node 21 and the processing node 23 may be located on the same physical machine or may be located on different physical machines. For example, the processing nodes 21 and 23 are containers located one level above a physical machine or a virtual machine. Further, for example, the processing node 22 is a physical machine or a virtual machine.

The control device 10 has a storage section 11 and a processing section 12. The storage unit 11 may be a volatile semiconductor memory such as a RAM (Random Access Memory), or may be a nonvolatile storage such as an HDD (Hard Disk Drive) or a flash memory. The processing unit 12 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the processing unit 12 may include a specific purpose electronic circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor executes a program stored in a memory such as a RAM (or the storage unit 11). A collection of multiple processors is sometimes referred to as a "multiprocessor" or simply "processor."

The storage unit 11 stores fault information 13 and 14 (first and second fault information) and a teacher label 15. The failure information 13 indicates failures that occurred in the application 24 in the past. Fault information 14 indicates a fault that occurred in the processing node 22 in the past. The teacher label 15 indicates whether there is a relationship between the failure information 13 and the failure information 14. For example, the teacher label 15 is a flag indicating whether or not the cause of the failure in the application 24 is a failure in the processing node 22.

The failure information 13 includes, for example, information that identifies the application 24 in which the failure has occurred, and an error message (first error message) indicating the details of the failure. The failure information 13 may be generated by management software (for example, a container library) of the processing node 21 that executes the application 24. Further, the failure information 13 may be generated by the control device 10 from a log output by the management software of the processing node 21. Similarly, the failure information 14 includes, for example, information that identifies the processing node 22 in which the failure has occurred, and an error message (second error message) indicating the details of the failure. The failure information 14 may be generated by management software of the processing node 22 (for example, a host OS of a physical machine or a guest OS of a virtual machine). Further, the failure information 14 may be generated by the control device 10 from a log output by the management software of the processing node 22.

Examples of failures in the application 24 include failure in data communication with other applications, timeout in data processing, and the like. Examples of failures in the processing node 22 include failure to access hardware such as memory, HDD, and communication interface, and abnormal termination of management processes such as communication processes.

The teacher label 15 reflects the result of the system administrator's analysis of the cause of the failure of the application 24. Teacher label 15 may be created by a system administrator. Further, the teacher label 15 may be generated by the control device 10 from a failure response record created by a system administrator. For example, if it is concluded during failure handling that the cause of the failure in the application 24 is a failure in the processing node 22, the teacher label 15 indicates "related." On the other hand, if it is concluded during failure handling that the cause of the failure of the application 24 is other than the failure of the processing node 22, the teacher label 15 indicates "unrelated."

The processing unit 12 generates training data using the fault information 13 and 14 and the teacher label 15, and generates a model 17 by machine learning using the generated training data. In generating the training data, the processing unit 12 generates feature information 16 including evaluation values 16a, 16b, and 16c (first, second, and third evaluation values) based on the failure information 13 and 14. Then, the processing unit 12 generates training data in which the feature information 16 and the teacher label 15 are associated with each other. The feature information 16 corresponds to an explanatory variable, and the teacher label 15 corresponds to an objective variable.

The evaluation value 16a is an index indicating the parent-child relationship between processing nodes. The evaluation value 16a indicates the hierarchical relationship between the processing node 21 that executes the application 24 in which the fault has occurred and the processing node 22 in which the fault has occurred. For example, the evaluation value 16a indicates a hierarchical relationship between a container and a physical machine or a virtual machine. The evaluation value 16a may indicate whether or not the processing node 21 exists above the processing node 22. Furthermore, the evaluation value 16a may indicate the distance (hierarchical difference) between the hierarchy of the processing node 21 and the hierarchy of the processing node 22.

The evaluation value 16b is an index indicating the dependency relationship between applications. The evaluation value 16b indicates the dependency relationship between the application 25 executed on the processing node 23 placed above the processing node 22 in which the fault has occurred and the application 24 in which the fault has occurred. For example, a communication relationship between the application 24 and the application 25 can be used as the dependency relationship. The evaluation value 16b may indicate whether the application 24 and the application 25 perform data communication directly or indirectly. Furthermore, the evaluation value 16b may indicate the distance between the application 24 and the application 25 in a service mesh graph indicating communication relationships between a plurality of applications.

The evaluation value 16c indicates the degree of similarity between the error message included in the fault information 13 and the error message included in the fault information 14. As an index of similarity, for example, cosine similarity of Bag of Words is used. However, other metrics such as edit distance (Levenshtein distance) can also be used. Additionally, in calculating the similarity, preprocessing may be performed such as filtering out predetermined types of character strings that can become noise, such as numbers, from each error message, or extracting only characteristic keywords. .

One record of training data is generated for each pair of fault information. The processing unit 12 generates a training data record from a different pair of failure information in the same manner as the pair of failure information 13 and 14 . When the training data is generated, the processing unit 12 generates the model 17 by machine learning using the training data. The model 17 estimates the presence or absence of a relationship between two pieces of fault information from input data corresponding to feature information about the two pieces of fault information. The model 17 may determine whether there is a relationship between two failures, or may determine whether the cause of one failure is the other failure. Furthermore, the model 17 may output its confidence level.

Model 17 includes parameters whose values are determined by machine learning. The model 17 is, for example, a regression model generated by logistic regression analysis. However, the model 17 may be another type of model such as a support vector machine (SVM), a random forest, or a neural network.

According to the control device 10 of the first embodiment, from the failure information 13 and 14 indicating past failures, it is possible to determine whether there is a relationship between an application failure and a failure of a processing node included in the infrastructure as a virtual environment. A model 17 is generated that determines. By using the generated model 17, it is possible to evaluate the possibility that the cause of an application failure is a failure of a specific processing node. Therefore, it is possible to support the system administrator in handling failures, and it is possible to shorten the system administrator's work time.

Furthermore, the characteristic information 16 that is input to the model 17 includes an evaluation value 16a that indicates the hierarchical relationship between the processing node that executes the application in which the failure occurred and the processing node in which the failure occurred at the same time. Therefore, it is possible to express how a failure propagates from a lower hierarchy to an upper hierarchy. Further, the characteristic information 16 includes an evaluation value 16b indicating the dependency relationship between the application in which the failure has occurred and the application being executed in the hierarchy above the processing node in which the failure has occurred. Therefore, it is possible to express a manner in which a failure propagates between applications due to a communication error or the like. Further, the feature information 16 includes an evaluation value 16c indicating the similarity of error messages. Therefore, it is possible to express a situation in which the same type of failure occurs in the application and the processing node due to the propagation of the failure. With the above, it becomes possible to accurately determine the relationship between two failures.

[Second embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram illustrating an example of an information processing system according to the second embodiment.

The information processing system of the second embodiment includes a client terminal 31, a management server 100, and a monitored system 200. The client terminal 31, the management server 100, and the monitored system 200 are connected to the network 30. The network 30 may include a LAN (Local Area Network) or a wide area network such as the Internet. Management server 100 corresponds to control device 10 of the first embodiment. The monitored system 200 corresponds to the information processing system 20 of the first embodiment.

The client terminal 31 is a client computer used by a system administrator who manages the monitored system 200. When a failure occurs in the monitored system 200, the client terminal 31 receives failure information from the management server 100 and displays it. When two or more failures occur at the same time, the failure information received from the management server 100 may include information indicating the possibility that the root cause of one failure is another failure. The system administrator refers to the fault information displayed on the client terminal 31 and performs troubleshooting work including identifying the cause of the fault. The client terminal 31 receives input of failure handling information from the system administrator, and transmits the failure handling information to the management server 100.

The management server 100 is a server computer that monitors the monitored system 200. The management server 100 collects configuration information indicating the arrangement of components included in the monitored system 200. Additionally, the management server 100 collects service information indicating logical relationships between applications executed on the monitored system 200. The management server 100 also collects failure information indicating failures that have occurred in the monitored system 200. When the management server 100 detects a failure, it transmits failure information to the client terminal 31. Additionally, the management server 100 receives failure response information from the client terminal 31 and stores it.

If two or more failures occur at the same time, the management server 100 evaluates the possibility that the root cause of one failure is another failure, includes the evaluation result in failure information, and sends it to the client terminal 31. . A cause determination model generated by machine learning is used to evaluate the relationship between two disorders. The cause determination model is a regression model that calculates a degree of certainty indicating the probability that the two failures are related from information on the two failures. The management server 100 generates training data from previously collected configuration information, service information, and failure information, and uses this training data to generate a cause determination model. Furthermore, the management server 100 updates the cause determination model based on feedback from the client terminal 31 so that the determination accuracy increases.

The monitored system 200 is an information processing system that executes applications using computer virtualization technology. The monitored system 200 may be an on-premises system (in-house system) owned by a service provider, or a cloud system owned by a cloud provider and used by the service provider for a fee.

The monitored system 200 includes a plurality of physical machines including physical machines 201 and 202, and one or more communication devices including a switch 203. Physical machines 201 and 202 are connected to a switch 203. Physical machines 201 and 202 are server computers that form a plurality of virtual processing nodes using virtualization software. The virtual processing node includes a narrowly defined virtual machine formed using a hypervisor and a container formed using a container engine. A virtual machine is a highly independent virtual processing node that runs a guest OS, whereas a container is a lightweight virtual processing node that does not run a guest OS.

The monitored system 200 executes an application implemented based on a microservice architecture. Functions for realizing services such as web services are subdivided and implemented as multiple applications, and different applications are executed on different virtual processing nodes. Applications include, for example, web servers, business logic servers, database servers, and the like. Multiple applications communicate and cooperate with each other to realize a series of services. In the second embodiment, an application is executed in a container, as will be described later. This facilitates scale-out (increase in the number of servers) and scale-in (decrease in the number of servers) in response to load fluctuations. Additionally, it becomes easy to add applications with new functions.

FIG. 3 is a block diagram showing an example of hardware of the management server.
Management server 100 includes CPU 101 , RAM 102 , HDD 103 , image interface 104 , input interface 105 , media reader 106 , and communication interface 107 . These units included in the management server 100 are connected to a bus. The CPU 101 corresponds to the processing unit 12 of the first embodiment. RAM 102 or HDD 103 corresponds to storage unit 11 in the first embodiment. The client terminal 31 and the physical machines 201 and 202 can also be realized using the same hardware as the management server 100.

The CPU 101 is a processor that executes program instructions. The CPU 101 loads at least part of the program and data stored in the HDD 103 into the RAM 102, and executes the program. The CPU 101 may include multiple processor cores, and the management server 100 may include multiple processors. A collection of multiple processors is sometimes referred to as a "multiprocessor" or simply "processor."

The RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the CPU 101 and data used by the CPU 101 for calculations. The management server 100 may include a type of memory other than RAM, or may include a plurality of memories.

The HDD 103 is a nonvolatile storage that stores software programs such as an OS, middleware, and application software, and data. The management server 100 may include other types of storage such as flash memory and SSD (Solid State Drive), or may include multiple storages.

The image interface 104 outputs an image to the display device 111 connected to the management server 100 according to instructions from the CPU 101. As the display device 111, any type of display device can be used, such as a CRT (Cathode Ray Tube) display, a Liquid Crystal Display (LCD), an Organic Electro-Luminescence (OEL) display, or a projector. . An output device other than the display device 111, such as a printer, may be connected to the management server 100.

The input interface 105 receives input signals from the input device 112 connected to the management server 100. Any type of input device can be used as the input device 112, such as a mouse, touch panel, touch pad, keyboard, etc. Multiple types of input devices may be connected to the management server 100.

The medium reader 106 is a reading device that reads programs and data recorded on the recording medium 113. As the recording medium 113, any type of recording medium can be used, such as a magnetic disk such as a flexible disk (FD) or HDD, an optical disk such as a compact disc (CD) or a digital versatile disc (DVD), or a semiconductor memory. I can do it. For example, the media reader 106 copies programs and data read from the recording medium 113 to other recording media such as the RAM 102 and the HDD 103. The read program is executed by the CPU 101, for example. Note that the recording medium 113 may be a portable recording medium, and may be used for distributing programs and data. Further, the recording medium 113 and the HDD 103 are sometimes referred to as computer-readable recording media.

The communication interface 107 is connected to the network 30 and communicates with the client terminal 31 and the physical machines 201 and 202 via the network 30. The communication interface 107 is a wired communication interface connected to a wired communication device such as a switch or a router. However, the communication interface 107 may be a wireless communication interface connected to a wireless communication device such as a base station or an access point.

Next, the virtual environment of the monitored system 200 will be explained.
FIG. 4 is a diagram illustrating an example hierarchy of virtualization infrastructure.
The monitored system 200 uses both hypervisor virtualization and container virtualization as computer virtualization technologies. In hypervisor-type virtualization, a virtual machine including a guest OS is formed using a hypervisor, which is virtualization software. In container-type virtualization, a container engine, which is virtualization software, is used to form a container that does not include a guest OS. Physical machines, virtual machines, and containers are all processing nodes that can be seen as computers by applications. However, virtual machines and containers are virtual processing nodes that are not physical machines themselves.

In the second embodiment, a container executes an application. The monitored system 200 uses both a two-tier virtual environment in which containers are placed on top of physical machines and a three-tier virtual environment in which virtual machines are placed on top of physical machines and containers are placed on top of virtual machines. . In a two-tier virtual environment, one or more containers run on a physical machine, and one or more applications run on each container. In a three-tier virtual environment, one or more virtual machines run on a physical machine, one or more containers run on each virtual machine, and one or more applications run on each container. It is also possible to have a two-layer virtual environment and a three-layer virtual environment coexist in one physical machine.

In a two-tier virtual environment, a physical machine executes a host OS 241 and a container engine 244a. The host OS 241 manages the hardware resources of the physical machine and controls the execution of processes on the physical machine. The hardware resources include the computing power of the CPU, the storage area of the RAM, the storage area of the HDD, the communication band of the communication interface, and the like. Memory space is managed separately into kernel space and user space. The host OS 241 is sometimes referred to as a base OS from the perspective of the container.

The host OS 241 generates a log indicating the operating status of the physical machine. A log generated by the host OS 241 may indicate a failure in the physical machine. Physical machine failures include abnormal access to hardware resources and abnormal termination of important management processes. A physical machine has an identifier such as a host name, an IP (Internet Protocol) address, and a MAC (Medium Access Control) address. The host OS 241 knows the identifier of the physical machine.

The container engine 244a is virtualization software that forms one or more containers directly above a physical machine. The container engine 244a allocates a part of the hardware resources of the physical machine to the container. The container engine 244a provides a part of the user space managed by the host OS 241 to the container as a virtual user space. The container engine 244a knows the containers running on the physical machine.

Further, in a two-tier virtual environment, a container executes a container library 245a and a sidecar proxy 246a. The container library 245a controls execution of processes on containers using hardware resources allocated to the containers. Container library 245a uses virtual user space allocated from container engine 244a. Although the container library 245a is not an OS itself, it provides limited OS functions to applications as an API (Application Programming Interface). The container library 245a is sometimes referred to as a container OS because it may look like an OS from an application.

The container library 245a generates a log indicating the operating status of the container. The logs generated by the container library 245a may indicate failures in applications executed by the container. Application failures include communication failures with other applications and abnormal termination of application processes. A container has an identifier such as a host name or address. The container library 245a knows the container identifier.

Sidecar proxy 246a is proxy server software. Sidecar proxy 246a relays communications at the application layer when applications running in containers communicate with other applications (particularly applications on other containers). This allows the sidecar proxy 246a to understand message communication performed between applications. The container library 245a or sidecar proxy 246a knows the applications running on the container.

Furthermore, in a two-tier virtual environment, a service node 247a is placed above the container. The service node 247a is one application that is segmented and implemented to realize a certain service. Service node 247a is executed under the control of container library 245a. The service node 247a is, for example, a server application such as a web server, business logic server, or database server. Service node 247a is implemented by a single user program or a collection of two or more user programs. Further, the service node 247a is executed as a single process or a collection of two or more processes. The service node 247a has an identifier such as a server name or a URL (Uniform Resource Locator).

In a three-tier virtual environment, a physical machine runs the hypervisor 242. Hypervisor 242 is virtualization software that forms one or more virtual machines directly above a physical machine. The hypervisor 242 allocates some of the hardware resources of the physical machine to the virtual machine. The hypervisor 242 knows the virtual machines running on the physical machines. Note that even in a three-layer virtual environment, the physical machine runs a management OS.

Further, in a three-tier virtual environment, a virtual machine executes a guest OS 243 and a container engine 244b. The guest OS 243 manages hardware resources allocated to virtual machines and controls execution of processes on the virtual machines. The guest OS 243 is sometimes called a base OS from the perspective of the container. The guest OS 243 generates a log indicating the operating status of the virtual machine. A log generated by the guest OS 243 may indicate a failure of the virtual machine. Virtual machine failures include abnormal access to allocated hardware resources and abnormal termination of important management processes. A virtual machine has an identifier such as a host name or address. The guest OS 243 knows the virtual machine identifier.

The container engine 244b is virtualization software that forms one or more containers directly above a virtual machine. The container engine 244b further allocates a portion of the hardware resources allocated to the virtual machine to the container. The container engine 244b provides a part of the user space managed by the guest OS 243 to the container as a virtual user space. The container engine 244b knows the containers running on the virtual machines.

Further, in a three-tier virtual environment, a container executes a container library 245b and a sidecar proxy 246b. The functionality of the container library 245b is similar to that of the container library 245a. The functionality of sidecar proxy 246b is similar to sidecar proxy 246a. Furthermore, in a three-layer virtual environment, the service node 247b is placed above the container. Service node 247b is one application. Service node 247b may communicate and cooperate with service node 247a.

Here, a failure occurring in the function of an application such as the service nodes 247a, 247b may be referred to as an application failure or an "application failure" for short. Further, a failure that occurs in the function of a processing node such as a physical machine, virtual machine, or container is sometimes referred to as an infrastructure failure or "infrastructure failure" for short. In the second embodiment, infrastructure failures are mainly assumed to be physical machine failures and virtual machine failures.

FIG. 5 is a diagram showing an example of a system configuration graph.
In the description of the second embodiment, the arrangement of processing nodes and service nodes as shown in FIG. 5 will be used as an example. The monitored system 200 includes physical machines 201, 202, virtual machines 211, 212, and containers 221, 222, 223, 224, 225, 226 as processing nodes. Additionally, the monitored system 200 includes service nodes 231, 232, 233, 234, 235, and 236. The topology of these processing nodes and service nodes is a tree structure with the switch 203 as the root.

A virtual machine 211 (VM1) and containers 221, 222 (C1, C2) are placed directly above the physical machine 201 (M1). Containers 223 and 224 (C3, C4) are placed directly above the virtual machine 211. A virtual machine 212 (VM2) and a container 226 (C6) are placed directly above the physical machine 202 (M2). A container 225 (C5) is placed directly above the virtual machine 212.

Container 221 executes service node 231 (AP1). Container 222 runs service node 232 (AP2). Container 223 executes service node 233 (AP3). Container 224 runs service node 234 (AP4). Container 225 executes service node 235 (AP5). Container 226 runs service node 236 (AP6).

FIG. 6 is a diagram showing an example of a service mesh graph.
The service nodes 231, 232, 233, 234, 235, and 236 cooperate to realize one service by communicating with each other. For example, a lightweight communication API such as REST (Representational State Transfer) is used for communication between service nodes. By using sidecar proxies corresponding to the sidecar proxies 246a and 246b described above, it is possible to grasp the status of message communication between service nodes.

Some pairs of service nodes may communicate directly, while others may not. By describing an edge between two service nodes that may communicate directly, a service mesh graph as shown in FIG. 6 can be generated. A service mesh graph is an undirected graph that includes nodes indicating service nodes and edges between nodes indicating communication between service nodes. In the description of the second embodiment, the service mesh graph shown in FIG. 6 is used as an example.

Service node 231 (AP1) communicates with service node 233. Service node 232 (AP2) communicates with service nodes 233, 234, and 236. Service node 233 (AP3) communicates with service nodes 231, 232, and 234. Service node 234 (AP4) communicates with service nodes 232 and 233. Service node 235 (AP5) communicates with service node 236. Service node 236 (AP6) communicates with service nodes 232 and 235. Pairs of service nodes other than those listed above do not communicate directly. However, since this service mesh graph is a connected graph, there is a path connecting any two service nodes. Therefore, all service nodes are cooperating.

Next, generation of a cause determination model will be explained.
FIG. 7 is a diagram showing an example of generation of a cause determination model.
The management server 100 generates a cause determination model 151. The cause determination model 151 is a regression model generated by logistic regression analysis. The cause determination model 151 uses a feature vector indicating the relationship between one application failure and one infrastructure failure as an explanatory variable. Further, the cause determination model 151 uses a confidence level indicating the probability that the cause of the one application failure is the one infrastructure failure as an objective variable.

Therefore, the cause determination model 151 determines whether or not the root cause of an application failure is an infrastructure failure regarding an application failure and an infrastructure failure that occur at the same time in the monitored system 200. An infrastructure failure may propagate to a service node, such as when an abnormal termination of a communication process in a certain processing node causes a message communication failure in a certain service node. In that case, the system administrator can respond to the application failure by eliminating the infrastructure failure that is causing it. Therefore, by using the cause determination model 151, useful information can be provided to the system administrator.

In generating the cause determination model 151, the management server 100 generates a record in which a feature vector 141 and a teacher label 142 are associated with each other for each pair of application failure and infrastructure failure that occurred in the past, and adds the record to the training data. . The feature vector 141 is a four-dimensional vector including evaluation values v ₁ , v ₂ , v ₃ , and v ₄ . The evaluation value v ₁ represents the parent-child distance 143. The evaluation value v ₂ represents the service distance 144 . The evaluation value _v3 represents a text similarity of 145. The evaluation value _v4 represents a time difference of 146. The teacher label 142 is a flag indicating whether or not the cause of the application failure of interest is an infrastructure failure of interest. Whether or not this is the cause of the failure can be determined as a result of the failure handling work performed by the system administrator in the past. If the infrastructure failure is not the cause of the application failure, the teacher label 142 is "0". If an infrastructure failure is the cause of the application failure, the teacher label 142 is "1".

The parent-child distance 143 indicates the hierarchical relationship of processing nodes between an application failure and an infrastructure failure. Specifically, the parent-child distance 143 indicates the hierarchical relationship between the container that executes the service node where the application failure occurred, that is, the container that detected the application failure, and the physical machine or virtual machine that detected the infrastructure failure. . When the above container is placed above the above physical machine or virtual machine, that is, when a parent-child relationship exists, the hierarchical difference between the container and the physical machine or virtual machine is the parent-child distance 143. Become. On the other hand, if the above-mentioned container is not placed above the above-mentioned physical machine or virtual machine, that is, if there is no parent-child relationship, the parent-child distance 143 is considered to be "0".

For example, in FIG. 5, if an application failure occurs in the service node 233 and an infrastructure failure occurs in the virtual machine 211, the parent-child distance 143 is "1". When an application failure occurs in the service node 233 and an infrastructure failure occurs in the physical machine 201, the parent-child distance 143 becomes "2". When an application failure occurs in the service node 233 and an infrastructure failure occurs in the physical machine 202, the parent-child distance 143 becomes "0". The smaller the parent-child distance 143 (excluding "0"), the higher the possibility that an infrastructure failure is the cause of the application failure. Note that the parent-child distance 143 can be calculated, for example, by calculating the depth of each processing node in a tree structure with the switch 203 as the root, and finding the difference in depth between two processing nodes.

The service distance 144 indicates the communication relationship between service nodes related to application failures and infrastructure failures. Specifically, the service distance 144 indicates the communication relationship between the service node where the application failure has occurred and another service node running on the basis of the physical machine or virtual machine that detected the infrastructure failure. If a path exists on the service mesh graph between the service node where the application failure has occurred and the other service node mentioned above, the length (distance) of the path is the service distance 144. If there are two or more of the above other service nodes, the minimum value of the distances calculated for each of the two or more other service nodes is the service distance 144. On the other hand, if there is no path to any other service node, the service distance 144 is considered to be "0".

For example, in FIG. 5, assume that an application failure occurs in the service node 233 and an infrastructure failure occurs in the virtual machine 211. In this case, service nodes 233 and 234 are running above the virtual machine 211 that has an infrastructure failure. Then, in the service mesh graph of FIG. 6, the minimum distance between {AP3} and {AP3, AP4} is "0". Therefore, the service distance 144 is "0".

Further, for example, in FIG. 5, assume that an application failure occurs in the service node 233 and an infrastructure failure occurs in the physical machine 202. In this case, service nodes 235 and 236 are running above the physical machine 202 that has an infrastructure failure. Then, in the service mesh graph of FIG. 6, the minimum distance between {AP3} and {AP5, AP6} is "2". Therefore, the service distance 144 is "2". The smaller the service distance 144 (excluding "0"), the higher the possibility that an infrastructure failure is the cause of the application failure. Note that the service distance 144 can be calculated, for example, by executing a shortest path search algorithm such as Dijkstra's algorithm on the service mesh graph.

The text similarity 145 indicates the similarity of error messages between an application failure and an infrastructure failure. The more common words there are between the error message of the application failure and the error message of the infrastructure failure, the greater the text similarity 145 becomes. The larger the text similarity 145, the higher the possibility that an infrastructure failure is the cause of the application failure.

In the second embodiment, the cosine similarity of Bug of Words is used as the text similarity 145. When generating Bug of Words, the management server 100 divides a character string into words for each error message, and counts the number of occurrences of the same word. The management server 100 generates a vector listing the number of occurrences of each word as a Bug of Words for each error message. The management server 100 calculates cosine similarity between two vectors corresponding to two error messages. The cosine similarity is a real number greater than or equal to 0 and less than or equal to 1. The closer the cosine similarity is to 1, the higher the similarity between the two error messages, and the closer the cosine similarity is to 0, the lower the similarity between the two error messages.

However, the cosine similarity is an example of the text similarity 145. As an index of text similarity 145, other indexes such as one using distributed representation and one using edit distance (Levenshtein distance) are also considered. Preprocessing may also be performed, such as removing identifiers such as host names and addresses from error messages.

A time difference 146 indicates a delay between the occurrence of an application failure and the occurrence of an infrastructure failure. If the application failure occurrence time is after the infrastructure failure occurrence time, the time difference 146 is the difference obtained by subtracting the infrastructure failure occurrence time from the application failure occurrence time. In the second embodiment, time (hour) is used as the unit of time difference 146. On the other hand, if the application failure occurrence time is before the infrastructure failure occurrence time, the time difference 146 is considered to be "0". For example, if the app failure occurrence time is 16:00 on April 1, 2020, and the infrastructure failure occurrence time is 15:00 on April 1, 2020, the time difference 146 will be "1". . The smaller the time difference 146 (excluding "0"), the higher the possibility that an infrastructure failure is the cause of the application failure.

By including the evaluation value v ₁ indicating the parent-child distance 143 in the feature vector 141, it is possible to evaluate vertical fault propagation in which a fault in a processing node in a lower hierarchy affects a processing node in an upper hierarchy. . Further, since the feature vector 141 includes the evaluation value v ₂ indicating the service distance 144, it is possible to evaluate horizontal fault propagation in which a fault in one service node affects another service node. Further, since the feature vector 141 includes the evaluation value _v3 indicating the text similarity 145, it is possible to evaluate the similarity of the failure contents. Further, since the feature vector 141 includes the evaluation value _v4 indicating the time difference 146, it is possible to evaluate the simultaneity of occurrence of the application failure and the infrastructure failure.

On the other hand, the feature vector 141 does not include evaluation values regarding identifiers such as host names and addresses. This is because if the identifier of the processing node or service node is used, the determination accuracy of the cause determination model 151 may decrease if the configuration of the processing node changes, and the cause determination model 151 may have to be regenerated. It is.

In the monitored system 200, the correspondence between physical machines and virtual machines may change due to addition, movement, or deletion of virtual machines. Furthermore, in the monitored system 200, the correspondence between a physical machine or a virtual machine and a container may change due to addition, movement, or deletion of a container. In this regard, if an identifier is used as an input to the cause determination model 151, there is a possibility that the usefulness of the cause determination model 151 may be reduced due to configuration changes. On the other hand, by not using an identifier as an input to the cause determination model 151, the cause determination model 151 can continue to be used even if the arrangement of virtual machines or containers is changed.

The management server 100 generates the cause determination model 151 using training data including the feature vector 141 and teacher label 142 as described above. The cause determination model 151 is expressed, for example, as a logistic function as shown in equation (1). The logistic function of Equation (1) calculates the confidence level P from the evaluation values v ₁ ′, v ₂ ′, v ₃ , and v ₄ ′. The confidence level P calculated here is a real number greater than 0 and less than 1. The closer the confidence level P is to 1, the higher the probability that an infrastructure failure is the cause of the application failure, and the closer the confidence level P is to 0, the lower the possibility that the infrastructure failure is the cause of the application failure.

The evaluation values v ₁ ′, v ₂ ′, and v ₄ ′ are converted from the evaluation values v ₁ , v ₂ , and v ₄ as described later. Moreover, this logistic function includes parameters α, β ₁ , β ₂ , β ₃ , and β ₄ . The values of parameters α, β ₁ , β ₂ , β ₃ , and β ₄ are determined through machine learning using training data. Parameter α is a constant. The parameter β ₁ is a weighting coefficient of the evaluation value v ₁ ′. The parameter β ₂ is a weighting coefficient of the evaluation value v ₂ ′. Parameter β ₃ is a weighting coefficient of evaluation value v ₃ . Parameter β ₄ is a weighting coefficient of evaluation value v ₄ ′.

The evaluation value v ₁ ' is converted from the evaluation value v ₁ as shown in Equation (2). When v ₁ =0, v ₁ '=0; otherwise, the larger v ₁ is, the smaller v ₁ ' is. The evaluation value v ₁ ′ is a real number from 0 to 1. The evaluation value v ₂ ′ is converted from the evaluation value v ₂ as shown in Equation (3). When v ₂ =0, v ₂ ′=0, and in other cases, the larger v ₂ is, the smaller v ₂ ′ is. The evaluation value v ₂ ′ is a real number from 0 to 1. The evaluation value v ₄ ' is converted from the evaluation value v ₄ as shown in equation (4). When v ₄ =0, v ₄ ′=0, and in other cases, the larger v ₄ is, the smaller v ₄ ′ is. The evaluation value v ₄ ′ is a real number greater than or equal to 0.

The management server 100 uses the cause determination model 151 generated in this manner to evaluate the relationship between an application failure and an infrastructure failure that occurred at the same time. When a certain application failure occurs, two or more infrastructure failures may occur immediately after. In that case, the management server 100 uses the cause determination model 151 to calculate the degree of certainty between the application failure and each of the two or more infrastructure failures. The management server 100 sorts two or more infrastructure failures in descending order of certainty and presents them to the system administrator.

FIG. 8 is a diagram showing an example of a system management screen.
The system management screen 152 is transmitted from the management server 100 to the client terminal 31 and displayed on the display of the client terminal 31. System management screen 152 includes a message indicating that a failure has occurred in a certain application. The system management screen 152 also includes a message indicating that a failure has occurred in a processing node such as a physical machine or a virtual machine at the same time as the application failure. Furthermore, the system management screen 152 includes, for each infrastructure failure, a confidence value indicating the possibility that the failure is the cause of the application failure. Infrastructure failure messages are sorted in descending order of confidence.

For example, the system management screen 152 reports an application failure of the service node 233 (AP3). The system management screen 152 also reports that there is an infrastructure failure in the virtual machine 211 (VM1) and that there is a 90% possibility that the infrastructure failure is the cause of the application failure. The system management screen 152 also reports that there is an infrastructure failure in the physical machine 202 (M2) and that there is a 40% probability that the infrastructure failure is the cause of the application failure. This allows system administrators to efficiently analyze the causes of application failures and resolve them.

Next, the functions of the management server 100 will be explained.
FIG. 9 is a block diagram showing an example of the functions of the management server.
The management server 100 includes a failure information storage section 121, a configuration information storage section 122, a service information storage section 123, and a model storage section 124. These storage units are realized using, for example, a storage area of the RAM 102 or the HDD 103. The management server 100 also includes a fault monitoring section 125, a configuration monitoring section 126, a service monitoring section 127, a learning section 128, and a cause determining section 129. These processing units are realized using, for example, a program.

The fault information storage unit 121 stores fault information indicating a fault that has occurred in the monitored system 200. Further, the failure information storage unit 121 stores failure response information indicating the system administrator's response to the failure. The configuration information storage unit 122 stores configuration information indicating the arrangement of physical machines, virtual machines, and containers. The service information storage unit 123 stores information on a service mesh graph indicating communication relationships between a plurality of service nodes. The service information storage unit 123 also stores information indicating the arrangement of service nodes in containers. The model storage unit 124 stores the cause determination model 151 generated by the management server 100.

The fault monitoring unit 125 monitors the monitored system 200 for faults. The fault monitoring unit 125 collects information regarding faults from the monitored system 200 and stores the fault information in the fault information storage unit 121. Information regarding the failure can be collected from the host OS of the physical machine, the guest OS of the virtual machine, the container library of the container, and the like. As a monitoring method, each processing node such as a physical machine, a virtual machine, and a container may notify the management server 100 of the failure when it detects a failure. Further, the failure monitoring unit 125 may periodically collect error logs from each processing node. Alternatively, the failure monitoring unit 125 may periodically collect various logs from each processing node, analyze the logs to determine the presence or absence of a failure, and extract failure information from the logs when a failure is detected. .

The configuration monitoring unit 126 monitors the configuration of the virtual environment of the monitored system 200. When the fault monitoring unit 125 detects that the arrangement of the physical machine, virtual machine, and container has been changed, it stores configuration information indicating the changed arrangement in the configuration information storage unit 122. The relationship between physical machines and virtual machines can be gleaned from the hypervisor running on the physical machine. The relationship between a physical or virtual machine and a container can be gleaned from a container engine running on the physical or virtual machine.

The service monitoring unit 127 monitors a plurality of service nodes (applications) placed in the monitored system 200. When the service monitoring unit 127 detects that the communication relationship between the plurality of service nodes has changed, it stores information indicating the changed communication relationship in the service information storage unit 123. Communication relationships between service nodes can be gleaned from sidecar proxies running in containers. Further, when the service monitoring unit 127 detects that the arrangement of service nodes in the container has been changed, it stores information indicating the changed arrangement in the service information storage unit 123. Service node placement can be gleaned from container libraries or sidecar proxies running in containers.

The learning unit 128 generates training data from the information stored in the failure information storage unit 121, configuration information storage unit 122, and service information storage unit 123, and generates a cause determination model 151 through logistic regression analysis. The learning unit 128 stores the cause determination model 151 in the model storage unit 124. Furthermore, when the system administrator provides failure response information corresponding to the prediction of the cause of the failure by the cause determination unit 129, the learning unit 128 improves the determination accuracy based on the error between the prediction and the correct answer. The cause determination model 151 is updated.

When a new application failure is detected, the cause determination unit 129 generates a system management screen 152 based on the failure information stored in the failure information storage unit 121 and sends it to the client terminal 31. At this time, the cause determination unit 129 generates a feature vector from the information stored in the failure information storage unit 121, configuration information storage unit 122, and service information storage unit 123, and generates a feature vector from the cause determination model 151 stored in the model storage unit 124. Enter. As a result, the cause determining unit 129 calculates the reliability of each infrastructure failure occurring at the same time as the application failure, and transmits the calculated reliability in the system management screen 152.

Thereafter, the cause determination unit 129 receives the failure handling information from the client terminal 31 and stores it in the failure information storage unit 121. The failure handling information includes a correct teacher label indicating whether or not the pair of application failure and infrastructure failure was the cause of the failure for which the confidence was calculated. The cause determination unit 129 causes the learning unit 128 to update the cause determination model 151.

FIG. 10 is a diagram showing an example of a failure table.
The failure table 131 is stored in the failure information storage unit 121. The failure table 131 stores a plurality of records each including items of failure ID, time, detected node, failure type, message, response flag, and cause ID. An identifier for identifying a failure is registered as the failure ID. As the time, the time when the failure occurred or the time when the failure was recognized is registered. The identifier of the processing node that detected the failure is registered as the detection node. The processing node that detects the failure can be a physical machine, a virtual machine, or a container.

As the failure type, the category of application failure or infrastructure failure is registered. A service node (application) failure detected in a container is an application failure. A failure detected in a physical or virtual machine is an infrastructure failure. The text of the error message included in the log is registered as the message. As the response flag, a flag indicating whether the fault has already been resolved or not resolved by the system administrator's troubleshooting work is registered. When the system administrator concludes that the cause of the failure is another failure through failure handling work, the failure ID of the cause failure is registered as the cause ID. The response flag and cause ID are information that is added based on the failure response information.

FIG. 11 is a diagram showing an example of a configuration table.
The configuration table 132 is stored in the configuration information storage unit 122. The configuration table 132 stores a plurality of records each including items of time, parent node, child node, and distance. The time at which the configuration change was detected is registered as the time. The identifier of the lower processing node (closer to the physical machine) is registered as the parent node. As a child node, the identifier of a processing node located above the parent node (closer to the application) is registered.

The parent node and child node are each a physical machine, virtual machine, or container. A child node may also be running on a parent node. Further, another processing node may be executed on a parent node, and a child node may be executed on that processing node. A parent-child distance between a parent node and a child node is registered as the distance. Distance is the vertical hierarchy difference. If a virtual machine is running on a physical machine and a container is running on a virtual machine, the distance between the physical machine and the virtual machine is "1" and the distance between the virtual machine and the container is "1". . Further, the distance between the physical machine and the container is "2". Pairs in which there is no parent-child relationship in which one processing node is located above the other processing node, that is, pairs with a distance of "0", do not need to be registered in the configuration table 132.

FIG. 12 is a diagram showing an example of a service distance table and a service arrangement table.
The service distance table 133 is stored in the service information storage unit 123. The service distance table 133 stores a plurality of records each including items of time, start point node, end point node, and distance. The time at which a change in the service mesh graph is detected is registered as the time. The service mesh graph may change due to the addition or deletion of containers, updates to application programs, etc.

As the starting point node, the identifier of one of the two service nodes that directly communicate is registered. As the end node, the identifier of the other of the two service nodes that directly communicate is registered. Since the service mesh graph is an undirected graph, it is not necessary to register a record in which the start point node and end point node are swapped. The number of hops of the path between the starting point node and the ending point node in the service mesh graph is registered as the distance.

The service placement table 134 is stored in the service information storage unit 123. The service placement table 134 stores a plurality of records each including items of time, service node, and container. The time at which the change in the arrangement of the service node is detected is registered as the time. The identifier of the service node to be placed is registered as the service node. The identifier of the container in which the service node is placed is registered as the container. The arrangement of service nodes may change depending on the addition or deletion of containers.

FIG. 13 is a diagram showing an example of a training data table.
The training data table 135 is generated by the learning unit 128. A training data table 135 may be stored in the model storage unit 124. The training data table 135 includes a plurality of records each including items of application failure, infrastructure failure, evaluation values v ₁ , v ₂ , v ₃ , v ₄ , evaluation values v _{1 ′} , v ₂ ′, v ₄ ′, and teacher label. Remember.

As an application failure, a failure ID of a failure whose failure type is an application failure is registered. As an infrastructure failure, a failure ID of a failure whose failure type is an infrastructure failure is registered. As the evaluation value _v1 , the parent-child distance calculated for the pair of application failure and infrastructure failure is registered. The service distance calculated for the above pair is registered as the evaluation value _v2 . The text similarity calculated for the above pair is registered as the evaluation value _v3 . The time difference calculated for the above pair is registered as the evaluation value _v4 . Correction values converted from the evaluation values v ₁ , v ₂ , v ₄ are registered as the evaluation values v _{1 ′} , v _{2 ′} , v _{4 ′} . A flag indicating whether an infrastructure failure is the cause of the application failure is registered as a teacher label.

Next, the processing procedure of the management server 100 will be explained.
FIG. 14 is a flowchart showing an example of a model generation procedure.
(S10) The learning unit 128 separately extracts failure information about a plurality of application failures and failure information about a plurality of infrastructure failures from the failure table 131.

(S11) The learning unit 128 selects one set of an application failure and an infrastructure failure. Note that in step S10, when failure information of m application failures and failure information of n infrastructure failures are extracted, there are m×n candidates for pairs of application failures and infrastructure failures.

(S12) The learning unit 128 identifies the detection node that detected the application failure and the detection node that detected the infrastructure failure. The learning unit 128 searches the configuration table 132 for the parent-child distance between the two detected nodes and sets it as an evaluation value _v1 . Note that if the parent-child distance between two detected nodes is not registered in the configuration table 132, v ₁ =0. Furthermore, when referring to the configuration table 132, information immediately before the failure time is used.

(S13) The learning unit 128 refers to the configuration table 132 and searches for a container that has a parent-child relationship with the detection node that detected the infrastructure failure. The learning unit 128 refers to the service placement table 134 and searches for a service node to be executed in the searched container. Note that when referring to the service placement table 134, information immediately before the failure time is used.

(S14) The learning unit 128 searches the service distance table 133 for the service distance between the service node where the application failure has occurred and each of the service nodes searched in step S13. The learning unit 128 sets the minimum service distance among the searched service distances as an evaluation value _v2 . Note that if the service distance between different service nodes is not registered in the service distance table 133, the different service nodes are not connected. If all of the service nodes searched in step S13 are disconnected from the service node where the application failure has occurred, v ₂ =0. Furthermore, when referring to the service distance table 133, information immediately before the failure time is used.

(S15) The learning unit 128 extracts error messages from each of the failure information of the application failure and the failure information of the infrastructure failure. The learning unit 128 calculates the text similarity between the two extracted error messages and sets it as an evaluation value _v3 . For example, the learning unit 128 divides the text into words for each error message, calculates a Bug of Words vector, calculates the cosine similarity between the two vectors, and sets the evaluation value _v3 .

(S16) The learning unit 128 extracts the failure time from each of the failure information of the application failure and the failure information of the infrastructure failure. The learning unit 128 calculates the time difference between the two extracted failure times and sets it as an evaluation value _v4 . The unit of time difference is, for example, hour. Note that if the infrastructure failure is slower than the application failure, v ₄ =0.

(S17) The learning unit 128 converts the evaluation value v ₁ calculated in step S12 into an evaluation value v ₁ ' according to formula (2). Further, the learning unit 128 converts the evaluation value v ₂ calculated in step S14 into an evaluation value v ₂ ′ according to formula (3). Further, the learning unit 128 converts the evaluation value v ₄ calculated in step S16 into an evaluation value v ₄ ′ according to formula (4). The learning unit 128 generates a feature vector including evaluation values v _{1 ′} , v ₂ ′, v ₃ , and v _{4 ′} .

(S18) The learning unit 128 determines whether the infrastructure failure of interest is the cause of the application failure, based on the cause ID included in the failure information of the application failure. If the infrastructure failure is not the cause of the application failure, the teacher label is determined to be "0". If an infrastructure failure is the cause of the application failure, the teacher label is determined to be "1".

(S19) The learning unit 128 associates the feature vector generated in step S17 with the teacher label determined in step S18, and adds them to the training data.
(S20) The learning unit 128 determines whether all the pairs of application failures and infrastructure failures have been selected in step S11. If all the groups have been selected, the process advances to step S21, and if there are any unselected groups, the process returns to step S11.

(S21) The learning unit 128 generates the cause determination model 151 through logistic regression analysis using the training data generated through steps S10 to S20. Here, parameters α, β ₁ , β ₂ , β ₃ , and β ₄ included in equation (1) are determined. The learning unit 128 stores the generated cause determination model 151 in the model storage unit 124.

FIG. 15 is a flowchart illustrating an example of a procedure for determining the cause of a failure.
(S30) The cause determining unit 129 detects that a new application failure has occurred. Then, the cause determination unit 129 extracts from the failure table 131 the failure information of the new application failure and the failure information of the unresolved infrastructure failure.

(S31) The cause determining unit 129 selects one infrastructure failure.
(S32) The cause determination unit 129 calculates the evaluation value _v1 indicating the parent-child distance between the application failure detection node and the infrastructure failure detection node, in the same manner as in step S12 described above.

(S33) The cause determination unit 129 searches for a service node that is being executed in the upper layer of the processing node that detected the infrastructure failure, in the same manner as in step S13 described above.
(S34) The cause determining unit 129 calculates an evaluation value _v2 indicating the service distance between service nodes in the same manner as in step S14 described above.

(S35) The cause determination unit 129 calculates an evaluation value _v3 indicating the similarity of error messages between the application failure and the infrastructure failure in the same manner as in step S15 described above.
(S36) The cause determining unit 129 calculates an evaluation value _v4 indicating the time difference between the application failure and the infrastructure failure in the same manner as in step S16 described above.

(S37) The cause determination unit 129 converts the evaluation values v ₁ , v ₂ , v ₄ into evaluation values v ₁ ′, v ₂ ′, v ₄ ′ in the same manner as in step _S17 described above, A feature vector including ', v ₂ ', v ₃ and v ₄ ' is generated.

(S38) The cause determination unit 129 inputs the feature vector generated in step S37 to the cause determination model 151, and calculates the degree of certainty.
(S39) The cause determining unit 129 determines whether all infrastructure failures have been selected in step S31. If all infrastructure failures have been selected, the process advances to step S40, and if there are unselected infrastructure failures, the process returns to step S31.

(S40) The cause determining unit 129 sorts unresolved infrastructure failures in descending order of certainty.
(S41) The cause determination unit 129 generates a system management screen 152 that includes information on application failures, information on unresolved infrastructure failures arranged in descending order of certainty, and certainty of each infrastructure failure, and manages the system. The screen 152 is sent to the client terminal 31.

FIG. 16 is a flowchart illustrating an example of a model update procedure.
This model update is executed after the failure cause determination in FIG. 15.
(S50) The cause determining unit 129 receives failure handling information from the client terminal 31. The failure response information includes a failure ID of an application failure, a failure ID of an infrastructure failure, and a teacher label indicating whether the infrastructure failure was the cause of the application failure. The teacher label is the result of judgment by the system administrator through troubleshooting work. If the infrastructure failure is not the cause of the application failure, the teacher label will be "0". If the infrastructure failure is the cause of the application failure, the teacher label will be "1". The cause determination unit 129 updates the failure table 131 based on the received failure handling information.

(S51) If the feature vector generated in step S37 described above is saved, the learning unit 128 acquires the feature vector. On the other hand, if the feature vector is not saved, the learning unit 128 regenerates the feature vector in the same manner as steps S32 to S37.

(S52) The learning unit 128 acquires the certainty factor, if the certainty factor calculated in step S38 described above is saved. On the other hand, if the confidence level is not saved, the learning unit 128 recalculates the confidence level in the same manner as in step S38.

(S53) The learning unit 128 updates the cause determination model 151 through online learning using the cause determination model 151, feature vector, confidence level, and teacher label. Gradient methods such as stochastic gradient descent can be used for online learning. For example, the learning unit 128 calculates the error between the confidence level and the teacher label, and the amount of change in error when each of the values of parameters α, β ₁ , β ₂ , β ₃ , and β ₄ is changed by a minute amount. From this, the gradient of the error with respect to the parameters α, β ₁ , β ₂ , β ₃ , and β ₄ is calculated. The learning unit 128 changes the values of the parameters α, β ₁ , β ₂ , β ₃ , and β ₄ by an amount obtained by multiplying the error gradient by a predetermined learning rate. However, instead of performing online learning, the current set of feature vector and teacher label may be added to the training data, and the machine learning in step S21 described above may be re-executed.

According to the information processing system of the second embodiment, when an application failure and an infrastructure failure occur at the same time, the possibility that the infrastructure failure is the root cause of the application failure is evaluated, and the certainty level thereof is evaluated. is presented to the system administrator. If multiple infrastructure failures occur, the multiple infrastructure failures are presented in order of certainty. This reduces the burden of troubleshooting work on the system administrator and shortens the time required to resolve the trouble. Especially in complex virtual environments where physical machines, virtual machines, and containers are arranged in a hierarchical manner, infrastructure failures can cause problems even when many fine-grained applications are distributed and executed by many containers. You can streamline the cause analysis of application failures.

Additionally, a cause determination model generated from past failure information through machine learning is used to evaluate the relationship between application failures and infrastructure failures. The explanatory variables of the cause determination model include evaluation values that indicate the hierarchical relationship of processing nodes. Therefore, it is possible to consider the possibility of vertical fault propagation in which a fault occurring in a processing node in a lower hierarchy affects a processing node in an upper hierarchy. Furthermore, the feature vector includes an evaluation value indicating the communication relationship between applications. Therefore, it is possible to consider the possibility of horizontal fault propagation in which a fault in one application affects another application.

The feature vector also includes an evaluation value indicating the degree of similarity of the error messages. Therefore, the similarity of failure contents can be taken into consideration. The feature vector also includes an evaluation value that indicates the time difference between the application failure and the infrastructure failure. Therefore, the possibility of fault propagation can be evaluated from the aspect of delay time. In this way, by comprehensively utilizing the above four viewpoints, the determination accuracy of the cause determination model can be improved. Furthermore, identifiers of physical machines, virtual machines, containers, and applications are not used as input to the cause determination model. Therefore, even if the configuration of the virtual environment is changed, the determination accuracy of the generated cause determination model is unlikely to decrease, and the usefulness of the cause determination model can be maintained.

10 Control device 11 Storage unit 12 Processing unit 13, 14 Fault information 15 Teacher label 16 Feature information 16a, 16b, 16c Evaluation value 17 Model 20 Information processing system 21, 22, 23 Processing node 24, 25 Application

Claims (8)

to the computer,
Each of the plurality of processing nodes is a processing node capable of executing an application using allocated resources, and includes a plurality of processing nodes that can be arranged hierarchically using virtualization software, and each of the plurality of applications is a processing node capable of executing an application using the allocated resources. Regarding an information processing system executed by any of the processing nodes, first failure information indicating a failure of a first application, second failure information indicating a failure of a second processing node, and the first failure obtaining a teacher label indicating the presence or absence of a relationship between the information and the second failure information;
a first failure information indicating a hierarchical relationship of arrangement between a first processing node that executes the first application and the second processing node based on the first failure information and the second failure information; an evaluation value; a second evaluation value indicating a dependency relationship between the first application and a second application executed on a processing node arranged on the second processing node; a third evaluation value indicating the degree of similarity between a first error message included in the failure information and a second error message included in the second failure information;
Using training data in which feature information including the first evaluation value, the second evaluation value, and the third evaluation value is associated with the teacher label, the feature information regarding the two pieces of fault information is corresponded to. Generating a model that estimates the presence or absence of a relationship between the two failure information from the input data,
A control program that executes processing. The output of the model includes a confidence level indicating the strength of the association between the two failure information,
The computer further includes:
After the model is generated, obtaining third failure information indicating a failure of one application and a plurality of fourth failure information indicating failures of different processing nodes,
For each set of the third failure information and one fourth failure information among the plurality of fourth failure information, based on the third failure information and the one fourth failure information, generating input data, inputting the input data into the model and calculating the confidence level;
Prioritizing and outputting the plurality of fourth failure information based on the confidence level;
The control program according to claim 1, which causes the control program to execute a process. The computer further includes:
Obtain fault handling information indicating an analysis result of a fault in the first application, and if the cause of the fault indicated by the fault handling information includes a fault in the second processing node, generate the teacher label indicating that there is a relationship. and when the cause of the failure indicated by the failure handling information does not include a failure of the second processing node, generating the teacher label indicating no relation;
The control program according to claim 1, which causes the control program to execute a process. The first evaluation value indicates the distance between the first processing node hierarchy and the second processing node hierarchy, and the second evaluation value indicates the communication relationship between the plurality of applications. The third evaluation value indicates the distance between the first application and the second application in a mesh graph showing the difference between the words included in the first error message and the second error message. indicates the degree of similarity between the words contained in it,
The control program according to claim 1. The characteristic information further includes a fourth evaluation value indicating a time difference between a time when a failure occurs in the first application and a time when a failure occurs in the second processing node.
The control program according to claim 1. The plurality of processing nodes include two or more physical machines, two or more virtual machines each placed on one of the two or more physical machines, and each one placed on one of the two or more virtual machines. and two or more containers arranged,
The first processing node is any one of the two or more containers, and the second processing node is any one of the two or more physical machines and the two or more virtual machines.
The control program according to claim 1. The computer is
Each of the plurality of processing nodes is a processing node capable of executing an application using allocated resources, and includes a plurality of processing nodes that can be arranged hierarchically using virtualization software, and each of the plurality of applications is a processing node capable of executing an application using the allocated resources. Regarding an information processing system executed by any of the processing nodes, first failure information indicating a failure of a first application, second failure information indicating a failure of a second processing node, and the first failure obtaining a teacher label indicating the presence or absence of a relationship between the information and the second failure information;
a first failure information indicating a hierarchical relationship of arrangement between a first processing node that executes the first application and the second processing node based on the first failure information and the second failure information; an evaluation value; a second evaluation value indicating a dependency relationship between the first application and a second application executed on a processing node arranged on the second processing node; a third evaluation value indicating the degree of similarity between a first error message included in the failure information and a second error message included in the second failure information;
Using training data in which feature information including the first evaluation value, the second evaluation value, and the third evaluation value is associated with the teacher label, the feature information regarding the two pieces of fault information is corresponded to. Generating a model that estimates the presence or absence of a relationship between the two failure information from the input data,
Control method. Each of the plurality of processing nodes is a processing node capable of executing an application using allocated resources, and includes a plurality of processing nodes that can be arranged hierarchically using virtualization software, and each of the plurality of applications is a processing node capable of executing an application using the allocated resources. Regarding an information processing system executed by any of the processing nodes, first failure information indicating a failure of a first application, second failure information indicating a failure of a second processing node, and the first failure a storage unit that stores a teacher label indicating the presence or absence of a relationship between the information and the second failure information;
a first failure information indicating a hierarchical relationship of arrangement between a first processing node that executes the first application and the second processing node based on the first failure information and the second failure information; an evaluation value; a second evaluation value indicating a dependency relationship between the first application and a second application executed on a processing node arranged on the second processing node; and a third evaluation value indicating the degree of similarity between the first error message included in the failure information and the second error message included in the second failure information, and , using training data in which the feature information including the second evaluation value and the third evaluation value is associated with the teacher label, the two failure information are extracted from the input data corresponding to the feature information about the two failure information. a processing unit that generates a model that estimates the presence or absence of relevance of failure information;
A control device having:

Images