JP2022142456A - Abnormality handling program, abnormality handling system, and abnormality handling method - Google Patents

Abstract

To suppress a delay in handling for an abnormality of a container.SOLUTION: An abnormality handling program causes a computer to perform processing in which priorities of abnormalities having occurred in each of a plurality of containers are set, a logic for resolving the abnormalities is generated for each type of the abnormalities, and the logic is performed to the abnormalities in order of the priorities.SELECTED DRAWING: Figure 7

Description

The present invention relates to an anomaly handling program, an anomaly handling system, and an anomaly handling method.

One of the technologies for virtualizing computers is container virtualization technology. Container virtualization technology uses a part of the kernel of the OS (Operating System) to perform virtualization, so it has the advantage of having less virtualization overhead and being lighter than VM (Virtual Machine) virtualization technology. be. In the container virtualization technology, a plurality of mutually independent user spaces are generated. These user spaces are called containers, each of which runs an application program.

When building a business system using containers, an architecture called MSA (Micro Service Architecture), in which each container executes one application program, may be adopted. MSA is an architecture that divides a business system into multiple functions and executes application programs that implement each function in containers. As mentioned above, containers are lightweight, so MSA using containers has the advantage of being able to easily scale out business systems.

However, in MSA, for example, the number of containers increases rapidly for the purpose of reducing the load when the load of one container increases, and the dependency relationship between containers becomes complicated. Therefore, when an abnormality occurs in a container, it becomes extremely difficult for the operator of the business system to deal with the abnormality, resulting in a delay in dealing with the abnormality.

JP 2013-161305 A WO2013/035243

According to one aspect, it is an object to suppress delays in dealing with abnormalities in containers.

According to one aspect, the priority of anomalies occurring in each of a plurality of containers is set, logic for resolving the anomaly is generated for each type of anomaly, and the anomalies are treated in order of the priority. An anomaly handling program is provided for causing a computer to execute a process of executing the logic.

According to one aspect, it is possible to suppress delays in dealing with abnormalities in containers.

FIG. 1 is a schematic diagram showing a method of monitoring a business system. FIG. 2 is a schematic diagram (part 1) showing the problem. FIG. 3 is a schematic diagram (part 2) showing the problem. FIG. 4 is a schematic diagram showing another method of monitoring a business system. FIG. 5 is a schematic diagram (part 3) showing the problem. FIG. 6 is a schematic diagram (part 4) showing the problem. FIG. 7 is a functional configuration diagram of the abnormality handling system according to this embodiment. FIG. 8 is a schematic diagram of service information. FIGS. 9A and 9B are schematic diagrams showing an example of a method of acquiring service information. FIG. 10 is a schematic diagram of a method of acquiring the item "Using_service" included in the service information. FIG. 11 is a schematic diagram of anomaly information resources. FIG. 12 is a schematic diagram of a logic database. FIG. 13 is a schematic diagram of an operational policy. FIG. 14 is a flow chart showing the flow of processing when starting the operation of the business system. FIG. 15 is a flowchart of an abnormality coping method according to this embodiment. FIG. 16 is a flowchart of the abnormality handling process. FIG. 17 is a schematic diagram for explaining the abnormality handling according to the first example of the present embodiment. FIG. 18(a) is a schematic diagram of service information according to the first example of the present embodiment, and FIG. 18(b) is a schematic diagram of anomaly information resources generated by the control unit in the first example of the present embodiment. . FIG. 19 is a schematic diagram of logic generated by the logic generation unit in the first example of the present embodiment. FIG. 20 is a schematic diagram for explaining the abnormality handling according to the second example of the present embodiment. FIG. 21 is a schematic diagram showing service information according to the second example of the present embodiment. 22 is a schematic diagram of a service topology generated by the control unit using the service information of FIG. 21. FIG. FIG. 23 is a schematic diagram of anomaly information resources generated by the control unit in the second example of the present embodiment. FIG. 24 is a hardware configuration diagram of an abnormality handling device according to this embodiment.

Prior to the description of the present embodiment, matters examined by the inventors of the present application will be described.

FIG. 1 is a schematic diagram showing a method of monitoring a business system.

FIG. 1 illustrates a business system 1 realized by a plurality of services 4. As shown in FIG. The business system 1 is a system that employs MSA, and individual functions of MSA are implemented by services 4 .

Each service 4 is realized by each application program of a plurality of containers activated on the container base 2. FIG. The container infrastructure 2 is a program that automatically deploys multiple containers, and examples thereof include Kubernetes (registered trademark) and OpenShift (registered trademark).

In this example, each container is monitored by container monitoring software 5 in order to determine whether an abnormality has occurred in the container that implements the service 4 . The container monitoring software 5 displays an alert on the operation terminal 6 of the operator of the business system 1 when an abnormality of the container is detected. Then, the operator of the business system 1 deals with the abnormality in the order in which the alerts are notified.

However, since the container monitoring software 5 displays alerts regardless of minor or serious abnormalities, this method may delay the operator of the business system 1 in dealing with serious abnormalities.

FIG. 2 is a schematic diagram illustrating this problem. In the example in Figure 2, it is assumed that each container has an abnormality of "minor A" to "minor D", which is a minor abnormality, and an abnormality of "major A", which is a serious serious abnormality. .

In this case, since the operator deals with the errors in the order in which they occur, there is a delay in dealing with the serious error of "serious A", which causes a problem in the operation of the business system 1 itself. Furthermore, even if the operator of the business system 1 decides the order of dealing with anomalies in consideration of the urgency of the anomaly, it takes time to determine the order of dealing with the anomaly, which may delay the handling of the anomaly.

It is also conceivable to automate the handling of anomalies using a script rather than having the operator deal with the anomalies manually. However, if the script is simply constructed, the script will deal with the anomaly in the order that the alert was notified, as in the above case, so the handling of the serious anomaly will be delayed.

Moreover, in this method, the operator needs to check whether the abnormality has been resolved after the abnormality has been dealt with, and if the abnormality has not been resolved, the same abnormality must be dealt with again.

FIG. 3 is a schematic diagram illustrating this problem. In FIG. 3, it is assumed that the operator has dealt with the same abnormality many times. This increases the burden on the operator and makes it unbearable.

FIG. 4 is a schematic diagram showing another method of monitoring a business system. In FIG. 4, the same elements as those explained in FIG. 1 are denoted by the same reference numerals as in FIG. 1, and the explanation thereof will be omitted below.

In the example of FIG. 4, the health check mechanism 2a provided in the container base 2 detects an abnormality in the container executing the service 4, and when the abnormality is detected, the health check mechanism 2a eliminates the abnormality.

However, the types of abnormalities that can be monitored by the health check mechanism 2a are limited to abnormalities such as the stoppage of the container itself, and the health check mechanism 2a cannot detect abnormalities inside the container.

Furthermore, the health check mechanism 2a cannot detect high-level abnormalities occurring across multiple containers.

FIG. 5 is a schematic diagram showing the problem. As shown in FIG. 5, the health check mechanism 2a obtains a log of the container by executing a command valid only inside the container, and determines whether or not there is an abnormality based on the log. Therefore, even if a command is executed in one container, it is not possible to acquire the log of a container other than that container, and it is impossible to detect an abnormality across a plurality of services 4 .

In addition, when the health check mechanism 2a cannot resolve a certain abnormality, it repeatedly deals with the abnormality, which may delay the handling of a serious abnormality.

FIG. 6 is a schematic diagram showing the problem. As shown in FIG. 6, while the health check mechanism 2a is repeatedly dealing with an abnormality in the service 4 of a certain container, the abnormality in the service 4 running in another container is left unaddressed. I will be late. The present embodiment will be described below.

(this embodiment)
FIG. 7 is a functional configuration diagram of the abnormality handling system according to this embodiment.

As shown in FIG. 7, the error handling system 20 has a business system 21 and an error handling device 22 . Among them, the business system 21 is a system that employs MSA realized by a plurality of services 23 . Each service 23 has a container 24 and a sidecar proxy 25 . Each container 24 executes one application program for realizing one of a plurality of functions of the business system 21, and the functions of the business system 21 are realized by these application programs.

Each container 24 and sidecar proxy 25 runs on top of container infrastructure 26 . The container platform 26 is a container execution realized by executing a container engine such as Docker (registered trademark) for starting the containers 24 and a container management program such as Kubernetes for managing each container 24 on a computer. environment. The computer that executes the container engine and container management program is not particularly limited, and these programs can be executed on physical machines or virtual machines.

The sidecar proxy 25 is a program for acquiring the parameter of each item of the service information 27 related to the container 24 deployed in the same service 23 as itself, and notifying it to the abnormality handling device 22 .

FIG. 8 is a schematic diagram of the service information 27. As shown in FIG. As shown in FIG. 8, the service information 27 includes "Name", "Id", "Status", "Priority", "Stdout_path", "Logfile_path", "Internalcmd_path", "Using_db", "Using_service", "Operation_type ”, and “SLA” items.

Among these, "Name" is the name of the service 23, and "Id" is an identifier that uniquely identifies the service 23. FIG.

“Status” is information indicating the status of the service 23 . The information includes "Running" indicating that the container 24 included in the service 23 is running, "Error" indicating that an abnormality has occurred in the container 24, and that the abnormality is being dealt with. There is a "ResolvingError" showing. "Status" also includes "Stopped" indicating that the container 24 included in the service 23 is stopped.

“Priority” is a parameter for determining the order of handling an abnormality when an abnormality occurs in the container 24 . "Stdout_path" indicates the standard output destination in the service 23, and "Logfile_path" indicates the output destination of the log file.

“Internalcmd_path” indicates a command path for obtaining information inside the container 24 included in the service 23 .

“Using_db” is the name of the database used by the container 24 included in the service 23 . “Using_service” is the name of the service 23 that includes other containers 24 that are dependent on the container 24 included in the service 23 .

“Operation_type” is information indicating whether the service 23 is essential for realizing the business system 21 .

“SLA” is the SLA (Service Level Agreement) of the business system 21 . As an example, the availability of business system 21 may be taken as an SLA.

The acquisition method of the service information 27 is not particularly limited.

9A and 9B are schematic diagrams showing an example of a method of acquiring service information 27. FIG.

Among them, FIG. 9A is a schematic diagram of a method using the storage area 23a allocated to the service 23. FIG. The storage area 23 a is a storage area accessible from both the container 24 and the sidecar proxy 25 belonging to the same service 23 . For example, the container 24 stores operation logs and error logs in the storage area 23a. Then, the sidecar proxy 25 acquires items included in the service information 27 from the storage area 23a among these logs, and notifies the service information acquiring unit 41 of the items.

On the other hand, FIG. 9(b) shows a model in which the sidecar proxy 25 directly acquires each item included in the service information 27 from the container 24 without going through the storage area 23a and notifies the service information acquisition unit 41 of it. It is a diagram.

In this case, the sidecar proxy 25 executes the internal command "/bin/internal_cmd" inside the container 24 to collect each item included in the service information 27, and collects these items.

FIG. 10 is a schematic diagram of a method of acquiring the item "Using_service" included in the service information 27. As shown in FIG.

Here, it is assumed that the service 23 of "ServiceA" depends on each of the services 23 of "ServiceB" and "ServiceC". The "ServiceA" container 24 is an example of a first container, and the "ServiceB" container 24 is an example of a second container. In this case, the sidecar proxy 25 in the service 23 of "Service A" monitors data such as communication packets transmitted from the container 24 of its own service 23. FIG. Then, the sidecar proxy 25 identifies the service 23 to which the source container 24 belongs and the service 23 to which the destination container 24 belongs by analyzing the header of the communication packet. After that, the sidecar proxy 25 creates a communication table 23b in which the source and the destination are associated, and notifies the service information acquisition unit 41 of it. Then, the service information acquisition unit 41 notifies the control unit 42 of the communication table 23b.

Based on the communication table 23b, the control unit 42 identifies the container 24 included in the source "ServiceA" and the container 24 included in the destination "ServiceB". Then, the control unit 42 determines that the container 24 included in “ServiceA” depends on the container 24 included in “ServiceB”, and sets “ServiceB” as the “Using_service” item of the service information 27 .

Similarly, the control unit 42 also determines from the communication table 23b that the container 24 included in "ServiceA" depends on the container 24 included in "ServiceC". Therefore, the control unit 42 further sets “ServiceC” as the “Using_service” item of the service information 27 .

Refer to FIG. 7 again. The abnormality handling device 22 is a device that, when an abnormality occurs in each container 24 of the business system 21, takes measures to eliminate the abnormality. As an example, the error handling device 22 includes an analysis unit 31 , an error handling unit 32 , a storage unit 33 , an operation policy generation unit 34 , an operation policy application unit 35 and a reception unit 36 .

Among them, the analysis unit 31 is a processing unit that analyzes the service information 27 described above, and has a service information acquisition unit 41 and a control unit 42 .

The service information acquisition unit 41 is a processing unit that acquires the service information 27 from each sidecar proxy 25 .

The control unit 42 determines whether there is an abnormality in the container 24 by analyzing the service information 27 collected by the service information acquisition unit 41 .

For example, when the “Status” of the service information 27 is “Error”, the control unit 42 determines that an abnormality has occurred in the container 24 included in the service 23 related to the service information 27 .

Also, the control unit 42 uses “Using_db” and “Using_service” of the service information 27 to generate a service topology that indicates the dependency relationship between the multiple services 23 . Further, when the "Using_db" and "Using_service" of the service information 27 are different from those at the time of previous acquisition, the control unit 42 determines that the service topology indicating the interdependency between the services 23 has been changed.

When the control unit 42 determines that an abnormality has occurred in the container 24 as described above, the control unit 42 generates the abnormality information resource 44 based on the service information 27 and stores it in the storage unit 33 . The anomaly information resource 44 is a file indicating information about an anomaly that has occurred in the service 23, and is generated by the control unit 42 for each anomaly that has occurred.

FIG. 11 is a schematic diagram of the anomaly information resource 44. As shown in FIG. As shown in FIG. 11, the abnormality information resource 44 has items of "Kind", "Id", "Status", "Priority", "Service", and "Retry_count".

Among these, "Kind" is information indicating the type of abnormality. The setting method of "Kind" is not particularly limited. For example, when the "Status" of the service information 27 is "Error", the control unit 42 acquires the standard error output from the path indicated by "Stdout_path" of the service information 27, Get the log file from Then, the control unit 42 sets the value of "Kind" indicating the type of abnormality based on the acquired standard error output and log file.

“Id” is an identifier that uniquely identifies the anomaly information resource 44 . “Status” is information indicating the result of coping with the abnormality. The information includes "Sucess", which indicates that the error has been resolved, "Failed", which indicates that the error has not been resolved, and "ResolvingError", which indicates that the error is being resolved. There is

"Priority" is a numerical value indicating the priority of which abnormality should be dealt with first among a plurality of abnormalities, and the smaller the value, the higher the priority. The method of setting the value of "Priority" is not particularly limited. You can set the value of

"Service" is the name of the service 23 in which the error occurred. "Retry_count" indicates the number of times the error has been dealt with. The initial value of "Retry_count" is "0", and the control unit 42 increments "Retry_count" by 1 each time the logic execution unit 49 executes the logic for resolving the abnormality.

Refer to FIG. 7 again. After generating the abnormality information resource 44, the control unit 42 requests the abnormality handling unit 32 to handle the abnormality.

The abnormality handling unit 32 is a processing unit that, when receiving a request from the control unit 42 , takes measures to resolve an abnormality that has occurred in the service 23 . As an example, the abnormality handling unit 32 has an abnormality identification unit 46 , a scheduling unit 47 , a logic generation unit 48 and a logic execution unit 49 .

Among these, the abnormality identification unit 46 is a processing unit that reads a plurality of abnormality information resources 44 in the storage unit 33 when receiving a request from the control unit 42 and identifies the type of abnormality corresponding to these abnormality information resources 44. is. For example, the anomaly identification unit 46 identifies the kind of anomaly from “Kind” of the anomaly information resource 44 . Further, after identifying the type of abnormality, the abnormality identification unit 46 requests the scheduling unit 47 to perform scheduling for dealing with the abnormality.

The scheduling unit 47 is a processing unit that, upon receiving a request from the abnormality identification unit 46, schedules measures to be taken against an abnormality. As an example, the scheduling unit 47 specifies the priority of anomalies from "Priority" of the anomaly information resource 44, and performs scheduling so that the anomalies with the highest priority are processed first.

The logic generation unit 48 is a processing unit that generates, for each type of abnormality identified by the abnormality identification unit 46, logic for resolving the abnormality. For example, the logic generation unit 48 generates logic by referring to the logic database 51 stored in the storage unit 33 .

FIG. 12 is a schematic diagram of the logic database 51. As shown in FIG. As shown in FIG. 12, the logic database 51 is information in which "type of abnormality", "name of abnormality", and "logic" are associated with each other.

"Kind of anomaly" is the same information as "Kind" of the anomaly information resource 44, and is information indicating the kind of anomaly. "Abnormality name" is information indicating the name of the abnormality. "Logic" is information indicating the processing content for resolving the abnormality.

For example, consider the case where the 'failure type' is 'network timeout between services'. The "name of anomaly" in this case is "NW_timeout". The “logic” is “[COMMAND: “<command to be executed>”]”, and by executing this “<command to be executed>”, the abnormality of “network timeout between services” is resolved.

Then, the logic generation unit 48 generates logic “[COMMAND: “<command to be executed>”]”.

It should be noted that a script for resolving an error as logic, such as "Error connecting to DB", may be used.

Refer to FIG. 7 again. The logic execution unit 49 is a processing unit that tries to eliminate the abnormality by executing the logic generated by the logic generation unit 48 . It should be noted that, when the "Retry_count" of the anomaly information resource 44 of a certain anomaly exceeds a predetermined threshold value, the logic generation unit 48 stops executing the logic for that anomaly. In this case, the logic generator 48 executes the logic for the abnormality with the highest priority among the remaining abnormalities.

The operation policy generation unit 34 is a processing unit that generates an operation policy for the business system 21 based on the SLA that the control unit 42 has acquired from the service information 27 .

FIG. 13 is a schematic diagram of an operational policy. The operational policy is a parameter for controlling the resource usage rate of the container 24 and a setting parameter for the sidecar proxy 25 for controlling communication between the services 23 . Such setting parameters include the parameters “Name”, “Id”, “Priority”, “Stdout_path”, “Logfile_path”, “Internalcmd_path”, and “Operation_type” in the service information 27 . Initial values of these values are set by the operator, but are automatically adjusted by the operation policy generation unit 34 when the operation of the business system 21 is started. For example, the operational policy generator 34 updates these setting parameters so that the business system 21 satisfies the SLA. The operational policy generation unit 34 also stores the updated operational policy in the operational policy database 52 .

Refer to FIG. 7 again. The operational policy application unit 35 is a processing unit that applies the operational policy to each container 24 by referring to the operational policy database 52 .

The accepting unit 36 is a processing unit that accepts input of individual parameters contained in the logic database 51 from the operator and stores them in the storage unit 33 . The accepting unit 36 also accepts input of an initial value of the operational policy from the operator, and stores it in the operational policy database 52 of the storage unit 33 .

Next, the flow of processing when starting the operation of the business system 12 will be described.

FIG. 14 is a flow chart showing the flow of processing when starting the operation of the business system 12 .

First, the reception unit 36 receives input of initial values of individual parameters of the operation policy (see FIG. 13) from the operator, and stores them in the operation policy database 52 (step S11).

Next, the operational policy generation unit 34 refers to the operational policy database 52 and generates an operational policy (step S12).

Next, the operational policy application unit 35 refers to the operational policy database 52 and acquires the operational policy (step S13).

Subsequently, the operational policy application unit 35 applies the acquired operational policy to each container 24 (step S14).

Next, the reception unit 36 receives inputs of individual parameters of the logic database 51 from the operator and stores them in the storage unit 33 (step S15).

Next, the logic generation unit 48 creates the logic database 51 from these parameters and stores it in the storage unit 33 (step S16).

With the above, the basic processing for starting the operation of the business system 12 is completed.

Next, an abnormality coping method according to this embodiment will be described.

FIG. 15 is a flowchart of an abnormality coping method according to this embodiment. First, the operation policy application unit 35 refers to the operation policy database 52, and if there is a change in the operation policy, applies the changed operation policy to each container 24 (step S21).

Next, the sidecar proxy 25 collects the information of each item included in the service information 27 from the container 24 in the same service 23 as the sidecar proxy 25, and notifies the service information acquisition unit 41 of them (step S22).

Next, the service information acquisition unit 41 acquires the service information 27 from each sidecar proxy 25 (step S23).

Next, the control unit 42 analyzes the service information 27 (step S24). For example, based on the communication table 23b, the control unit 42 generates a service topology that indicates dependencies between the services 23. FIG. Also, the control unit 42 identifies the SLA of the business system 12 based on the service information 27 .

Next, as a result of analyzing the service information 27, the control unit 42 determines whether the service topology has been changed (step S25).

Then, if it is determined that the service topology has been changed (step S25: affirmative), the process proceeds to step S26. In step S<b>26 , the operational policy generator 34 generates an operational policy desirable for the service topology after change and stores it in the operational policy database 52 .

On the other hand, if it is determined that the service topology has not been changed (step S25: No), the process proceeds to step S27.

In step S27, the control unit 42 analyzes the service information 27 and determines whether the SLA of the business system 12 exceeds the standard. Here, if it is determined that the SLA exceeds the standard (step S27: affirmative), the process proceeds to step S26. In step S26, the operational policy generator 34 changes the operational policy so that the SLA of the business system 12 satisfies the criteria.

On the other hand, if it is determined that the SLA does not exceed the standard (step S27: No), the process proceeds to step S28.

In step S28, the control unit 42 determines whether the container 24 has an abnormality. For example, when the “Status” of the service information 27 is “Error”, the control unit 42 determines that an abnormality has occurred in the container 24 included in the service 23 indicated by the “Name” of the service information 27 .

Here, if it is determined that no abnormality has occurred (step S28: No), the process proceeds to step S29.

In step S29, the control unit 42 determines whether the business system 12 has completed the business. Here, if it is determined that the work has not been completed (step S29: No), the process returns to step S22. On the other hand, if it is determined that the business has ended (step S29: affirmative), the process ends.

Further, when the control unit 42 determines that an abnormality has occurred in the above-described step S28, the abnormality handling process of step S30 is performed, and then the process proceeds to step S29. With the above, the basic processing of the flowchart of FIG. 15 is completed.

FIG. 16 is a flowchart of the abnormality handling process in step S30 described above.

First, the control unit 42 generates the abnormality information resource 44 based on the service information 27 and stores it in the storage unit 33 (step S41). At this time, the control unit 42 identifies the type of abnormality based on the service information 27 and sets the value of “Kind” according to the type of abnormality in the abnormality information resource 44 . Further, the control unit 42 requests the abnormality handling unit 32 to handle the abnormality.

Next, the abnormality identification unit 46 of the abnormality handling unit 32 identifies the type of abnormality upon receiving a request from the control unit 42 (step S42). For example, the anomaly identification unit 46 reads the anomaly information resource 44 in the storage unit 33 and identifies the type of anomaly from the “Kind” of the anomaly information resource 44 . Further, the anomaly identifying unit 46 requests the scheduling unit 47 to schedule measures to deal with the anomaly.

Next, the scheduling unit 47 receives a request from the abnormality identification unit 46 and schedules measures to deal with the abnormality (step S43). For example, the scheduling unit 47 specifies the priority of anomalies from "Priority" of the anomaly information resource 44, and performs scheduling so that the anomalies with the highest priority are processed first.

Next, the logic generation unit 48 generates logic for resolving the abnormality for each type of abnormality identified by the abnormality identification unit 46 (step S44). As an example, the logic generation unit 48 identifies logic corresponding to the type of abnormality identified in step S42 by referring to the logic database 51 stored in the storage unit 33, and generates the logic.

Next, the logic execution unit 49 executes the logic generated by the logic generation unit 48 (step S45).

Next, the control unit 42 newly acquires the service information 27 and generates the abnormality information resource 44 based on the service information 27 (step S46).

Next, the logic execution unit 49 determines whether or not the abnormality has been resolved by executing the logic (step S47). For example, the logic execution unit 49 determines that the abnormality has been resolved when the "Status" of the abnormality information resource 44 generated in step S46 is "Running", and determines that the abnormality has been resolved when the "Status" remains "Error". Determine that it has not been resolved.

Here, if it is determined that the abnormality has been resolved (step S47: affirmative), the process proceeds to step S50.

In step S<b>50 , the control unit 42 deletes the abnormality information resource 44 corresponding to the resolved abnormality from the storage unit 33 .

On the other hand, if it is determined that the abnormality has not been resolved (step S47: No), the process proceeds to step S48.

In step S48, the logic execution unit 49 determines whether the "Retry_count" of the abnormality information resource 44 indicating the number of retries for dealing with an abnormality exceeds a threshold. The threshold is set from the viewpoint of preventing delays in dealing with other anomalies due to repeated dealing with a certain anomaly.

Here, when it is determined that the threshold value is exceeded (step S48: affirmative), the control unit 42 proceeds to the above-described step S50, and deletes the abnormality information resource 44 related to the abnormality dealt with in step S45 from the storage unit 33. As a result, it is possible to prevent delays in coping with other anomalies caused by repeating an anomaly with respect to a specific anomaly.

In this case, the abnormality has not been resolved, but the control unit 42 may display an alert on the display device or the like to notify the operator that the abnormality has not been resolved.

On the other hand, if it is determined that the threshold is not exceeded (step S48: No), the process proceeds to step S49.

In step S49, the control unit 42 determines whether there is an abnormality that has not been dealt with. Here, if it is determined that there is an abnormality that has not been dealt with (step S49: affirmative), the process returns to step S43.

On the other hand, if it is determined that there is no abnormality that has not been dealt with (step S49: affirmative), the processing is terminated and the caller is returned to.

With the above, the basic processing of the abnormality coping method according to the present embodiment is completed.

According to the present embodiment described above, the control unit 42 sets the "Priority" of the abnormality information resource 44 that indicates the priority of handling an abnormality (step S41), and the abnormality is handled in order of priority (step S43). , S45). Therefore, it is possible to prevent the handling of a serious abnormality from being postponed, and to suppress the delay in handling the abnormality.

Furthermore, since the logic generation unit 48 generates logic for each type of abnormality (step S44), it is possible to automatically execute logic suitable for resolving the abnormality, and the operator of the business system 12 needs to determine the content of the countermeasure. There is no

Moreover, when it is determined in step S48 that the number of retries for a certain abnormality has exceeded the threshold, the logic execution unit 49 stops dealing with that abnormality. Therefore, it is possible to prevent delays in dealing with other anomalies caused by dealing with the same anomaly many times.

Next, a specific example of handling an abnormality will be described.

- 1st example FIG. 17 : is a schematic diagram for demonstrating the abnormality handling based on a 1st example.

As shown in FIG. 17, in the first example, it is assumed that the business system 12 is realized by four services 23 of "ServiceA" to "ServiceD". It is also assumed that, of these services 23, two containers 24, "ServiceA" and "ServiceD", have failed.

FIG. 18(a) is a schematic diagram of the service information 27 in this case. As shown in FIG. 18A, the "Status" of "ServiceB" and "ServiceC" in which no abnormality has occurred in the container 24 is "Running". On the other hand, the "Status" of "ServiceA" and "ServiceD" in which an abnormality has occurred in the container 24 becomes "Error".

FIG. 18B is a schematic diagram of the abnormality information resource 44 generated by the control unit 42 in the first example. Since the abnormality information resource 44 is generated by the control unit 42 for each service 23 in which an abnormality has occurred in the container 24, the control unit 42 generates the abnormality information resource 44 for "ServiceA" and "ServiceD" in this example.

Here, the control unit 42 sets the "Priority" of the abnormality information resource 44 related to "ServiceD" to "0", and sets the "Priority" of the abnormality information resource 44 related to "ServiceA" to a higher "1". shall be set to

FIG. 19 is a schematic diagram of the logic generated by the logic generator 48 in this case.

In the logic, a script for resolving the abnormality of the container 24 of "ServiceA" and a script for resolving the abnormality of the container 24 of "ServiceD" are described. As mentioned above, the priority of abnormality of "ServiceA" is higher than that of "ServiceD". Therefore, the scheduling unit 47 performs scheduling so that "ServiceA" is handled before "ServiceD" is handled. As a result of this scheduling, the logic generation unit 48 writes the script for resolving the abnormality of the container 24 of "ServiceA" before the script for resolving the abnormality of the container 24 of "ServiceD".

As a result, the logic execution unit 49 first deals with the abnormality of the container 24 of "ServiceA", which has the highest priority, and then deals with the abnormality of the container 24 of "ServiceD".

As a result, it is possible to suppress delays in dealing with high-priority and serious abnormalities, and the business system 12 can be operated stably. Moreover, since the control unit 42 automatically sets the priority of abnormalities, and the scheduling unit 47 automatically schedules according to the priority, the operator of the business system 12 does not need to decide the order of responding to abnormalities. .

Second Example FIG. 20 is a schematic diagram for explaining how to deal with an abnormality according to a second example. As shown in FIG. 20, in the second example, a business system 21 is implemented by four services 23 of "ServiceA1", "ServiceA2", "ServiceA3", and "ServiceB" and a database 29 of "DatabaseA". Assume that there are In FIG. 20, arrows indicate dependencies between the services 23 . The service 23 at the root of the arrow indicates the service activated by the container 24 that is the transmission source of the communication packet. Also, the service 23 at the tip of the arrow indicates the container 24 of the transmission destination of the communication packet.

An arrow directed to the database 29 indicates that the container 24 in the service 23 at the root of the arrow accesses the database 29 .

Below, a case where there is an abnormality in each of the containers 24 of "ServiceA1" and "ServiceA2" will be described.

FIG. 21 is a schematic diagram showing service information 27 in this case. Since the containers 24 of "ServiceA1" and "ServiceA2" each have an abnormality as described above, the "Status" of these services 23 is "Error".

Also, the items "Using_db" and "Using_service" store values that reflect the dependencies shown in FIG. For example, "ServiceA2" and "ServiceA3" are stored in "Using_service" of "ServiceA1". Since "ServiceA1" does not access "DatabaseA", "NULL" indicating no access is stored in "Using_db".

On the other hand, since "ServiceA2" accesses "DatabaseA", "DatabaseA" is stored in "Using_db" of "ServiceA2". Also, since "ServiceA2" does not depend on other services 23, "Using_service" of "ServiceA2" is "NULL".

"CircuitBreakOK" in "Operation_type" of "ServiceA3" indicates that "ServiceA3" may be deleted from the business system 21 when an abnormality occurs in the container 24 executing the service 23 of "ServiceA3". show.

FIG. 22 is a schematic diagram of a service topology generated by the control unit 42 using the service information 27 of FIG.

Here, the control unit 42 generates a neighbor list representing the service topology. "ServiceA1-ServiceA2-ServiceA3" on the first line of this adjacency list indicates that the services 23 to which "ServiceA1" communicates are "ServiceA2" and "ServiceA3". "ServiceA2-DatabaseA" on the second line indicates that "ServiceA2" accesses "DatabaseA". Also, "DatabaseA" in the last line indicates that there is no access destination for "DatabaseA".

FIG. 23 is a schematic diagram of the abnormality information resource 44 generated by the control unit 42 in this example.

The example of FIG. 23 shows a case where the logic execution unit 49 has already executed the logic once for each abnormality of "ServiceA1" and "ServiceA2", but the abnormality has not been resolved. In this case, the "Status" of each of "ServiceA1" and "ServiceA2" is "Failed".

Also, it is assumed that the "Priority" of "ServiceA1" and "ServiceA2" are both "1".

When the two services 23 have the same "Priority" as described above, the scheduling unit 47 schedules the container 24 included in the service 23 to which the communication packet is sent to execute the logic first. In this example, the destination of the communication packet is "ServiceA2", so the scheduling unit 47 schedules the logic of "ServiceA2" to be executed before "ServiceA1".

In response to the scheduling, the logic execution unit 49 executes the logic of "ServiceA2" before "ServiceA1" and attempts to resolve the abnormality of the container 24 of "ServiceA2".

Conversely, it is also possible to deal with "ServiceA1" first. ServiceA1" may immediately malfunction.

On the other hand, in this example, the transmission destination "ServiceA2" is dealt with first, and then "ServiceA1" is dealt with. Therefore, there is little possibility that an error will occur again in "ServiceA2" while "ServiceA1" is being dealt with after the error in "ServiceA2" has been resolved, and wasteful handling of the error can be suppressed.

(Hardware configuration)
Next, the hardware configuration of the abnormality handling device 22 according to this embodiment will be described.

FIG. 24 is a hardware configuration diagram of the abnormality handling device 22 according to this embodiment.

As shown in FIG. 24, the abnormality handling device 22 has a storage device 22a, a memory 22b, a processor 22c, a communication interface 22d, an input device 22f, a display device 22g, and a medium reading device 22h. These units are interconnected by a bus 22j.

Among these, the storage device 22a is a non-volatile storage such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores the anomaly handling program 100 according to the present embodiment.

It should be noted that the abnormality handling program 100 may be recorded in a computer-readable recording medium 22i, and the abnormality handling program 100 may be read by the processor 22c via the medium reading device 22h.

Examples of such a recording medium 22i include physical portable recording media such as CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), and USB (Universal Serial Bus) memory. A semiconductor memory such as a flash memory or a hard disk drive may also be used as the recording medium 22i. These recording media 22i are not temporary media like carrier waves that have no physical form.

Furthermore, the abnormality handling program 100 may be stored in a device connected to a public line, the Internet, a LAN, or the like. In that case, the processor 22c may read and execute the abnormality handling program 100. FIG.

On the other hand, the memory 22b is hardware such as a DRAM (Dynamic Random Access Memory) that temporarily stores data.

The processor 22c is hardware such as a CPU or a GPU (Graphical Processing Unit) that controls each part of the anomaly handling device 22 . The processor 22c also executes the anomaly handling program 100 in cooperation with the memory 22b.

As a result, the analysis unit 31, the abnormality handling unit 32, the operation policy generation unit 34, the operation policy application unit 35, and the reception unit 36 of the abnormality handling device 22 shown in FIG. 7 are realized.

Also, the storage unit 33 (see FIG. 7) is realized by the storage device 22a and the memory 22b.

Furthermore, the communication interface 22d is hardware such as a NIC (Network Interface Card) for connecting the abnormality handling device 22 to a network such as the Internet or a LAN (Local Area Network). Via this communication interface 22d, each of the container 24 and the container base 26 can communicate with the abnormality handling device 22. FIG.

The input device 22f is hardware such as a keyboard and a mouse for the operator to input each parameter and the initial value of the operation policy included in the logic database 51 .

The display device 22g is a display device such as a liquid crystal display for displaying various data input to the error handling device 22 by the operator via the input device 22f.

The medium reading device 22h is hardware such as a CD drive, a DVD drive, and a USB interface for reading the recording medium 22i.

1 Business system 2 Container base 2a Health check mechanism 4 Service 5 Container monitoring software 6 Operation terminal 12 Business system 20 Abnormality handling system 21 Business system 22 Abnormality Response device 23 Service 23a Storage area 23b Communication table 24 Container 25 Sidecar proxy 26 Container base 27 Service information 29 Database 31 Analysis unit 32 Abnormality handling unit , 33 storage unit 34 operation policy generation unit 35 operation policy application unit 36 reception unit 41 service information acquisition unit 42 control unit 44 abnormality information resource 46 abnormality identification unit 47 Scheduling unit 48 Logic generation unit 49 Logic execution unit 51 Logic database 52 Operation policy database 100 Abnormality handling program.

Claims (5)

Set the priority of abnormalities that occurred in each of multiple containers,
generating logic for resolving the anomaly for each type of anomaly;
executing the logic for the anomalies in order of the priority;
An anomaly handling program that causes a computer to execute processing. identifying a first container that is a source of data and a second container that is a destination of data from among the plurality of containers;
if the priority of each of the anomaly of the first container and the anomaly of the second container are the same, then executing the logic for the anomaly of the second container first;
2. The abnormality handling program according to claim 1, further causing said computer to execute processing. stopping execution of the logic for the same anomaly when the number of executions of the logic for the same anomaly exceeds a threshold;
2. The abnormality handling program according to claim 1, further causing said computer to execute processing. a control unit that performs control to set the priority of anomalies that have occurred in each of a plurality of containers;
a generation unit that generates logic for resolving the anomaly for each type of anomaly;
an execution unit that executes the logic for the abnormality in order of the priority;
An anomaly coping system characterized by having: the computer
Set the priority of abnormalities that occurred in each of multiple containers,
generating logic for resolving the anomaly for each type of anomaly;
executing the logic for the anomalies in order of the priority;
An abnormality coping method characterized by executing processing.

Images