JP7311335B2 - DISTRIBUTED CONTAINER MONITORING SYSTEM AND DISTRIBUTED CONTAINER MONITORING METHOD - Google Patents

Description

The present invention relates to a distributed container monitoring system and a distributed container monitoring method.

Conventionally, in a container environment, each server resource maintains the management status etc. as a blockchain, so that when a failure occurs in a server resource without using a central management system, the container that was running on the server resource concerned A container environment is restored by determining the relocation destination of the container and relocating the container (see, for example, Patent Literature 1).

JP 2018-156465 A

In the technique described in Patent Document 1, an agent monitors server resources to determine whether or not a failure has occurred in the server resources. In this case, fault monitoring may not be performed properly.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to perform failure monitoring more appropriately.

The above and other objects and novel features of the present invention will become apparent from the description of the specification and the accompanying drawings.

A brief outline of typical inventions disclosed in the present application is as follows.

In a distributed container monitoring system for monitoring the operating status of a container to be monitored according to a representative embodiment of the present invention, a first monitoring container for monitoring the container to be monitored and a first monitoring container for monitoring the first monitoring container are provided. 2 monitoring containers, wherein the first monitoring container includes a first signal transmission unit for transmitting a signal indicating status confirmation to the container to be monitored, and monitoring the signal transmitted by the first signal transmission unit. a first detection unit that detects a failure of the container to be monitored based on the response status of the container to be monitored, and the second monitoring container sends a signal indicating status confirmation to the first monitoring container. a second signal transmission unit for transmitting; and a second detection unit for detecting a failure of the first monitoring container based on the response status of the first monitoring container to the signal transmitted by the second signal transmission unit. .

Among the inventions disclosed in the present application, the effects obtained by representative ones are briefly described below.

That is, according to the representative embodiment of the present invention, fault monitoring can be performed more appropriately.

1 is a diagram showing an overview of the configuration of a distributed container monitoring system according to this embodiment; FIG. 3 is a functional block diagram of a monitoring container according to this embodiment; FIG. FIG. 10 is a sequence diagram showing the flow of processing when the monitoring container monitors the business container and the monitoring container and detects a failure in the business container according to the present embodiment; FIG. 10 is a sequence diagram showing the flow of processing when a monitoring container monitors a business container and a monitoring container and detects a failure in the monitoring container according to the present embodiment; It is an example of a figure explaining the example of a screen of a monitoring situation.

Hereinafter, this embodiment will be described in detail based on the drawings. In principle, the same parts are denoted by the same reference numerals throughout the drawings for describing the embodiments, and repeated descriptions thereof will be omitted. On the other hand, parts that have been described with reference numerals in one drawing may be referred to with the same reference numerals, although they are not shown again in the description of other drawings.

<Overview>
FIG. 1 is a diagram showing an overview of the configuration of a distributed container monitoring system 1 according to this embodiment. As shown in FIG. 1, the distributed container monitoring system 1 has servers 10 (servers 10 a to 10 c ) and a control server 20 .

The distributed container monitoring system 1 is a system that monitors the operation status of monitored containers (containers that execute business applications, etc.). Here, a container is realized by virtualization technology.

The servers 10a to 10c and the control server 20 are so-called server devices, and can exchange information and signals with each other via the network 30. FIG.

The server 10 and the control server 20 run an OS (Operating System), a DBMS (DataBase Management System), and a Web server program developed on a memory from a recording device such as an HDD (Hard Disk Drive) by a CPU (Central Processing Unit) (not shown). and other middleware and software running on it. This implements various functions to be described later.

The control server 20 manages containers and instructs the server 10 to create containers. The control server 20 acquires from the server 10 information on the operation status of the container being executed by the server 10, and outputs the acquired information.

The control server 20 implements various functions by executing existing distributed container operation management software such as Kubernetes.

The server 10 creates containers based on instructions from the control server 20 . Based on instructions from the control server 20, the server 10 selects a business container 11 (container to be monitored), which is a container for executing business applications, and a monitoring container 12, which is a container capable of monitoring the business container. Generate.

Also, the monitoring container 12 in the server 10 monitors the business container 11 . Also, the monitoring container 12 other than the monitoring container 12 monitoring the business container 11 monitors the monitoring container 12 monitoring the business container 11 .

In the distributed container monitoring system 1 shown in FIG. 1, the server 10a (identifier is server S1) has a monitoring container 12a (identifier is monitoring container M1). The server 10b (identifier is server S2) has a business container 11 and a monitoring container 12b (identifier is monitoring container M2). The server 10c (identifier is server S3) has a monitoring container 12c (identifier is monitoring container M3).

As described above, in the distributed container monitoring system 1, the monitoring containers 12 are distributed to each server resource (server 10a to server 10c).

Next, functions of the monitoring container 12 will be described with reference to FIG. FIG. 2 is a functional block diagram of the monitoring container 12. As shown in FIG.

As shown in FIG. 2, the monitoring container 12 includes a role determination unit 121, a first signal transmission unit 122, a first detection unit 123, a second signal transmission unit 124, a second detection unit 125, a detection result output unit 126, and a It has a recovery unit 127 .

The role determining unit 121 determines whether each of the monitoring containers 12 functions as a monitoring container 12 (first monitoring container) that monitors the business container 11 or functions as a container (second monitoring container) that monitors the monitoring container 12 that monitors the business container 11 . monitoring container).

For example, upon receiving a leader candidacy inquiry from the control server 20 , the role determining unit 121 sends a signal indicating the leader candidacy to the control server 20 or other monitoring container 12 . Here, a leader means a container that monitors the business container 11 .

Also, the role determination unit 121 receives a signal indicating a leader candidacy from another monitoring container 12 . The role determining unit 121 determines the monitoring container 12 that has transmitted the signal indicating the leader candidate earliest to be the monitoring container 12 that monitors the business container 11 . Also, another monitoring container 12 is determined to be the monitoring container 12 that monitors the business container 11 .

Further, when the self-monitoring container 12 is the monitoring container 12 that monitors the business container 11 , the role determination unit 121 periodically transmits a signal indicating the leader continuation assertion to the other monitoring containers 12 .

Further, when it is detected that a failure has occurred in the monitoring container 12 that monitors the business container 11, the role determining unit 121 transmits a signal indicating a leader candidate to the control server 20 and other monitoring containers 12. . In this manner, the role determination unit 121 re-determines the role when a failure occurs in the monitoring container 12 serving as the leader. In this way, the role determining unit 121, triggered by the occurrence of a failure in the monitoring container 12 that is the leader, sets the monitoring container 12 other than the failed monitoring container 12 as the leader.

The first signal transmission unit 122 is a part that transmits a signal indicating status confirmation to the business container 11, which is a container to be monitored.

When the role determining unit 121 determines that the own container functions as the monitoring container 12 that monitors the business container 11, the first signal transmitting unit 122 confirms the survival of the business container 11 at a predetermined timing. a signal (health check signal) to inquire about the

After transmitting the survival confirmation signal, first signal transmitting section 122 notifies first detecting section 123 of the transmission.

The first detection unit 123 is a part that detects a failure of the business container 11 based on the response status of the business container 11 to the signal transmitted by the first signal transmission unit 122 .

When the first detection unit 123 acquires the notification that the life confirmation signal has been transmitted by the first signal transmission unit 122 , the first detection unit 123 receives a response signal to the life confirmation signal from the business container 11 .

The first detection unit 123 detects a failure of the business container 11 based on the content of the response signal or the reception state of the response signal.

For example, if the response signal is not transmitted within a predetermined period after the survival confirmation signal is transmitted from the first signal transmission unit 122, the first detection unit 123 detects the status of the business container 11 based on this result. to detect failures in

Note that the first detection unit 123 may detect a failure of the business container 11 when the timing of receiving the response signal is gradually delayed. This is because the business container 11 has a high probability of failure.

When receiving the response signal, the first detection unit 123 outputs the response signal to the detection result output unit 126 . Further, when the first detection unit 123 detects a failure of the business container 11 , it sends the detection of the failure to the detection result output unit 126 .

Note that the first detection unit 123 may send the response signal itself or information about the response signal (information about the time until the response signal is received) to the detection result output unit 126 .

The second signal transmission unit 124 is a part that transmits a signal indicating status confirmation to the monitoring container 12 that monitors the business container 11 .

When the role determination unit 121 determines that the own container functions as the monitoring container 12 that monitors the monitoring container 12 that monitors the business container 11, the second signal transmission unit 124 determines the monitoring target at a predetermined timing. A survival confirmation signal is transmitted to the monitoring container 12 .

After transmitting the survival confirmation signal, second signal transmission section 124 notifies second detection section 125 of the transmission.

The second detection unit 125 is a part that detects a failure of the monitoring container 12 to be monitored based on the response status of the monitoring container 12 to be monitored to the signal transmitted by the second signal transmission unit 124 .

When the second detection unit 125 acquires the notification that the survival confirmation signal has been transmitted by the second signal transmission unit 124, the second detection unit 125 receives a response signal to the survival confirmation signal from the monitoring container 12 to be monitored. When the monitoring container 12 to be monitored is operating normally and receives the life confirmation signal, it transmits a response signal to the sender of the life confirmation signal.

The second detection unit 125 detects a failure of the monitoring container 12 to be monitored based on the content of the response signal or the reception state of the response signal.

For example, when the response signal is not transmitted within a predetermined period after the survival confirmation signal is transmitted from the second signal transmission unit 124, the second detection unit 125 determines the monitoring target based on this result. A failure of the monitoring container 12 is detected.

Note that the second detection unit 125 may detect a failure of the monitoring container 12 to be monitored when the timing of receiving the response signal is gradually delayed. This is because the probability of failure of the monitoring container 12 to be monitored is high.

When receiving the response signal, the second detection unit 125 sends the response signal to the detection result output unit 126 . Further, when the second detection unit 125 detects a failure of the monitoring container 12 to be monitored, the second detection unit 125 sends a notification to the effect that the failure has been detected to the detection result output unit 126 . Note that the second detection unit 125 may send the response signal itself or information about the response signal (information about the time until the response signal is received) to the detection result output unit 126 .

The detection result output unit 126 is a part that outputs the detection result by the first detection unit 123 or the second detection unit 125 .

For example, the detection result output unit 126 transmits the detection result by the first detection unit 123 or the second detection unit 125 to the control server 20 .

Note that the detection result output unit 126 may acquire information about the response signal itself or the response signal acquired from the first detection unit 123 or the second detection unit 125 and transmit this information to the control server 20. .

The recovery unit 127 is a part that performs recovery processing of the container environment when the first detection unit 123 detects a failure in the business container 11, which is a container to be monitored.

Upon receiving notification from the first detection unit 123 that a failure of the business container 11 has been detected, the recovery unit 127 requests the server 10 other than the business container 11 in which the failure occurred to create a container, and restores the business. Make a request to create the container 11 . In this case, the server 10 to which the request is made creates the business container 11 and operates the business container 11 . In this manner, the recovery unit 127 performs recovery processing of the container environment when a failure of the business container 11 is detected.

<Processing procedure>
Next, with reference to FIG. 3, processing when the monitoring container 12 in this embodiment monitors the business container 11 and the monitoring container 12 and detects a failure in the business container 11 will be described.

FIG. 3 is a sequence diagram showing the flow of processing when the monitoring container 12 monitors the business container 11 and the monitoring container 12 and detects a failure in the business container 11 in this embodiment.

First, it is assumed that the business container 11 is running in the server 10b (step S1). Upon receiving a leader candidate inquiry from the control server 20 , the role determining unit 121 transmits a signal indicating a leader candidate to the other monitoring containers 12 . For example, the role determining unit 121 of the monitoring container 12a transmits a signal indicating leader candidacy to the monitoring containers 12b and 12c (steps S2 and S3).

The role determining unit 121 of the monitoring container 12a determines the monitoring container 12a to be the monitoring container that monitors the business container 11 when the monitoring container 12a transmits the signal indicating the leader candidate earliest. Also, the role determining unit 121 of the monitoring container 12a determines the monitoring container 12b and the monitoring container 12c as monitoring containers that monitor the monitoring container 12a.

The first signal transmission unit 122 of the monitoring container 12a transmits a signal indicating that the business container 11 is alive. It is determined that there is no (step S4).

In addition, the role determination unit 121 of the monitoring container 12a transmits a signal indicating the leader continuation assertion to the monitoring container 12b and the monitoring container 12c every predetermined period (Steps S5 and S6).

Further, the second signal transmission unit 124 of the monitoring container 12b transmits a signal indicating confirmation of survival to the monitoring container 12a, and the second detection unit 125 of the monitoring container 12b receives a response signal from the monitoring container 12a to confirm that the monitoring container 12b is alive. Confirm (step S7).

Further, the second signal transmission unit 124 of the monitoring container 12c transmits a signal indicating survival confirmation to the monitoring container 12a, and the second detection unit 125 of the monitoring container 12c receives a response signal from the monitoring container 12a to confirm that the monitoring container 12c is alive. Confirm (step S8).

Also, the first signal transmission unit 122 of the monitoring container 12a transmits a signal indicating that the business container 11 is alive (step S9). A failure occurs in the server 10b between steps S4 to S9.

In addition, the role determining unit 121 of the monitoring container 12a transmits a signal indicating the leader continuation assertion to the monitoring container 12b and the monitoring container 12c every predetermined period (steps S10 and S11).

Further, the second signal transmission unit 124 of the monitoring container 12c transmits a signal indicating survival confirmation to the monitoring container 12a, and the second detection unit 125 of the monitoring container 12c receives a response signal from the monitoring container 12a to confirm that the monitoring container 12c is alive. Confirm (step S12).

The first detection unit 123 of the monitoring container 12a detects that a failure has occurred in the business container 11 because there is no response signal for a predetermined period after the signal indicating the confirmation of survival transmitted in step S9 is transmitted. In response, the recovery unit 127 transmits a container recovery instruction signal to the server 10c (step S13), and the server 10c generates (recovers) a business container in response to this (step S14).

In step S15, leader continuation is asserted in the same manner as in step S6, and in step S16, leader survival confirmation processing is performed in the same manner as in step S8.

Further, in step S17, leader continuation is asserted as in step S6, and in step S18, leader survival confirmation processing is performed in the same manner as in step S8.

Also, in step S19, the first signal transmission unit 122 of the monitoring container 12a transmits a signal indicating that the business container 11 is alive to the business container 11 of the server 10c (step S19).

Next, with reference to FIG. 4, processing when the business container 11 and the monitoring container 12 are monitored by the monitoring container 12 in this embodiment and a failure is detected in the monitoring container 12 will be described. FIG. 4 is a sequence diagram showing the flow of processing when the monitoring container 12 monitors the business container 11 and the monitoring container 12 and detects a failure in the monitoring container 12 in this embodiment.

Since steps S31 to S38 are the same as steps S1 to S8 in the sequence diagram shown in FIG. 3, description thereof will be omitted.

In step S39, as in step S34, the first signal transmission unit 122 of the monitoring container 12a transmits a signal indicating that the business container 11 is alive, and the first detection unit 123 receives a response signal from the business container 11. If so, it is determined that the business container 11 is not in trouble (step S39).

In steps S40 to S43, similarly to steps S35 to S38, leader continuation assertion and leader survival confirmation processing are performed.

After step S43, a failure occurs in the server 10a. After that, the second signal transmission units 124 of the servers 10b and 10c transmit a signal indicating survival confirmation to the monitoring container 12a (steps S44 and S45).

The second detection units 125 of the servers 10b and 10c detect that a failure has occurred in the monitoring container 12a after a waiting period (step S46) has elapsed since the second signal transmission unit 124 transmitted a signal indicating survival confirmation. detect.

In response to this, the role determining unit 121 of the monitoring container 12b transmits a signal indicating the leader candidacy to the monitoring container 12c, and determines the monitoring container 12b to be the monitoring container that monitors the business container 11 (step S47).

The first signal transmission unit 122 of the monitoring container 12b transmits a signal indicating that the business container 11 is alive. It is determined that there is no (step S48).

In addition, the role determining unit 121 of the monitoring container 12b transmits a signal indicating the assertion of leader continuation to the monitoring container 12c every predetermined period (step S49). Further, the second signal transmission unit 124 of the monitoring container 12c transmits a signal indicating confirmation of survival to the monitoring container 12b, and the second detection unit 125 of the monitoring container 12c receives a response signal from the monitoring container 12b to confirm that the container is alive. Confirm (step S50).

Also, the first signal transmission unit 122 of the monitoring container 12b transmits a signal indicating that the business container 11 is alive, and when the first detection unit 123 receives a response signal from the business container 11, the business container 11 fails It is determined that it is not (step S51).

Next, an example of a monitoring status screen will be described with reference to FIG. FIG. 5 is a diagram for explaining an example of a monitor status screen.

The screen in FIG. 5 is a screen generated by the control server 20 based on information acquired from the detection result output unit 126, for example.

The example of FIG. 5 shows that three servers are managed. Specifically, the server whose node name is "Worker#1" has a monitoring container whose identifier is the monitoring container M1. Also, the server whose node name is "Worker#2" has a business container whose identifier is the business container C1 and a monitoring container whose identifier is the monitoring container M2. Also, the server with the node name "Worker#3" has a business container with the identifier of the business container C2 and a monitoring container with the identifier of the monitoring container M3.

The example of FIG. 5 shows that the monitoring container M1 monitors the business containers C1 and C2, and the monitoring containers M2 and M3 monitor the monitoring container M1.

"HealthCheck Response Time" shows a graph of the response time to the signal indicating the confirmation of the existence of the business container C1 and the business container C2.

Further, "Message Count" shows the transition of the number of messages for each message level (Info, Warn, Error).

The detail field D1 shows the business container that sent the message, the message transmission date, the message transmission time, the message level, the node to which the business container belongs, and the specific message content.

Although the monitoring container 12 has the role determination unit 121 in the above embodiment, the control server 20 may have the role determination unit 121 .

In the above-described embodiment, the case where the role determination unit 121 dynamically determines the leader was described, but the leader may be fixed in advance.

Although the case where the monitoring container 12 has the first signal transmission unit 122, the first detection unit 123, the second signal transmission unit 124, and the second detection unit 125 has been described, the present invention is not limited to this. If it is decided, it is not necessary to have all of them.

<Effect>
In the monitoring container 12 (monitoring container 12a in the above embodiment) that monitors the business container in the distributed container monitoring system 1 described above, the first signal transmission unit 122 sends the A signal indicating status confirmation is transmitted, and the first detection unit 123 detects a failure of the business container 11 based on the response status of the business container 11 to the signal transmitted by the first signal transmission unit 122 .

Also, in the monitoring container 12 (monitoring container 12c in the above embodiment) monitoring the monitoring container 12 monitoring the business container, the second signal transmitting unit 124 transmits a signal indicating status confirmation to the monitoring container 12a. Then, the second detection unit 125 detects a failure of the monitoring container 12a based on the response status of the monitoring container 12a to the signal transmitted by the second signal transmission unit 124. FIG.

In this case, since the distributed container monitoring system 1 further monitors the monitoring container 12 that monitors the business container 11, even if a failure occurs in the monitoring container 12 that monitors the business container 11, the failure can be dealt with promptly. can be done. That is, failure monitoring can be performed more appropriately. Furthermore, in the distributed container monitoring system 1 of the above-described embodiment, server resources can be utilized to the maximum and fault monitoring can be performed with a simpler implementation without using blockchain.

Also, the role determining unit 121 determines whether each of the monitoring containers 12 functions as the monitoring container 12 that monitors the business container 11 or functions as the monitoring container 12 that monitors the monitoring container 12 that monitors the business container 11 . do.

As described above, in the distributed container monitoring system 1, the role determination unit 121 can dynamically divide the roles by allocating the roles of the plurality of monitoring containers 12. FIG.

Further, when the second detection unit 125 detects a failure of the monitoring container 12 that monitors the business container 11 , the role determination unit 121 functions as the monitoring container 12 that monitors the business container 11 . Let

In this way, the distributed container monitoring system 1 detects a failure in the monitoring container 12 that monitors the business container 11, and uses another monitoring container 12 as the monitoring container 12 that monitors the business container 11. By doing so, the business container 11 can be appropriately monitored.

Further, when the first detection unit 123 detects a failure of the business container 11, which is a container to be monitored, the restoration unit 127 performs restoration processing of the container environment. As a result, the distributed container monitoring system 1 can appropriately perform fault recovery for the business container 11 . Further, depending on the application that executes the business container 11, recovery may be performed by the function of the application itself, but the recovery unit 127 can perform failure recovery without relying on it.

Although the invention made by the present inventors has been specifically described based on the embodiments, the present invention is not limited to the above embodiments, and can be variously modified without departing from the scope of the invention. Needless to say. For example, the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. . Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

Further, in each of the above drawings, control lines and information lines are those considered to be necessary for explanation, and not all control lines and information lines for implementation are necessarily shown. In fact, it may be considered that almost all configurations are interconnected.

INDUSTRIAL APPLICABILITY The present invention can be used in a distributed container monitoring system that monitors the operational status of containers to be monitored.

DESCRIPTION OF SYMBOLS 1... Distributed container monitoring system 10... Server 11... Business container 12... Monitoring container 121... Role determination part 122... First signal transmission part 123... First detection part,
124...Second signal transmission unit, 125...Second detection unit, 126...Detection result output unit, 127...Recovery unit.

Claims (5)

A distributed container monitoring system that monitors the operating status of containers to be monitored,
a first monitoring container that monitors the monitored container;
a second monitoring container that monitors the first monitoring container;
The first monitoring container includes:
a first signal transmission unit that transmits a signal indicating status confirmation to the container to be monitored;
a first detection unit that detects a failure of the container to be monitored based on the response status of the container to be monitored to the signal transmitted by the first signal transmission unit;
The second monitoring container comprises:
a second signal transmission unit that transmits a signal indicating status confirmation to the first surveillance container;
a second detection unit that detects a failure of the first monitoring container based on the response status of the first monitoring container to the signal transmitted by the second signal transmission unit;
Distributed container monitoring system. A distributed container monitoring system according to claim 1,
having a plurality of monitoring containers that are containers capable of monitoring the container to be monitored;
each of the monitoring containers further comprising a role determining unit that determines whether each of the monitoring containers functions as the first monitoring container or the second monitoring container;
Distributed container monitoring system. A distributed container monitoring system according to claim 2,
When the second detection unit detects a failure of the first monitoring container, the role determination unit causes another monitoring container to function as the first monitoring container.
Distributed container monitoring system. A distributed container monitoring system according to claim 1 or 2,
The first monitoring container includes:
Further comprising a restoration unit that performs restoration processing of the container environment when the failure of the container to be monitored is detected by the first detection unit,
Distributed container monitoring system. A distributed container monitoring method executed by a distributed container monitoring system for monitoring the operating status of containers to be monitored,
a first monitoring container that monitors the monitored container;
a second monitoring container that monitors the first monitoring container;
In the first monitoring container,
a first signal transmission step of transmitting a signal indicating status confirmation to the container to be monitored;
a first detection step of detecting a failure of the container to be monitored based on the response status of the container to be monitored to the signal transmitted in the first signal transmission step;
In the second monitoring container,
a second signal transmission step of transmitting a signal indicating status confirmation to the first surveillance container;
a second detection step of detecting a failure of the first monitoring container based on the response status of the first monitoring container to the signal transmitted in the second signal transmission step;
Distributed container monitoring method.

Images