JP2020135336A - Monitoring system, and monitoring method, and monitoring program - Google Patents

Abstract

To provide a monitoring system capable of saving the users' time of resetting the threshold for monitoring an information system whose version is updated.SOLUTION: A monitoring system 20 includes an update unit 21 that updates a range that is generated on the basis of each performance value of predetermined monitoring items of a plurality of monitoring object devices, namely the range in which excess of the performance value shows abnormal predetermined monitoring item, on the basis of each performance value of one or more monitoring object devices when the versions of one or more, of the plurality of monitoring object devices, are updated.SELECTED DRAWING: Figure 12

Description

The present invention relates to monitoring systems, monitoring methods and monitoring programs.

With the spread of cloud computing, DevOps, container-type virtualization technology, etc., new versions that capture user demand are easily released even for large-scale service and business system versions. That is, versions of services and the like are updated more frequently than before the above-mentioned technology became widespread.

Service providers are required to quickly capture user demand. Therefore, software for monitoring application performance information is also required to respond to the speed at which the version of the business system to be monitored is updated.

When the frequency of version updates was not higher than before the above technologies became widespread, software developers reconsidered and defined what kind of information should be monitored each time a version was updated. I was able to take.

Even now that the above technologies have become widespread and versions are often updated without the user's awareness, software developers are required to take measures so that the software can be continuously monitored. ..

For example, when monitoring the performance of an information system that provides a service whose version is updated frequently with software, the operation manager sets the content of monitoring one by one every time the version is updated. If the method of setting the monitoring contents one by one is adopted every time the version is updated, it takes a lot of time to set the monitoring contents as a whole.

In information system monitoring, a method is usually adopted in which a threshold value is set for the performance information to be collected and an excess of the performance information threshold value is detected as a performance abnormality. However, there are also ways to monitor an information system without setting a threshold.

As a method of monitoring an information system without setting a threshold value, for example, there is a baseline monitoring method described in Patent Document 1. The baseline monitoring method described in Patent Document 1 is a method focusing on the existence of periodicity in the load applied to the information system.

Specifically, the baseline monitoring method described in Patent Document 1 generates a baseline based on the performance information for a predetermined period, and uses the generated baseline instead of the threshold value. In baseline monitoring, the operations manager is not required to set thresholds for monitoring each time a version is updated.

Further, Patent Document 2 describes a management computer focusing on shortening the period of baseline generation. The management computer described in Patent Document 2 uses the performance information of a plurality of virtual calculation units belonging to the same autoscaling group to generate a baseline. The management computer described in Patent Document 2 proposes a method in which a plurality of performance information is used for generating a baseline, that is, a method capable of generating a baseline even if the learning period of the baseline is short.

Further, Patent Document 3 describes an application management system focusing on the continuity of monitoring before and after updating the version. The application management system described in Patent Document 3 states that the performance has changed by comparing the performance information of the application before the version is updated with the performance information of the application after the version is updated. Present the likely item to the user.

Japanese Unexamined Patent Publication No. 2004-164637 International Publication No. 2017/168844 International Publication No. 2018/037561

When the baseline monitoring method described in Patent Document 1 is used, there is a problem that it takes time to regenerate the baseline. Even if it is adopted in the operation of a system in which the version is updated frequently, the baseline monitoring method described in Patent Document 1 may regenerate the baseline as quickly as the version is updated. Have difficulty.

Further, the management computer described in Patent Document 2 does not assume the update of the version. In general, when a version is updated, there is a high possibility that changes will occur in the performance trends and load conditions of information systems. Therefore, if the baseline of the old version is used as it is and monitoring is performed, there is a risk that a change in the performance tendency may be mistakenly detected as an abnormality.

Usually, the baseline calculation method is a method that uses the average value and variance of the performance information. Where the state is stable at the baseline before the version update, the variance is small. The reason why changes in performance trends are mistakenly detected as abnormal is that there is a risk that the value at the point where the version is updated and the load status changes exceeds the baseline range of the previous version and is detected as abnormal. This is to increase.

Due to the above problem, in order to continuously perform highly accurate monitoring, it is required to regenerate the baseline even when the method described in Patent Document 2 having a short learning period is used. There is a problem of being able to do it. That is, even if the method described in Patent Document 2 is used, the problem that it is difficult to regenerate the baseline at the same speed as the version update is not solved.

Further, when the application management system described in Patent Document 3 is used, the user is required to set the monitoring again based on the difference between the new version and the old version. When a user configures monitoring, the time it takes to make an operation error or reconfigure can be a factor that delays development with DevOps, for example.

Therefore, the present invention provides a monitoring system, a monitoring method, and a monitoring program that solve the above-mentioned problems and can save the user the trouble of resetting the monitoring threshold value of the information system whose version is updated. With the goal.

The monitoring system according to the present invention is a range generated based on each performance value of a predetermined monitoring item of a plurality of monitored devices, and a range in which the predetermined monitoring item is determined to be abnormal when the performance value is exceeded. Is characterized by including an update unit that updates based on each performance value of one or more monitored devices when each version of one or more monitored devices among a plurality of monitored devices is updated. ..

The monitoring method according to the present invention is a range generated based on each performance value of a predetermined monitoring item of a plurality of monitored devices, and a range in which the predetermined monitoring item is determined to be abnormal when the performance value is exceeded. Is updated based on each performance value of one or more monitored devices when each version of one or more monitored devices among the plurality of monitored devices is updated.

The monitoring program according to the present invention is a range generated by the computer based on each performance value of predetermined monitoring items of a plurality of monitored devices, and when the performance value is exceeded, it is determined that the predetermined monitoring item is abnormal. When each version of one or more monitored devices among a plurality of monitored devices is updated, an update process for updating the range to be performed is executed based on each performance value of one or more monitored devices. It is characterized by.

According to the present invention, it is possible to save the user the trouble of resetting the threshold value for monitoring the information system whose version is updated.

It is explanatory drawing which shows the use example of 1st Embodiment of the monitoring system by this invention. It is a block diagram which shows the configuration example of the monitoring system 100 of 1st Embodiment. It is explanatory drawing which shows the example of the performance information stored in the performance information storage unit 109. It is explanatory drawing which shows the example of the group information stored in the group information storage unit 110. It is explanatory drawing which shows the example of the baseline information which is stored in the baseline storage part 112. It is explanatory drawing which shows the example of the configuration change information stored in the configuration change information storage unit 111. It is explanatory drawing which shows the example of the change of the number of container instances at the time of rolling update. It is a flowchart which shows the operation of the performance abnormality detection processing in a normal time by the monitoring system 100 of 1st Embodiment. It is a flowchart which shows the operation of the baseline update process by the monitoring system 100 of 1st Embodiment. It is a flowchart which shows the operation of the performance abnormality detection processing at the time of version update by the monitoring system 100 of 1st Embodiment. It is explanatory drawing which shows the hardware configuration example of the monitoring system by this invention. It is a block diagram which shows the outline of the monitoring system by this invention.

Embodiment 1.
[Description of configuration]
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an explanatory diagram showing an example of use of the first embodiment of the monitoring system according to the present invention.

FIG. 1 shows an outline of the relationship between the monitoring system according to the present invention and an external component. As shown in FIG. 1, the monitoring system 100 of the present embodiment is communicably connected to the terminal 200, the container management system 300, and the container instances 400 to 40N (N is an integer of 1 or more). ..

Further, as shown in FIG. 1, the container management system 300 is communicably connected to the container instance 400 to the container instance 40N, respectively. Hereinafter, the roles of each component shown in FIG. 1 will be described.

In order to solve the above problems, in the present embodiment, a monitoring system 100 that monitors an information system whose version is updated frequently by a baseline is used. Baselines are generated using the averaging and distribution of performance information for container instances that belong to the same autoscaling group, similar to known techniques.

In this embodiment, since an information system using the container-type virtualization technology is assumed as an environment in which the version is updated frequently, the acquisition target of the performance information will be described as a container instance.

When it does not depend on the container-type virtualization technology, the acquisition target of the performance information may be the virtual calculation unit as in the known technology. The monitoring target of this embodiment is not limited to the container-type virtual environment, but may be a physical computer environment or a hypervisor-type virtual environment.

Further, the monitoring system 100 of the present embodiment is characterized in that it can continuously perform monitoring without regenerating the baseline before and after updating the version. The version update is a rolling update that is normally used for container instances.

The monitoring system 100 sequentially updates the baseline generated based on the performance information of the old version container instance by using the performance information of the new version container instance that is started one by one at the time of rolling update.

By monitoring the container instance using the sequentially updated baseline during the version update and after the version update is completed, the monitoring system 100 does not have to regenerate the baseline in accordance with the version update.

In addition, the monitoring system 100 of the present embodiment has a function of periodically collecting and accumulating performance information from the information system to be monitored, and a function of generating a baseline based on the accumulated performance information. It has a function to detect performance deterioration using the baseline and notify the operation manager. Further, the monitoring system 100 has a function of collecting and managing configuration information in order to grasp the update status of the version of the information system to be monitored.

The terminal 200 is one of the components provided by the monitoring system 100. As shown in FIG. 1, the terminal 200 is arranged outside the monitoring system 100.

A user of the monitoring system 100, such as an operation manager, operates the monitoring system 100 via the terminal 200, refers to the performance information collected by the monitoring system 100, receives a notification of performance deterioration detected by the monitoring system 100, and the like. Can be executed. The terminal 200 has a function generally required by the user for the monitoring system 100.

The container instance 400 to the container instance 40N are one or more virtual arithmetic units to be monitored by the monitoring system 100. Since the use of the container-type virtualization technology is assumed in this embodiment, the monitoring target is a container instance.

Any application or program runs in each container instance. The monitoring system 100 collects each performance information of the container instance 400 to the container instance 40N.

The container management system 300 has a function of managing each container instance by starting, stopping, autoscaling, or the like of each container instance. Generally, a device that manages a container instance is called an orchestrator or the like.

In the present embodiment, the device that manages the container instance is the container management system 300. The container management system 300 has information for associating each container instance with an autoscaling group.

The container management system 300 has a function of generating a plurality of container instances having a predetermined function and maintaining the generated number of container instances for the purpose of availability and load distribution. Further, the container management system 300 has a function of performing auto-scaling that automatically increases or decreases the number of container instances according to the load status.

The container management system 300 performs autoscaling for each autoscaling group. In this embodiment, the version update of each container instance is performed for each autoscaling group. The baseline is also generated by using the performance information of all the container instances belonging to the autoscaling group.

The container management system 300 receives a version update request for each container instance, and updates the version of each container instance by rolling update.

The monitoring system 100 collects information on container instances belonging to the same autoscaling group, information indicating the startup status of the container instances, and the like from the container management system 300. The monitoring system 100 uses the collected information to generate and update the baseline.

As described above, in the present embodiment, a case where the container-type virtualization technology is used will be described as an example. However, if the monitoring system 100 can collect information, the version is updated by rolling update, and the configuration information of the virtual arithmetic unit to be monitored can be acquired from the monitoring system 100, the arithmetic unit to be monitored may be a physical machine. However, it may be a virtual machine generated by the hypervisor type virtualization technology. The configuration information of the virtual calculation unit is information indicating a version update status and information of an autoscaling group.

Next, the internal and external components of the monitoring system 100 and the roles of each component will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration example of the monitoring system 100 of the first embodiment.

As shown in FIG. 2, the monitoring system 100 includes a performance information reception unit 101, a performance information processing unit 102, a performance abnormality detection unit 103, a management information reception unit 104, a configuration change management unit 105, and group information management. A unit 106, a baseline generation unit 107, an operation reception unit 108, a performance information storage unit 109, a group information storage unit 110, a configuration change information storage unit 111, and a baseline storage unit 112 are included.

The performance information receiving unit 101 has a function of receiving each performance information of the container instance 400 to the container instance 40N collected by the performance information collecting unit 410 to the performance information collecting unit 41N, respectively. The performance information reception unit 101 passes each received performance information to the performance information processing unit 102.

The performance information processing unit 102 has a function of receiving performance information from the performance information receiving unit 101, shaping the received performance information into a predetermined first format, and then storing the received performance information in the performance information storage unit 109.

The performance information storage unit 109 has a function of storing performance information to be monitored, for example, in a file or a database. The performance information storage unit 109 holds the performance information in the format shown in FIG. 3, which will be described later. When the request for provision is received, the performance information storage unit 109 provides the requested performance information to the request source.

The performance abnormality detection unit 103 acquires the performance information of the container instance and the baseline information for any container instance, and when the performance value of the container instance exceeds the baseline range, "there is an abnormality in the performance". It has a function of notifying the terminal 200 as.

In addition, the performance abnormality detection unit 103 notifies the user of a value that exceeds the baseline range (threshold value). In order to make it easier for the notified user to determine whether or not there is an abnormality, the performance abnormality detection unit 103 determines the range of the baseline of the new version and the range of the baseline of the old version at the time of version update such as rolling update. You may also notify the user.

The performance abnormality detection unit 103 determines whether or not there is an abnormality in the performance as follows, for example. When the degree of change is small, for example, the tendency of the performance information of the new version is similar to the tendency of the performance information of the old version. Therefore, if the performance value exceeds the threshold value, there is a high possibility that it is abnormal.

However, when the degree of change is large, the tendency of the performance information of the new version may have changed from the tendency of the performance information of the old version, and it becomes difficult to judge whether or not it is abnormal. Therefore, for example, whether or not there are other performance values that exceed the threshold value in the container instance is considered. If there are other performance values that exceed the threshold value, there is a high possibility that it is abnormal.

If there is no other performance value that exceeds the threshold value, it is difficult to determine whether or not there is an abnormality. Therefore, the performance abnormality detection unit 103 waits until, for example, the rolling update is completed.

The management information receiving unit 104 has a function of receiving the configuration information collected by the management information collecting unit 310. The management information reception unit 104 passes information on configuration changes such as version updates to the configuration change management unit 105. Further, the management information receiving unit 104 passes information indicating an autoscaling group to which the operating container instance belongs to the group information management unit 106.

The configuration change management unit 105 has a function of shaping the configuration change information passed from the management information reception unit 104 into a predetermined second format and then storing it in the configuration change information storage unit 111.

The configuration change information storage unit 111 has a function of storing configuration change information in, for example, a file or a database. The configuration change information storage unit 111 holds the configuration change information in the format shown in FIG. 6, which will be described later. When the request for provision is received, the configuration change information storage unit 111 provides the configuration change information of the request target to the request source.

The group information management unit 106 has a function of storing the group information, which is the information for associating the auto-scaling group and the container instance passed from the management information reception unit 104, in the group information storage unit 110.

The group information storage unit 110 has a function of storing group information in, for example, a file or a database. The group information storage unit 110 holds the group information in the format shown in FIG. 4, which will be described later. When the request for provision is received, the group information storage unit 110 provides the group information of the request target to the request source.

The baseline generation unit 107 has a function of generating baseline information indicating the contents of the baseline for each autoscaling group based on the performance information. The baseline generation unit 107 stores the generated baseline information in the baseline storage unit 112.

The baseline storage unit 112 has a function of storing baseline information in a predetermined third format.

The operation receiving unit 108 has a function of receiving an operation from the terminal 200. According to the received operation, the operation reception unit 108 instructs each component in the monitoring system 100 to regenerate the baseline, change the threshold value, and the like.

Further, as shown in FIG. 2, as components arranged outside the monitoring system 100, there are a terminal 200, a management information collecting unit 310, and a performance information collecting unit 410 to a performance information collecting unit 41N. The terminal 200, the management information collecting unit 310, and the performance information collecting unit 410 to the performance information collecting unit 41N are all components provided by the monitoring system 100.

The terminal 200 is a terminal accessible to users of the monitoring system 100 such as an operation manager. The terminal 200 has a function of receiving a notification from the performance abnormality detection unit 103 and referring to the content of the received notification, and a function of operating the monitoring system 100.

The user of the monitoring system 100 who has referred to the notification received by the terminal 200 can take a measure of instructing the container management system 300 to execute the scale-out in response to the performance abnormality in the container instance.

Alternatively, the user of the monitoring system 100 can take measures such as issuing an instruction to the monitoring system 100 to change the baseline or modify the application. Therefore, the user can suppress the influence of the performance deterioration or abnormality of the application.

The management information collection unit 310 has a function of collecting information on the container instance held by the container management system 300.

The performance information collecting unit 410 to the performance information collecting unit 41N has a function of collecting performance information of each container instance. The performance information collecting unit 410 to the performance information collecting unit 41N may be arranged inside each container instance or may be arranged outside.

The information held by each storage unit in the monitoring system 100 will be described below. FIG. 3 is an explanatory diagram showing an example of performance information stored in the performance information storage unit 109. As shown in FIG. 3, the performance information is composed of at least a collection time, an instance name, a CPU (Central Processing Unit) usage rate, and a memory usage amount.

The collection time is the time when the performance information is collected from the container instance to be monitored. The instance name is the name of the container instance for which performance information has been collected. The instance name is information that can uniquely identify the monitoring target.

The CPU usage rate (%) and memory usage (MB) are performance values collected from the container instance to be monitored. The items to be monitored may be items other than the CPU usage rate and the memory usage amount. Items to be monitored are arbitrarily specified.

The items to be monitored for which performance values are collected can be arbitrarily defined by the operation administrator. However, for container instances that belong to the same autoscaling group, the types of monitored items for which performance values are collected are unified.

For example, regarding the CPU usage rate and the memory usage amount in the performance information shown in FIG. 3, the operation manager can define that "the memory usage amount is not collected from the container instance belonging to the Postgres-v9 group" via the terminal 200. Similarly, the operation administrator can define that "additional disk usage (%) information is collected from the container instances belonging to the nginx group" via the terminal 200.

However, the operation manager does not define that "CPU usage is collected from Nginx-01 but not from Nginx-02". The reason for not defining it is that the baseline generation unit 107 uses the performance information of each container instance belonging to the same autoscaling group when generating the baseline.

In other words, if the performance information collected from container instances belonging to the same autoscaling group is different, there is a risk that the performance information will not be collected enough for the baseline generation and it will take time to generate the baseline. Because.

Another reason is that if a baseline generated biased toward performance information collected from a specific container instance is used, there is a risk that the accuracy of monitoring of other container instances will decrease.

FIG. 4 is an explanatory diagram showing an example of group information stored in the group information storage unit 110. As shown in FIG. 4, the group information is composed of a group name and an instance name.

The group name is the name of the auto-scaling group on which the above-mentioned auto-scaling is performed. The group name is a name that uniquely identifies the autoscaling group.

The instance name is a name that uniquely identifies the container instance. The instance name shown in FIG. 4 is the same as the instance name shown in FIG.

FIG. 5 is an explanatory diagram showing an example of baseline information stored in the baseline storage unit 112. As shown in FIG. 5, the baseline information is composed of at least a baseline time, a group name, a CPU usage rate, and a memory usage amount.

Baseline information is generated at predetermined intervals on the premise that there is periodicity in the load status of the information system. In the example shown in FIG. 5, the cycle of the load condition is assumed to be one week.

The baseline time is a time for associating the collection time of performance information with the baseline. In the example shown in FIG. 5, since the cycle of the load status corresponding to the baseline is one week, the baseline time is displayed by the day of the week and the time.

If the load status cycle supported by the baseline is one month, the baseline time is displayed as "XX day time". Any cycle may be specified for the cycle of the load condition to which the baseline corresponds.

The group name is the name of the autoscaling group. The group name shown in FIG. 5 is the same as the group name shown in FIG. That is, the baseline is shared by each container instance belonging to the same autoscaling group, similar to the baseline described in Patent Document 2.

In addition, the CPU usage rate and the memory usage amount are baselines at the baseline time for the container instances belonging to the autoscaling group indicated by the group name. The baseline calculation formula is, for example, the following formula.

The second term of equation (1) is σ (standard deviation) shown in FIG. Note that x _i in the equation (1) is a performance value at baseline time of any container instance belonging to the autoscaling group indicated by the group name.

The first term of the equation (1) is the average value of the performance values at the baseline time of all the container instances belonging to the autoscaling group indicated by the group name. In addition, n is the number of container instances belonging to the autoscaling group indicated by the group name.

The items to be generated as the baseline may be items other than the CPU usage rate and the memory usage amount. A baseline is generated for each item to be monitored shown in FIG.

Further, at the time of version update, the baseline generation unit 107 calculates the baseline according to the following formula.

In addition, m in the formula (2) is the sum of the number of container instances of the old version and the number of container instances of the new version. Moreover, m_old in the formula (2) is the number of container instances of the old version.

The baseline generation unit 107 acquires information on the running container instance based on the group information of the old version and the group information of the new version acquired from the group information storage unit 110. The baseline generation unit 107 updates the baseline based on the acquired information.

The term ((m + m_old) / m), which is a different term from the equation (1) in the equation (2), takes a value between 1 and 2. ((m + m_old) / m) approaches 2 as the number of container instances of the old version is running, and approaches 1 as the number of container instances of the new version is running.

The baseline generation unit 107 is required to update the baseline in a state where the performance information of the new version container instance is small and the performance information of the old version container instance is large at the beginning of the rolling update.

Therefore, by doubling the width of the baseline (σ shown in FIG. 5) at the maximum, the baseline generation unit 107 suppresses false detection of the performance information of the new version. However, obvious outliers and the like exceed the baseline range. That is, the performance abnormality detection unit 103 that continuously executes monitoring can detect the performance abnormality.

When all older versions of container instances are shut down, the baseline width is multiplied by 1. That is, the baseline calculation formula (2) matches the normal calculation formula (1). When all the old version container instances are stopped, the baseline update based on the performance information of the new version container instance is completed.

The calculation formula used for the baseline calculation at the time of version update in which rolling update or the like is performed may be a calculation formula other than the formula (2).

FIG. 6 is an explanatory diagram showing an example of configuration change information stored in the configuration change information storage unit 111. As shown in FIG. 6, the configuration change information is composed of the last update time, the type of change, the old version group name, the new version group name, the number of old version instances, and the number of new version instances. ..

The old version group name is the name defined as the name of the autoscaling group to which the container instance before the configuration change belongs. The new version group name is a name defined as the name of the autoscaling group to which the container instance newly released by the configuration change belongs.

As described above, the group name is a name that uniquely identifies the autoscaling group. That is, when the old version group name and the new version group name are different, the autoscaling group to which the old version container instance belongs and the autoscaling group to which the new version container instance belongs are different.

Note that the configuration change information may include information on other versions so that the versions are not distinguished by the group name. In this embodiment, as in the example shown in FIG. 6, the versions are distinguished by the group name.

The number of old version instances is the number of container instances that belong to the autoscaling group indicated by the old version group name and are in operation after the configuration change process is started.

The number of new version instances is the number of container instances that belong to the autoscaling group indicated by the new version group name and are in operation after the configuration change process is started.

FIG. 7 is an explanatory diagram showing an example of a change in the number of container instances at the time of rolling update. The black rectangle shown in FIG. 7 represents a running container instance. The broken line rectangle shown in FIG. 7 represents a stopped container instance.

Rolling update is a method of updating a system without stopping the system. Specifically, the rolling update starts the new version of container instances one by one and stops the old version of container instances one by one instead. The number of container instances that the rolling update starts or stops at one time is not limited to one.

FIG. 7A shows a container instance immediately after the start of update. As shown in FIG. 7A, in the rolling update, when one container instance of the old version is stopped, only one container instance of the new version is running.

FIG. 7B shows a container instance immediately before the end of the update. As shown in FIG. 7 (b), the rolling update is when four new version container instances are started. Only one old version of the container instance is running.

As described above, the number of old version instances shown in FIG. 6 is the number of old version container instances running during the version update shown in FIG. 7. The number of new version instances shown in FIG. 6 is the number of new version container instances running during the version update shown in FIG. 7.

The type of change is the type of configuration change. In addition to version updates, the types of configuration changes include scale-out, scale-in, new addition, and deletion. Any information can be specified as the type of change. In this embodiment, the type of change mainly means a version update. The last update time is the last time when the configuration change information is updated.

[Explanation of operation]
Hereinafter, the operation of the monitoring system 100 of the present embodiment will be described with reference to FIGS. 8 to 10.

A feature of the monitoring system 100 using the baseline is that the baseline is generated and updated by a method different from the normal time when the version is updated. The monitoring system 100 of the present embodiment performs a process in which the performance abnormality detection process using the baseline shown in FIG. 8 and the baseline update process at the time of version update shown in FIG. 9 are combined.

Further, the process when an abnormality is detected in the container instance of the new version during the version update is different from the process shown in FIG. Therefore, only the part different from the process shown in FIG. 8 will be described with reference to the performance abnormality detection process at the time of version update shown in FIG. Each process will be described below.

First, an operation of detecting a performance abnormality using the baseline of the monitoring system 100 of the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing the operation of the performance abnormality detection process in the normal state by the monitoring system 100 of the first embodiment.

First, the performance information collecting unit 410 to the performance information collecting unit 41N collects performance information from each container instance to be monitored at each collection interval specified by the monitoring system 100 (step S101). The performance information includes at least the information shown in FIG.

The performance information collecting unit 410 to the performance information collecting unit 41N transmits each collected performance information to the performance information receiving unit 101. After transmission, the performance information collecting unit 410 to the performance information collecting unit 41N stand by until the next collection timing. The performance abnormality detection process shown in FIG. 8 is periodically executed with the collection of performance information as an opportunity.

Next, the performance information receiving unit 101 inputs each performance information received from the performance information collecting unit 410 to the performance information collecting unit 41N into the performance information processing unit 102 (step S102).

Next, the performance information processing unit 102 stores each performance information input from the performance information receiving unit 101 in the performance information storage unit 109 (step S103).

After storing each performance information in the performance information storage unit 109, the performance information processing unit 102 inputs each stored performance information to the performance abnormality detection unit 103 (step S104).

Next, when each performance information is input from the performance information processing unit 102, the performance abnormality detection unit 103 groups the group name of the auto scaling group to which the container instance corresponding to the input performance information belongs based on the instance name. Inquire to the information storage unit 110. After making the inquiry, the performance abnormality detection unit 103 acquires the group name from the group information storage unit 110 (step S105).

For example, when the performance information of the container instance "Nginx-01" is input, the performance abnormality detection unit 103 inquires the group information storage unit 110 for the autoscaling group to which the container instance "Nginx-01" belongs. After making an inquiry, the performance abnormality detection unit 103 receives a notification from the group information storage unit 110 that it belongs to the group "nginx".

Next, the performance abnormality detection unit 103 refers to the baseline information stored in the baseline storage unit 112, and acquires the baseline information in which the group name and the collection time of the performance information match (step S106).

For example, the performance abnormality detection unit 103 acquires baseline information based on the collection time included in the performance information (for example, Wednesday 21:30) and the group name “nginx” acquired from the group information storage unit 110. To do.

Next, the performance abnormality detection unit 103 compares the acquired baseline of the baseline information with the value of the corresponding monitoring target item of the performance information over each performance information (step S107). By comparing, the performance abnormality detection unit 103 determines whether or not there is an abnormality in the performance item indicated by each performance information (step S108).

Specifically, when the value of the item to be monitored exceeds the range of the baseline, the performance abnormality detection unit 103 determines that the item to be monitored has a sign of deterioration or an abnormality. For example, when the CPU usage rate of the container instance "Nginx-01" exceeds the range of the CPU usage rate "60 ± 3σ" of the corresponding baseline information, the performance abnormality detection unit 103 determines that the CPU usage rate is abnormal. to decide.

When it is determined that there is no abnormality in the item to be monitored (No in step S108), the monitoring system 100 ends the performance abnormality detection process.

When it is determined that the item to be monitored has an abnormality (Yes in step S108), the performance abnormality detection unit 103 transmits the performance information determined to be abnormal to the terminal 200 (step S109).

Next, the user of the monitoring system 100 refers to the transmitted performance information and inputs an operation for resolving the abnormality to the operation reception unit 108 via the terminal 200 (step S110). After the operation is input, the monitoring system 100 ends the performance abnormality detection process.

Next, the operation of updating the baseline at the time of updating the version of the monitoring system 100 of the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing the operation of the baseline update process by the monitoring system 100 of the first embodiment.

The normal baseline generation method is a method based on the function of the baseline generation unit 107 described above. FIG. 9 shows the baseline update process at the time of version update. The process shown in FIG. 9 is started when a version update request is made to the container management system 300.

The management information collecting unit 310, which is always in operation in the container management system 300, detects that the container management system 300 has received the version update request. After the detection, the management information collecting unit 310 transmits the changed content of the configuration due to the version update and the current configuration content to the management information receiving unit 104 (step S201).

Next, the management information reception unit 104 classifies the received information into configuration change information and group information. After the classification, the management information receiving unit 104 inputs the configuration change information to the configuration change management unit 105 (step S202). Further, the management information receiving unit 104 inputs the group information to the group information management unit 106 (step S203).

Next, the configuration change management unit 105 stores the input configuration change information in the configuration change information storage unit 111 (step S204). Next, the group information management unit 106 stores the input group information in the group information storage unit 110 (step S205).

For example, the group information management unit 106 adds the group information of the new version of the container instance started by the rolling update to the group information storage unit 110. Further, the group information management unit 106 may delete the group information of the stopped old version container instance from the group information storage unit 110.

Next, the configuration change information storage unit 111 and the group information storage unit 110 input the updated information to the baseline generation unit 107 (step S206).

Next, the baseline generation unit 107 acquires the performance information of the newly added container instance from the performance information storage unit 109 based on the input updated information (step S207).

Next, the baseline generation unit 107 updates the baseline information using the equation (2) based on the acquired performance information (step S208).

Next, the baseline generation unit 107 stores the updated baseline information in the baseline storage unit 112 (step S209). After storing, the monitoring system 100 ends the baseline update process.

The baseline storage unit 112 registers the updated baseline information as the baseline information of the new version of the autoscaling group. When all the container instances of the old version are stopped, the baseline storage unit 112 may delete the baseline information of the old version.

Next, FIG. 10 shows an operation of detecting an abnormality in a new version of the container instance of the monitoring system 100 of the present embodiment by using the baseline information at the time of version update updated by the baseline update process shown in FIG. It will be explained with reference to.

FIG. 10 is a flowchart showing the operation of the performance abnormality detection process at the time of version update by the monitoring system 100 of the first embodiment. The performance abnormality detection process in the normal state is the process shown in FIG.

The processes of steps S301 to S305 are the same as the processes of steps S101 to S105 shown in FIG.

Next, the performance abnormality detection unit 103 acquires the configuration change information from the configuration change information storage unit 111 based on the acquired group name (step S306). Based on the acquired configuration change information, the performance abnormality detection unit 103 confirms the number of new version container instances in operation (step S307).

If only one new version of the container instance is running, the running container instance is the first new version of the container instance. That is, the performance information of the new version of the container instance has not yet generated a baseline.

Therefore, when only one new version of the container instance is running (No in step S308), the performance abnormality detection unit 103 does not change the updated baseline information to the acquisition target, and proceeds to the process of step S310. ..

When two or more new version container instances are running (Yes in step S308), the performance abnormality detection unit 103 changes the updated baseline information to the acquisition target (step S309). The baseline information changed to the acquisition target is the baseline information updated by the process shown in FIG.

The processes of steps S310 to S314 are the same as the processes of steps S106 to S110 shown in FIG.

Even if the performance value does not exceed the baseline range during the rolling update, if it exceeds the baseline range of the previous version, the performance abnormality detection unit 103 considers that there is a possibility of an abnormality. The user may be notified.

[Explanation of effect]
When monitoring the performance of a new version of a container instance, such as during a version update by rolling update, the baseline generated by using the performance information of the old version of the container instance may be used.

When the baseline of the old version is used, there is a concern that false anomalies may be detected due to changes in the performance tendency of the container instance of the new version in monitoring during a stable time period with a small distribution width.

During the version update, the monitoring system 100 of the present embodiment changes the calculation formula used for the calculation of the baseline at the normal time, and widens the range of dispersion of the baseline used. By widening the range of distribution, the monitoring system 100 prevents erroneous detection of changes in performance trends and load conditions.

Moreover, even if erroneous detection can be prevented by simply widening the range of dispersion, there is a risk that the originally detected abnormality will be overlooked. Therefore, the monitoring system 100 is being updated according to the progress of the rolling update, such as when the performance information of the new version is still small or when the performance information of the new version is sufficiently gathered, without fixing the distribution width. Use a formula that dynamically changes the width of the baseline variance of.

As described above, even in an information system whose version is updated frequently, the operation manager can continuously execute the monitoring by the baseline by using the monitoring system 100 without being aware of the version update. Moreover, even during the version update, the operation manager can perform monitoring while maintaining accuracy.

With the spread of container-type virtualization technology and DevOps, new versions that capture the needs of users are easily released even for large-scale service versions. That is, version updates and updated versions are released more frequently than before.

The operation manager who uses the monitoring system 100 of the present embodiment does not have to reset the monitoring when the version is updated in the monitoring of the information system whose version is updated frequently. The operation manager can continuously monitor the information system without erroneously detecting an abnormality.

Specifically, the monitoring system 100, which monitors the information system whose version is updated frequently and the performance status of the application running in the information system using the baseline, automatically performs the baseline during the version update by the rolling update. Update with.

Therefore, the user can continuously monitor the information system without performing the work triggered by the version update before the version update or after the version update, such as regenerating the baseline.

The application management system described in Patent Document 3 compares the performance information of the application before the version update with the performance information of the application after the version update, and extracts the metric whose performance is changed by the version update. To do.

The application management system described in Patent Document 3 can perform extraction of changing metrics and feedback to the user. However, users are required to take measures after execution, such as monitoring and changing threshold settings.

The monitoring system 100 of this embodiment also uses the performance information before and after the version update. However, when the monitoring system 100 is used, appropriate monitoring is automatically and continuously executed even if the user does not take measures against the change in the performance information due to the version update due to the rolling update.

The monitoring system 100 of the present embodiment can be used for an information system capable of monitoring by a baseline. In addition, the monitoring system 100 is more available for information systems that utilize cloud computing, container-type virtualization technology, or the like whose version is updated frequently.

Hereinafter, a specific example of the hardware configuration of the monitoring system 100 of the present embodiment will be described. FIG. 11 is an explanatory diagram showing a hardware configuration example of the monitoring system according to the present invention.

The monitoring system 100 shown in FIG. 11 includes a CPU 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14. Further, an input unit 15 for the user to operate and an output unit 16 for presenting the processing result or the progress of the processing content to the user may be provided.

The monitoring system 100 is realized by software when the CPU 11 shown in FIG. 11 executes a program that provides the functions of each component.

That is, each function is realized by software by the CPU 11 loading the program stored in the auxiliary storage unit 14 into the main storage unit 12 and executing the program to control the operation of the monitoring system 100.

The monitoring system 100 shown in FIG. 11 may include a DSP (Digital Signal Processor) instead of the CPU 11. Alternatively, the monitoring system 100 shown in FIG. 11 may include the CPU 11 and the DSP together.

The main storage unit 12 is used as a data work area or a data temporary storage area. The main storage unit 12 is, for example, a RAM (Random Access Memory).

The communication unit 13 has a function of inputting and outputting data to and from peripheral devices via a wired network or a wireless network (information communication network).

The auxiliary storage unit 14 is a non-temporary tangible storage medium. Examples of non-temporary tangible storage media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disk Read Only Memory), DVD-ROMs (Digital Versatile Disk Read Only Memory), and semiconductor memories.

The input unit 15 has a function of inputting data and processing instructions. The input unit 15 is an input device such as a keyboard or a mouse.

The output unit 16 has a function of outputting data. The output unit 16 is, for example, a display device such as a liquid crystal display device or a printing device such as a printer.

Further, as shown in FIG. 11, in the monitoring system 100, each component is connected to the system bus 17.

The auxiliary storage unit 14, for example, is a performance information reception unit 101, a performance information processing unit 102, a performance abnormality detection unit 103, a management information reception unit 104, a configuration change management unit 105, a group information management unit 106, a baseline generation unit 107, and The program for realizing the operation reception unit 108 is stored.

Further, the performance information storage unit 109, the group information storage unit 110, the configuration change information storage unit 111, and the baseline storage unit 112 are realized by, for example, the main storage unit 12. Further, the performance information reception unit 101, the performance abnormality detection unit 103, the management information reception unit 104, and the operation reception unit 108 are realized by, for example, the communication unit 13.

The monitoring system 100 may be realized by hardware. For example, the monitoring system 100 may be equipped with a circuit including hardware components such as an LSI (Large Scale Integration) that realizes the functions shown in FIG.

Further, a part or all of each component may be realized by a general-purpose circuit (circuitry), a dedicated circuit, a processor, or a combination thereof. These may be composed of a single chip (for example, the above LSI), or may be composed of a plurality of chips connected via a bus. A part or all of each component may be realized by a combination of the above-mentioned circuit or the like and a program.

When a part or all of each component is realized by a plurality of information processing devices and circuits, the plurality of information processing devices and circuits may be centrally arranged or distributedly arranged. For example, the information processing device, the circuit, and the like may be realized as a form in which each of the client and server system, the cloud computing system, and the like is connected via a communication network.

Next, the outline of the present invention will be described. FIG. 12 is a block diagram showing an outline of the monitoring system according to the present invention. The monitoring system 20 according to the present invention is a range (for example, a baseline) generated based on each performance value of a predetermined monitoring item of a plurality of monitored devices, and when the performance value is exceeded, the predetermined monitoring item becomes abnormal. An update unit that updates the range determined to be based on the performance values of one or more monitored devices when each version of one or more monitored devices among a plurality of monitored devices is updated. 21 (for example, baseline generation unit 107) is included.

With such a configuration, the monitoring system can save the user the trouble of resetting the threshold value for monitoring the information system whose version is updated.

Further, the monitoring system 20 uses an updated range to monitor a predetermined monitoring item of a plurality of monitored devices including one or more monitored devices for which each version has been updated (for example,). The performance abnormality detection unit 103) may be included.

With such a configuration, the monitoring system can continuously monitor the information system whose version is updated frequently without resetting the monitoring contents.

Further, the monitored device may be a container instance.

With such a configuration, the monitoring system can support a container-type virtual environment.

Further, the monitoring unit may monitor a plurality of container instances belonging to the same group using the same range.

With such a configuration, the monitoring system can accommodate container instances that are autoscaled.

Further, the update unit 21 updates the range when each version of two or more container instances among the plurality of container instances belonging to the same group is updated, and the monitoring unit updates the range by using the updated range. You may monitor each container instance.

With such a configuration, the monitoring system can monitor based on the performance information of the container instance with the newer version.

Further, the upper limit value of the range is a value obtained by adding the standard deviation of each performance value to the average value of each performance value, and the lower limit value of the range is a value obtained by subtracting the standard deviation from the average value. The range of 21 may be updated by multiplying the standard deviation by a value of 1 or more and 2 or less.

With such a configuration, the monitoring system can detect all the performance abnormalities of the container instance with the older version.

11 CPU
12 Main storage unit 13 Communication unit 14 Auxiliary storage unit 15 Input unit 16 Output unit 17 System bus 20, 100 Monitoring system 21 Update unit 101 Performance information reception unit 102 Performance information processing unit 103 Performance abnormality detection unit 104 Management information reception unit 105 Configuration Change management unit 106 Group information management unit 107 Baseline generation unit 108 Operation reception unit 109 Performance information storage unit 110 Group information storage unit 111 Configuration change information storage unit 112 Baseline storage unit 200 Terminal 300 Container management system 310 Management information collection unit 400 ~ 40N Container instance 410-41N Performance information collection unit

Claims (10)

The range generated based on each performance value of the predetermined monitoring items of the plurality of monitored devices, and the range in which the predetermined monitoring item is determined to be abnormal when the performance value is exceeded, is monitored by the plurality of monitoring items. A monitoring system including an update unit that updates each version of one or more monitored devices among the target devices based on each performance value of the one or more monitored devices. The monitoring system according to claim 1, further comprising a monitoring unit that monitors predetermined monitoring items of a plurality of monitored devices including one or more monitored devices for which each version has been updated using the updated range. The monitoring system according to claim 2, wherein the monitored device is a container instance. The monitoring system according to claim 3, wherein the monitoring unit monitors a plurality of container instances belonging to the same group using the same range. The update section updates the range when each version of two or more container instances out of multiple container instances belonging to the same group is updated.
The monitoring system according to claim 4, wherein the monitoring unit monitors each of the plurality of container instances using the updated range. The upper limit of the range is a value obtained by adding the standard deviation of each performance value to the average value of each performance value.
The lower limit of the range is a value obtained by subtracting the standard deviation from the average value.
The monitoring system according to any one of claims 1 to 5, wherein the updating unit updates the range by multiplying the standard deviation by a value of 1 or more and 2 or less. The range generated based on each performance value of the predetermined monitoring items of the plurality of monitored devices, and the range in which the predetermined monitoring item is determined to be abnormal when the performance value is exceeded, is monitored by the plurality of monitoring items. A monitoring method characterized in that when each version of one or more monitored devices among the target devices is updated, each version of the one or more monitored devices is updated based on each performance value of the one or more monitored devices. The monitoring method according to claim 7, wherein a predetermined monitoring item of a plurality of monitored devices including one or more monitored devices for which each version has been updated is monitored by using the updated range. On the computer
The range generated based on each performance value of the predetermined monitoring items of the plurality of monitored devices, and the range in which the predetermined monitoring item is determined to be abnormal when the performance value is exceeded, is monitored by the plurality of monitoring items. A monitoring program for executing an update process that updates each version of one or more monitored devices among the target devices based on the performance values of the one or more monitored devices. On the computer
The monitoring program according to claim 9, wherein a monitoring process for monitoring a predetermined monitoring item of a plurality of monitored devices including one or more monitored devices for which each version has been updated is executed by using the updated range. ..

Images