CN112925612A - Monitoring service static configuration management method based on Kubernetes - Google Patents
Monitoring service static configuration management method based on Kubernetes Download PDFInfo
- Publication number
- CN112925612A CN112925612A CN202110274812.5A CN202110274812A CN112925612A CN 112925612 A CN112925612 A CN 112925612A CN 202110274812 A CN202110274812 A CN 202110274812A CN 112925612 A CN112925612 A CN 112925612A
- Authority
- CN
- China
- Prior art keywords
- configuration
- control plane
- static
- monitoring
- kubernetes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45587—Isolation or security of virtual machine instances
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a monitoring service static configuration management method based on Kubernetes, which relates to the technical field of container monitoring and comprises the following steps: deploying a Prometheus server in a kubernets cluster; writing a static monitoring target configuration template and an alarm rule configuration template in a control plane; the control plane calls an interface to obtain the contents of the current Secret and the ConfigMap and the change information of the monitoring target; the control plane generates a new static monitoring target configuration file according to the acquired information; the control plane backups the content of the current Secret and calls an interface to update the new configuration file into the Secret; the control plane calls an interface to verify the updated content of the Secret, modifies the record of the configuration version table according to the verification result, and sends an email to inform operation and maintenance personnel or execute the next step; and the control plane generates new alarm rule configuration based on the monitoring information, calls an interface and confirms that the alarm rule configuration is updated successfully. The invention can effectively reduce the manual maintenance work of the monitoring service in the Kubernetes cluster and improve the efficiency.
Description
Technical Field
The invention relates to the technical field of container monitoring, in particular to a monitoring service static configuration management method based on Kubernetes.
Background
The container arrangement system Kubernets is widely applied to various cloud computing services, and more applications run on cloud nodes in a container form. As a set of complete monitoring service, Prometheus is provided with a time sequence database, supports remote storage, realizes flexible query language PromQL, has the advantages of rich monitoring components, good performance, active communities and the like, and becomes the first choice for monitoring Kubernets cluster containers. Prometheus provides good monitoring and alarm service for kubernets clusters and applications deployed within the clusters.
On one hand, kubernets provide a convenient and fast capacity expansion and reduction function for applications deployed in a cluster, and kubernets also have capacity expansion and reduction operations for cluster nodes, so that monitoring service promemeus is required to update configuration files in time to adapt to changes of monitoring targets. In Prometous configuration, the configuration content related to the monitoring target includes a static monitoring template and a dynamic monitoring target based on service discovery, which are different in specific change operation. Prometheus supports a variety of service discovery mechanisms, service discovery-based configuration can adaptively acquire and update monitoring targets according to screening and filtering conditions, static configuration requires manual modification of configuration files, and configuration entries may include site URLs, IP addresses, file paths, and the like.
On the other hand, when Prometheus is deployed in a kubernets cluster, the related configuration file is generally stored in ConfigMap and Secret resources of kubernets, and then configuration contents in these resources are written into a file in a container through a mount operation. The configuration file includes monitoring target configuration and alarm triggering rule configuration, and generally, for different types of monitoring targets, the rule configuration is distributed in different files in a Prometheus server container. In addition, with the continuous development and growth of various applications, the monitoring targets of promemeus are more and more, and may include physical machines, virtual machines, containers, networks, middleware services, user applications, and the like, while in a production environment, stable operation of a monitoring service is critical to the service, in order to ensure high stability of the monitoring service, a deployment architecture of the promemeus must be highly available, the promemeus server is deployed in a manner of multiple copies, and when the monitoring data volume is large, data needs to be fragmented to eliminate a load bottleneck of a single promemeus server. The deployment architecture greatly increases the complexity of Prometheus configuration maintenance, but for static configuration contents, errors are often introduced by manual maintenance, the manual maintenance efficiency is low, operation logs of operation and maintenance personnel are inconvenient to record, and the sustainable development of service management and application services is not facilitated.
Disclosure of Invention
Aiming at the requirements and the defects of the prior art development, the invention provides a monitoring service static configuration management method based on Kubernetes.
The invention discloses a monitoring service static configuration management method based on Kubernetes, which adopts the following technical scheme for solving the technical problems:
a monitoring service static configuration management method based on Kubernetes comprises the following implementation processes:
step S1, deploying Prometheus server in a Stateful set mode in a Kubernets cluster, and deploying config-reload containers to the Pod of the Prometheus server in a sidecar container mode;
step S2, writing a static monitoring target configuration template and an alarm rule configuration template in a control plane;
step S3, the control plane calls a Kubernetes APIServer interface, obtains the contents of Secret and ConfigMap in use, stores the contents into a configuration version table, marks the version number and the state, finds a system through polling or monitoring service, obtains the change information of a monitoring target, and writes the change information into a database;
step S4, the control plane generates a new static monitoring target configuration file according to the acquired monitoring target information;
step S5, the control plane backups the current Secret content, and calls Kubernetes APIServer interface to update the new static monitoring target configuration file to the Secret;
step S6, the control plane calls a Prometous configuration check interface to check the updated content of the Secret, modifies the record of the configuration version table according to the check result, and sends an email to inform operation and maintenance personnel or executes step S7 to load a new configuration file;
step S7, the control plane acquires the monitoring alarm information configured by the operation and maintenance personnel, and checks the configuration value by combining the alarm rule template file to generate a new alarm rule configuration, and the new alarm rule configuration is mounted in the configuration directory of the Prometeus server container;
and step S8, the control plane calls Prometous to configure a hot loading interface, and the alarm rule configuration is confirmed to be updated successfully.
Further, when step S1 is executed, it is further necessary to plan a selector rule according to actual requirements, deploy a common node level exporter in a DaemonSet manner, and deploy a common application level exporter in a SidecarSet manner.
Further, after step S1 is executed, the ConfigMap is used to store the monitoring alarm rule configuration file, the Secret is used to store the additional static monitoring target configuration file, and the static monitoring target configuration file is mounted to the corresponding Pod, and then step S2 is executed.
Further, when step S2 is executed, the static monitoring target configuration template takes job in Prometheus master configuration as a unit, the alarm rule configuration template corresponds to the static monitoring target configuration template, and the content of the alarm rule configuration template is stored in the control plane in a yaml format.
Further, when step S6 is executed to verify the updated content of Secret:
(a) if the verification fails, acquiring failure information and updating the failure information into an operation log, rolling back the Secret/ConfigMap by using the backup file in the step 5, modifying the record of the configuration version table, sending an email to inform operation and maintenance personnel, and ending the process;
(b) if the verification is successful, calling a Prometheus configuration hot loading interface, confirming that the new configuration mounted in the Prometheus server container through the Secret is correct and immediately effective, modifying the record of the configuration version table, and executing step S7 to load the new configuration file.
Further, in step S7, the new alarm rule configuration is mounted in the configuration directory of the prometheuser container by the ConfigMap.
Further, after step S7 is executed, the control plane calls a kubernets APIServer interface, first backups the content of the current ConfigMap, and then stores the file content as key and value in the ConfigMap.
Preferably, when step S3 is executed, the status recorded in the current configuration version table is modified to be non-modifiable through the control plane.
Compared with the prior art, the monitoring service static configuration management method based on Kubernetes has the following beneficial effects:
the invention adopts the Prometheus monitoring service, utilizes the diversified container deployment and application configuration mounting modes of Kubernetes, combines with the service discovery system, carries out the generation and distribution operation of Prometheus static configuration through a control plane, and utilizes the configuration verification and hot loading interface provided by Prometheus to ensure the reliability of the automatic configuration management function, thereby effectively reducing the manual maintenance work of the monitoring service in the Kubernetes cluster, improving the efficiency and ensuring the stability of the monitoring service.
Drawings
FIG. 1 is a schematic diagram of a method implementation of the present invention;
FIG. 2 is a flow chart of the method of the present invention.
Detailed Description
In order to make the technical scheme, the technical problems to be solved and the technical effects of the present invention more clearly apparent, the following technical scheme of the present invention is clearly and completely described with reference to the specific embodiments.
The first embodiment is as follows:
with reference to fig. 1 and 2, the present embodiment provides a monitoring service static configuration management method based on Kubernetes, and the implementation process includes:
step S1, deploying Prometheus server in a Stateful set mode in a Kubernets cluster, and deploying config-reload containers to the Pod of the Prometheus server in a sidecar container mode; meanwhile, according to actual requirements, a selector rule is planned, a universal node level exporter is deployed in a DaemonSet mode, and a universal application level exporter is deployed in a SidecSet mode.
Using ConfigMap to store the configuration file of the monitoring alarm rule, using Secret to store the additional configuration file of the static monitoring target, and mounting the configuration file into the corresponding Pod,
and step S2, writing a static monitoring target configuration template and an alarm rule configuration template in the control plane. The static monitoring target configuration template takes jobs in Prometous main configuration as a unit, the alarm rule configuration template corresponds to the static monitoring target configuration template, and the content of the alarm rule configuration template adopts a yaml format and is stored in the control plane.
Step S3, the control plane calls the Kubernetes APIServer interface, obtains the contents of Secret and ConfigMap in use, stores the contents into the configuration version table, marks the version number and the state, finds the system through polling or monitoring service, obtains the change information of the monitored target, and writes the change information into the database.
And step S4, modifying the state recorded in the current configuration version table in the step S3 by the control plane to be not modifiable, and generating a new static monitoring target configuration file according to the acquired monitoring target information.
Step S5, the control plane backs up the current Secret content and calls kubernets APIServer interface to update the new static monitoring target configuration file into the Secret.
Step S6, the control plane calls Prometheus configuration check interface to check the updated content of Secret:
(a) if the verification fails, acquiring failure information and updating the failure information into an operation log, rolling back the Secret/ConfigMap by using the backup file in the step 5, modifying the record of the configuration version table, sending an email to inform operation and maintenance personnel, and ending the process;
(b) if the verification is successful, calling a Prometheus configuration hot loading interface, confirming that the new configuration mounted in the Prometheus server container through the Secret is correct and immediately effective, modifying the record of the configuration version table, and executing step S7 to load the new configuration file.
Step S7, the control plane obtains the monitoring alarm information configured by the operation and maintenance personnel, and checks the configuration value by combining the alarm rule template file to generate a new alarm rule configuration, and the new alarm rule configuration is mounted to the configuration directory of the Prometheus server container through ConfigMap.
The control plane calls a Kubernetes APIServer interface, firstly backups the content of the current ConfigMap, and then stores the file content into the ConfigMap as key and value by taking the name of the rule configuration file as key.
And step S8, the control plane calls Prometous to configure a hot loading interface, and the alarm rule configuration is confirmed to be updated successfully.
In summary, the monitoring service static configuration management method based on Kubernetes solves the problem that the static configuration of the monitoring service promemeus needs manual maintenance, realizes the function of self-adaptively updating the promemeus static monitoring target and the alarm rule configuration thereof by combining a control plane with a service discovery mechanism, and ensures the reliability of automatic updating configuration operation by utilizing the configuration verification and hot loading functions of the promemeus.
The principles and embodiments of the present invention have been described in detail using specific examples, which are provided only to aid in understanding the core technical content of the present invention. Based on the above embodiments of the present invention, those skilled in the art should make any improvements and modifications to the present invention without departing from the principle of the present invention, and therefore, the present invention should fall into the protection scope of the present invention.
Claims (8)
1. A monitoring service static configuration management method based on Kubernetes is characterized in that the implementation process comprises the following steps:
step S1, deploying Prometheus server in a Stateful set mode in a Kubernets cluster, and deploying config-reload containers to the Pod of the Prometheus server in a sidecar container mode;
step S2, writing a static monitoring target configuration template and an alarm rule configuration template in a control plane;
step S3, the control plane calls a Kubernetes APIServer interface, obtains the contents of Secret and ConfigMap in use, stores the contents into a configuration version table, marks the version number and the state, finds a system through polling or monitoring service, obtains the change information of a monitoring target, and writes the change information into a database;
step S4, the control plane generates a new static monitoring target configuration file according to the acquired monitoring target information;
step S5, the control plane backups the current Secret content, and calls Kubernetes APIServer interface to update the new static monitoring target configuration file to the Secret;
step S6, the control plane calls a Prometous configuration check interface to check the updated content of the Secret, modifies the record of the configuration version table according to the check result, and sends an email to inform operation and maintenance personnel or executes step S7 to load a new configuration file;
step S7, the control plane acquires the monitoring alarm information configured by the operation and maintenance personnel, and checks the configuration value by combining the alarm rule template file to generate a new alarm rule configuration, and the new alarm rule configuration is mounted in the configuration directory of the Prometeus server container;
and step S8, the control plane calls Prometous to configure a hot loading interface, and the alarm rule configuration is confirmed to be updated successfully.
2. The monitoring service static configuration management method based on Kubernetes as claimed in claim 1, wherein when step S1 is executed, it is further required to plan a selector rule according to an actual requirement, deploy a common node level exporter in a DaemonSet manner, and deploy a common application level exporter in a SidecarSet manner.
3. The monitoring service static configuration management method based on Kubernetes as claimed in claim 1, wherein after step S1 is executed, a configuration file of monitoring alarm rules is stored using ConfigMap, an additional configuration file of static monitoring target is stored using Secret and mounted to a corresponding Pod, and then step S2 is executed.
4. The monitoring service static configuration management method based on Kubernetes as claimed in claim 2, wherein in step S2, the static monitoring target configuration template takes job in Prometheus main configuration as a unit, the alarm rule configuration template corresponds to the static monitoring target configuration template, and the content of the alarm rule configuration template is stored in the control plane in yaml format.
5. The monitoring service static configuration management method based on Kubernetes as claimed in claim 1, wherein when step S6 is executed, and the updated content of Secret is checked:
(a) if the verification fails, acquiring failure information and updating the failure information into an operation log, rolling back the Secret/ConfigMap by using the backup file in the step 5, modifying the record of the configuration version table, sending an email to inform operation and maintenance personnel, and ending the process;
(b) if the verification is successful, calling a Prometheus configuration hot loading interface, confirming that the new configuration mounted in the Prometheus server container through the Secret is correct and immediately effective, modifying the record of the configuration version table, and executing step S7 to load the new configuration file.
6. The monitoring service static configuration management method based on Kubernetes as claimed in claim 5, wherein in step S7, the new alarm rule configuration is installed under the configuration directory of the Prometheus server container through ConfigMap.
7. The monitoring service static configuration management method based on kubernets according to claim 6, wherein after step S7 is executed, the control plane calls a kubernets APIServer interface to first backup the current ConfigMap content, and then stores the file content as key and value into the ConfigMap with the rule configuration file name as key.
8. The monitoring service static configuration management method based on Kubernetes as claimed in claim 1, wherein the status recorded in the current configuration version table in step S3 is modified to be non-modifiable through the control plane.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110274812.5A CN112925612A (en) | 2021-03-15 | 2021-03-15 | Monitoring service static configuration management method based on Kubernetes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110274812.5A CN112925612A (en) | 2021-03-15 | 2021-03-15 | Monitoring service static configuration management method based on Kubernetes |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112925612A true CN112925612A (en) | 2021-06-08 |
Family
ID=76174959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110274812.5A Pending CN112925612A (en) | 2021-03-15 | 2021-03-15 | Monitoring service static configuration management method based on Kubernetes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112925612A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115733732A (en) * | 2022-11-18 | 2023-03-03 | 新浪技术（中国）有限公司 | Prometheus-based alarm method, device and system |
| CN116467139A (en) * | 2023-03-27 | 2023-07-21 | 深圳市明源云科技有限公司 | System alarm repetition rate detection method, electronic equipment and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666189A (en) * | 2020-06-12 | 2020-09-15 | 中信银行股份有限公司 | Method and system for declaratively visually configuring Prometheus monitoring alarm |
| CN112256401A (en) * | 2020-10-30 | 2021-01-22 | 浪潮云信息技术股份公司 | Prometheus high-availability system based on Kubernetes environment and implementation method |
| CN112328456A (en) * | 2021-01-04 | 2021-02-05 | 北京电信易通信息技术股份有限公司 | Cluster resource monitoring system based on service discovery |
-
2021
- 2021-03-15 CN CN202110274812.5A patent/CN112925612A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666189A (en) * | 2020-06-12 | 2020-09-15 | 中信银行股份有限公司 | Method and system for declaratively visually configuring Prometheus monitoring alarm |
| CN112256401A (en) * | 2020-10-30 | 2021-01-22 | 浪潮云信息技术股份公司 | Prometheus high-availability system based on Kubernetes environment and implementation method |
| CN112328456A (en) * | 2021-01-04 | 2021-02-05 | 北京电信易通信息技术股份有限公司 | Cluster resource monitoring system based on service discovery |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115733732A (en) * | 2022-11-18 | 2023-03-03 | 新浪技术（中国）有限公司 | Prometheus-based alarm method, device and system |
| CN116467139A (en) * | 2023-03-27 | 2023-07-21 | 深圳市明源云科技有限公司 | System alarm repetition rate detection method, electronic equipment and readable storage medium |
| CN116467139B (en) * | 2023-03-27 | 2024-06-07 | 深圳市明源云科技有限公司 | System alarm repetition rate detection method, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107515776B (en) | Method for upgrading service continuously, node to be upgraded and readable storage medium | |
| US10642694B2 (en) | Monitoring containers in a distributed computing system | |
| CN101964820B (en) | Method and system for keeping data consistency | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| CN110012088B (en) | Kubernets-based Redis master-slave cluster automatic deployment scheme | |
| US20100241895A1 (en) | Method and apparatus for realizing application high availability | |
| CN109766330B (en) | Data slicing method and device, electronic equipment and storage medium | |
| CN111045708B (en) | Software upgrading method, electronic device and computer readable storage medium | |
| CN112003728B (en) | Kubernetes cluster-based application master and standby implementation method and device | |
| US8612799B2 (en) | Method and apparatus of backing up subversion repository | |
| CN112925612A (en) | Monitoring service static configuration management method based on Kubernetes | |
| CN113626286A (en) | Multi-cluster instance processing method and device, electronic equipment and storage medium | |
| CN114138754A (en) | Software deployment method and device based on Kubernetes platform | |
| WO2011072716A1 (en) | A method of updating versioned software using a shared cache | |
| CN105740049A (en) | Control method and apparatus | |
| CN116760913B (en) | Method and system for issuing k8s cluster protocol conversion platform configuration | |
| CN112181049B (en) | Cluster time synchronization method, device, system, equipment and readable storage medium | |
| CN105323271B (en) | Cloud computing system and processing method and device thereof | |
| CN112230978A (en) | Multi-data-source dynamic switching method, electronic equipment and storage medium | |
| CN111756562B (en) | Cluster takeover method, system and related components | |
| CN118034804A (en) | High availability method and system based on source generation redis | |
| CN116483473A (en) | Storage plug-in method and system based on GlusterFS | |
| CN113934575A (en) | Big data backup system and method based on distributed copy | |
| CN116308639A (en) | Design method for high concurrency order ID under distributed architecture | |
| CN118035199A (en) | Nfs server control method and device for preventing read-write abnormality during high availability switching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210608 |