CN112380097A - Prometheus-based method for customizing monitoring index - Google Patents
Prometheus-based method for customizing monitoring index Download PDFInfo
- Publication number
- CN112380097A CN112380097A CN202011294501.7A CN202011294501A CN112380097A CN 112380097 A CN112380097 A CN 112380097A CN 202011294501 A CN202011294501 A CN 202011294501A CN 112380097 A CN112380097 A CN 112380097A
- Authority
- CN
- China
- Prior art keywords
- prometheus
- data
- directory
- data file
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 25
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000012806 monitoring device Methods 0.000 claims 1
- 238000011161 development Methods 0.000 abstract description 9
- 238000012423 maintenance Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000002354 daily effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 206010019233 Headaches Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 231100000869 headache Toxicity 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/24—Object-oriented
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/31—Programming languages or programming paradigms
- G06F8/315—Object-oriented languages
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Stored Programmes (AREA)
Abstract
The invention relates to a method for self-defining monitoring indexes based on Prometheus, which is characterized by comprising the following steps of: configuring a Prometheus. yml file on a Prometheus Server, and adding a target node into a Prometheus collection task; a node _ exporter is started on the target node and two parameters are set: list-address ═ ip: port number "used to expose an http port; -a collector. textfile. directory ═ directory path "for setting a directory of text data to be collected; and generating a data file xxx by adopting a script language and storing the data file xxx under the directory. The invention can rapidly complete the development of the user-defined monitoring index and reduce the operation and maintenance cost of the server.
Description
Technical Field
The invention relates to the technical field of server operation and maintenance, in particular to a method for customizing a monitoring index based on Prometheus.
Background
With the development of society and the improvement of technology, the data volume is increased in multiples, the demand on the server is correspondingly increased, and the daily maintenance and management of the server also become a problem that operation and maintenance personnel feel headache; meanwhile, with the advent of various application components, such as Kafka, Mysql, ES, Hbase, etc., the great application of these components in the actual production environment increases the difficulty of daily operation and maintenance; various application systems, various micro-service systems and the like are developed, so that operation and maintenance personnel need to spend a large amount of time to check various problems every day. This results in: firstly, the efficiency of processing the problem is low, and secondly, the processing after the fact causes bad influence.
With the emergence of open source Prometheus, the current situation is solved, and a server, a component, an application system and the like are monitored in a binary exporter and custom monitoring mode. The concrete characteristics are as follows:
1. multidimensional data model: time series data identified by metric name and key value pairs;
2. PromSQL: a flexible query language that can utilize multidimensional data to complete complex queries;
3. independent of distributed storage, a single server node can work directly;
4. collecting time sequence data in a pull mode based on HTTP;
5. pushing time sequence data is supported by a Pushgateway component;
6. the target may be discovered through service discovery or static configuration.
The Prometeus regularly fetches metrics data on target nodes through a Prometeus Server, each fetched target needs to expose a port of http service, namely, an exporter is deployed on each target node to expose the port to be monitored. The pull mode of pull is used by default, and the push mode can also be used, which needs to push the monitoring data on the target host into the push mode, and then prometheus pushes the data from the push mode. prometheus stores the acquired data in the TSDB, and then uses the built-in promQL to perform query. Its alarm function is provided by Alertmanager, a component of Prometheus' alarm management and sending alarms. The native diagram function of Prometoxus is too simple, Prometoxus data can be accessed to Grafana, and Grafana can perform unified management.
Although Prometheus well supports monitoring of servers, component services and the like, the requirement for custom indexes in an actual production environment is quite large, and the custom index development needs to be familiar with the corresponding development language Go, so that a large amount of learning cost is generated for developers who cannot learn the Go language, and the custom index can be developed after the developers learn the Go language. Like SQL is familiar to most developers, if the universal language or the language familiar to most people can be used for developing self-defined monitoring indexes, great convenience is brought in the development process, the development efficiency is improved, and the cost is saved.
Disclosure of Invention
The invention aims to provide a method for customizing a monitoring index based on Prometheus so as to solve the problem. Therefore, the invention adopts the following specific technical scheme:
a method for customizing a monitoring index based on Prometheus is characterized by comprising the following steps:
configuring a Prometheus. yml file on a Prometheus Server, and adding a target node into a Prometheus collection task;
a node _ exporter is started on the target node and two parameters are set: list-address ═ ip: port number "used to expose an http port; -a collector. textfile. directory ═ directory path "for setting a directory of text data to be collected;
and generating a data file xxx by adopting a script language and storing the data file xxx under the directory.
Further, the scripting language may include Java, Python, Shell, or the like.
Further, the data form of the data file is in the format of key-value, and value can only be a numeric type.
Furthermore, instance, joba and other self-defined tags are specified in the data file, and multiple tags can be added to the same index to distinguish values of different dimensions.
Further, in the case of multiple device IPs, only data with the same index and different device IPs can be written into one data file, but the prometheus monitors all devices, otherwise, when data is collected, prometheus finds that the indexes in the two files are the same, and overwrites the previous data with the last data.
By adopting the technical scheme, the invention has the beneficial effects that: the invention can rapidly complete the development of the user-defined monitoring index and reduce the operation and maintenance cost of the server.
Drawings
To further illustrate the various embodiments, the invention provides the accompanying drawings. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the embodiments. Those skilled in the art will appreciate still other possible embodiments and advantages of the present invention with reference to these figures. Elements in the figures are not drawn to scale and like reference numerals are generally used to indicate like elements.
FIG. 1 is a schematic diagram of the method of the present invention.
Detailed Description
The invention will now be further described with reference to the accompanying drawings and detailed description.
The invention mainly aims to realize the customization of personalized monitoring indexes according to business requirements when Prometheus is used for monitoring, operating and maintaining host equipment, services and the like. With the development and progress of science and technology, people's demands for services tend to be more granular, and pay more attention to the subtle changes to restore the essence of events. Taking a server as an example, the traffic of some general indexes such as CPU, memory, disk, network card, etc. cannot meet the monitoring of the server, and people prefer to realize the monitoring in different directions according to the functions provided by different servers, such as storage, calculation, backup, etc., and according to the current situation, only the customized and customizable indexes can meet the business requirements of people in different scenes.
The method mainly utilizes the capability of collecting file data provided by the node _ exporter to specify a path of- -collector. textfile.direction: "when the node _ exporter is started, and the Prometheus can regularly collect the file with the file name xxx.prom in the directory, so that the self-defined monitoring index and data can be written into the time sequence database of the Prometheus for the query of API or PromQL.
Specifically, an architecture diagram of the customized monitoring index of the present invention is shown in fig. 1. The process of realizing the user-defined monitoring index is as follows:
configuring a prometheus. yml file in a prometheus Server, and adding a target node into a prometheus collection task;
a node _ exporter is started on the target node and two parameters are set: list-address "ip: port number" (e.g. -web-address "ip:9100") for exposing an http port (9100), and a collector.
The data file is generated, and the data acquisition can be completed by using familiar languages, such as Java, Python, Shell and the like. The data writing method includes the following steps:
1) defaulting to instance and job of the node, the form of data is in the format of key-value, and value can only be a numerical type: the node _ cpu _ metric 24, when prometheus is fetching data, will add the ip of the target host and the configured jobname as the label of the pointer by default, that is, the data displayed when using PromQL to query the pointer is: node _ cpu _ metric { instance ═ node: 9100", job ═ node" } 24.
2) And (3) specifying instance, joband other self-defined tags, wherein multiple tags can be added to the same index to distinguish values of different dimensions. Taking the port state of pos device as an example: pos _ status { instance ═ nodeip1:9200 ″, job ═ pos ═ ifName ═ NF-00-CJ } 1. The example IP for which the pointer is specified here is nodeip1, i.e., the IP of the real pos device, rather than the local IP that would otherwise obtain the pointer for the pos device. The data written in this way will show, if queried by the PromQL: pos _ status { instance ═ nodeip:9100 ', jobi ═ node ', export _ instance ═ nodeip1:9200 ', export _ jobb ═ pos ', and ifName } NF-00-CJ '. The result of this is that, as mentioned above, at the time of collecting data, prometheus adds two labels by default, instance is ip of the target node and job is job _ name configured by prometheus. If the prometheus finds the key with the label instance and job in the pull data, it renames the exported _ instance and exported _ job. If you do not need to default the added tag, but need the written instance and job to overwrite the original value, only need to add the configured honor labels under the job _ name in the prometheus. yml file.
3) Data from different data sources with the same index. For example, we can only communicate with all pos devices through one nodeip node, and promemeus can only pull the data of the nodeip node. In this case, if pos devices are to be monitored, data of all pos devices needs to be acquired on the nodeip node, and then the pull data can be timed by prometheus. In the case of multiple pois, only data with the same index and different pois can be written into a file, and prometheus can monitor all pos devices. Because the data is characterized by the format of key-value, if it is divided into multiple files, prometheus finds that there are multiple identical key values when pulling data, and it overwrites the key value read first with the key value read last. For example: the total data packet number index of two pos machines can be monitored through the proxy only if two pieces of data, namely pos _ output _ total _ packet { jobis ═ pos, 'instance ═ ip1: 9100' }100pos _ output _ total _ packet { jobis ═ pos, 'instance ═ ip2: 9100' }100, exist in one file.
The invention mainly solves the problem that when prometheus is used for monitoring, the user-defined monitoring index can be simply and quickly realized to complete the actual service requirement. Due to flexibility and variability of services in an actual production environment, index data acquired by an exporter provided by an official website cannot meet user requirements, but if you need to develop a new index based on a source code, the expert needs to master Go language and compiling process, which increases learning cost virtually, and fails to provide support for urgent needs. The invention well solves the problem, and as long as you can use prometheus, familiar shell language or any other language, if the index value can be obtained, and then the data is written into the file, the development of the self-defined monitoring index can be completed quickly. In the future, different business requirements are more complex, personalized requirement customization is also the mainstream, and the customized monitoring index is the preferred mode for realizing personalized monitoring.
While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (5)
1. A method for customizing a monitoring index based on Prometheus is characterized by comprising the following steps:
configuring a Prometheus. yml file on a Prometheus Server, and adding a target node into a Prometheus collection task;
a node _ exporter is started on the target node and two parameters are set: list-address ═ ip: port number "used to expose an http port; -a collector. textfile. directory ═ directory path "for setting a directory of text data to be collected;
and generating a data file xxx by adopting a script language and storing the data file xxx under the directory.
2. The method of claim 1, wherein the scripting language comprises Java, Python, or Shell.
3. The method of claim 1, wherein the data form of the data file is in a key-value format, and the value is only a numeric type.
4. A method as claimed in claim 3, characterized in that in the case of a plurality of device IPs, only data of the same index and different device IPs can be written into one data file, but the prometheus monitors all monitoring devices, otherwise when collecting data, prometheus finds the same index in both files, and overwrites the previous data with the last data.
5. The method of claim 3, wherein instance, job and other custom tags are specified in a data file, and the same index can add multiple tags to distinguish values of different dimensions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011294501.7A CN112380097A (en) | 2020-11-18 | 2020-11-18 | Prometheus-based method for customizing monitoring index |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011294501.7A CN112380097A (en) | 2020-11-18 | 2020-11-18 | Prometheus-based method for customizing monitoring index |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112380097A true CN112380097A (en) | 2021-02-19 |
Family
ID=74584252
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011294501.7A Pending CN112380097A (en) | 2020-11-18 | 2020-11-18 | Prometheus-based method for customizing monitoring index |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380097A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114723072A (en) * | 2022-05-05 | 2022-07-08 | 中国电信股份有限公司 | Exporter combination method, system, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920283A (en) * | 2018-08-06 | 2018-11-30 | 成都知道创宇信息技术有限公司 | Server guard method based on Prometheus performance monitoring system |
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN110347377A (en) * | 2019-07-08 | 2019-10-18 | 紫光云技术有限公司 | A kind of Prometheus exporter database monitoring system |
| WO2019227689A1 (en) * | 2018-06-01 | 2019-12-05 | 平安科技（深圳）有限公司 | Data monitoring method and apparatus, and computer device and storage medium |
-
2020
- 2020-11-18 CN CN202011294501.7A patent/CN112380097A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019227689A1 (en) * | 2018-06-01 | 2019-12-05 | 平安科技（深圳）有限公司 | Data monitoring method and apparatus, and computer device and storage medium |
| CN108920283A (en) * | 2018-08-06 | 2018-11-30 | 成都知道创宇信息技术有限公司 | Server guard method based on Prometheus performance monitoring system |
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN110347377A (en) * | 2019-07-08 | 2019-10-18 | 紫光云技术有限公司 | A kind of Prometheus exporter database monitoring system |
Non-Patent Citations (2)
| Title |
|---|
| 极客分享: ""【Prometheus】 node_exporter的常用配置项详解", 《极客分享》 * |
| 赣江: "使用prometheus监控node_exporter告警案例", 《程序宅基地》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114723072A (en) * | 2022-05-05 | 2022-07-08 | 中国电信股份有限公司 | Exporter combination method, system, device and storage medium |
| CN114723072B (en) * | 2022-05-05 | 2023-10-03 | 中国电信股份有限公司 | Exporter combination method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101862235B1 (en) | Specifying user interface elements | |
| CN106067080B (en) | Configurable workflow capabilities are provided | |
| CN100565510C (en) | Data access layer class generator | |
| US20040268306A1 (en) | Methods, systems and computer program products for language independent data communication and display | |
| CN111427748B (en) | Task alarm method, system, equipment and storage medium | |
| US20070234306A1 (en) | Tracing performance of machine-readable instructions | |
| US10261877B2 (en) | Systems and methods for testing mobile devices | |
| CN109614088B (en) | Form component generation method and device | |
| CN105512261A (en) | Method and system for expressing front end lightweight statistical data | |
| Cejka et al. | Java embedded storage for time series and meta data in Smart Grids | |
| CN107391528B (en) | Front-end component dependent information searching method and equipment | |
| US7051015B1 (en) | System and method for implementing a flexible data-driven target object model | |
| CN112380097A (en) | Prometheus-based method for customizing monitoring index | |
| Aydt | The Pablo Self-De ning Data Format," | |
| CN112732663A (en) | Log information processing method and device | |
| CN115567526B (en) | Data monitoring method, device, equipment and medium | |
| CN112579406A (en) | Log call chain generation method and device | |
| CN115269228A (en) | Data adaptive transmission method, device, equipment and medium | |
| CN114125073B (en) | Data interaction method, storage medium and electronic device | |
| CN108628909A (en) | Information-pushing method and device | |
| CN115390912B (en) | Resource discovery method, device, computer equipment and storage medium | |
| CN117056175B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN114745448B (en) | Multi-channel protocol adaptation processing method, system and device | |
| Fontenay | sdmxuse: command to import data from statistical agencies using the SDMX standard | |
| Krishnan et al. | Resource Forecast and Ramp Visualization for Situational Awareness (RAVIS) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210219 |
|
| RJ01 | Rejection of invention patent application after publication |