CN116820874A - Enterprise-level big data component and method for monitoring and alarming application - Google Patents
Enterprise-level big data component and method for monitoring and alarming application Download PDFInfo
- Publication number
- CN116820874A CN116820874A CN202310627769.5A CN202310627769A CN116820874A CN 116820874 A CN116820874 A CN 116820874A CN 202310627769 A CN202310627769 A CN 202310627769A CN 116820874 A CN116820874 A CN 116820874A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- exporter
- data
- consul
- enterprise
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 69
- 238000000034 method Methods 0.000 title claims abstract description 26
- 241000412611 Consul Species 0.000 claims abstract description 27
- 230000010354 integration Effects 0.000 claims abstract description 6
- 230000008569 process Effects 0.000 claims description 6
- 238000004141 dimensional analysis Methods 0.000 claims description 4
- 238000013480 data collection Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 abstract description 3
- 238000003032 molecular docking Methods 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 239000000306 component Substances 0.000 description 19
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 3
- 238000012800 visualization Methods 0.000 description 3
- 230000036541 health Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- -1 hosts Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the technical field of operation and maintenance monitoring systems, in particular to an enterprise-level big data assembly and an application monitoring and alarming method, which comprises the following steps: registering a plurality of exporters in Consul, wherein Consul is used for realizing service discovery and configuration of a distributed system; prometaus supports integration with Consul, automatically discovers targets of Exporter examples registered in Consul, and automatically reads indexes of monitoring objects; the beneficial effects are as follows: the enterprise-level big data component and the method for monitoring and alarming application provided by the invention realize the collection of various monitoring target data based on Hadoop_exporter, open-source node_ exporter, mysql _exporter and the like of a plurality of big data components such as HDFS, HBase, hive and the like which are custom developed by Prometaus docking. Exors register with condul, and promethaus periodically polls and finds the target registered with condul-related exors. According to the invention, the Hadoop_exporter and the open source Exporter are developed through access customization, and the monitoring indexes are personalized, so that the manual monitoring index intervention is effectively reduced, and the high degree of automatic monitoring is realized.
Description
Technical Field
The invention relates to the technical field of operation and maintenance monitoring systems, in particular to an enterprise-level big data assembly and an application monitoring and alarming method.
Background
The large data clusters are rapidly expanding in data size, with the consequent large-scale virtual machines and large data clusters.
In the prior art, the large data cluster generates the data magnitude scale of mass data, the use condition of a large data cluster virtual machine and the health state of a large data assembly are important concerns of an enterprise monitoring system. The automatic monitoring and alarming of modules such as big data components, hosts, core business and the like increase the daily operation and maintenance cost of enterprises, monitor the running state of the big data of the enterprises and the monitoring states of the core components and the business in real time and form a real-time alarming mechanism in time so as to form the important attention field of the enterprises.
However, the conventional enterprise monitoring system focuses on the display of static index data of a monitored target. Taking a host as an example, mainly monitoring indexes such as CPU, memory, storage, capacity and the like; taking Hdfs in Hadoop as an example, the directory number, the block number, the occupied space and the like of the Hdfs files are mainly monitored. Real-time dynamic data of a plurality of monitoring indexes of a specific component in a certain time period cannot be effectively provided; at present, a plurality of open source monitoring software exist in the market, but generally, one monitoring software is difficult to meet a complex business scene.
Disclosure of Invention
The invention aims to provide an enterprise-level big data component and an application monitoring and alarming method so as to solve the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions: an enterprise-level big data component and method of application monitoring and alerting, the method comprising the steps of:
registering a plurality of exporters in Consul, wherein Consul is used for realizing service discovery and configuration of a distributed system;
prometaus supports integration with Consul, automatically discovers targets of Exporter examples registered in Consul, and automatically reads indexes of monitoring objects;
the Prometheus monitoring data is displayed through a Grafana GUI component, and the interface of the Prometheus monitoring data realizes multi-dimensional analysis and inquiry;
the alarm function is provided by an alert manager, and the alert function realizes the push of the alarm information through a receiver such as a butt mail receiver, a WeChat receiver and the like.
Preferably, a plurality of exors are registered in Consul, each exor instance is called a target, and the monitoring data samples are periodically acquired from the targets by the Prometaus through polling, and are stored in a database, and the exors are roughly classified into 4 categories: host Exporter, base component Exporter, big data component Exporter, core service Exporter.
Preferably, prometheus periodically acquires monitoring data from the targets in a training manner, and then automatically acquires the data.
Preferably, prometheus Server is mainly responsible for data collection, storage and providing data query support to the outside; the actual collection of the monitoring sample data is completed by the Exporter, which is an independent running process, and an HTTP service for obtaining the monitoring data is exposed to the outside, and Prometheus Server only needs to obtain the monitoring data from the Exporter exposure HTTP service at regular time.
Preferably, prometheus Server end defines threshold alarm Rules for these monitor index data already stored in the local store based on PromQL by pulling the target monitor index data in condul.
Preferably, the alert manager processes the alarm packet and the delete duplicate information and sends the processed alarm packet and delete duplicate information to the correct receiver through a route; and the receiver sends the alarm message to the appointed user according to different rules according to the alarm mode, and the alert manager supports mail, weChat and SMS alarm modes.
Compared with the prior art, the invention has the beneficial effects that:
the enterprise-level big data component and the method for monitoring and alarming application provided by the invention realize the collection of various monitoring target data based on Hadoop_exporter, open-source node_ exporter, mysql _exporter and the like of a plurality of big data components such as HDFS, HBase, hive and the like which are custom developed by Prometaus docking. Exors register with condul, and promethaus periodically polls and finds the target registered with condul-related exors. According to the invention, the Hadoop_exporter and the open source Exporter are developed through access customization, and the monitoring indexes are personalized, so that the manual monitoring index intervention is effectively reduced, and the high degree of automatic monitoring is realized. The alarm information intelligent integration can be realized by setting the alarm rule of the monitoring threshold and automatically interfacing with the receiver such as WeChat, mail and the like.
Drawings
FIG. 1 is a diagram of the overall architecture of the present invention;
FIG. 2 is a Log directory structure diagram of the present invention;
fig. 3 is a flowchart of the Prometheus Server end of the present invention.
Detailed Description
In order to make the objects, technical solutions, and advantages of the present invention more apparent, the embodiments of the present invention will be further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are some, but not all, embodiments of the present invention, are intended to be illustrative only and not limiting of the embodiments of the present invention, and that all other embodiments obtained by persons of ordinary skill in the art without making any inventive effort are within the scope of the present invention.
Example 1
Referring to fig. 1 to 3, the present invention provides a technical solution: an enterprise-level big data component and method of application monitoring and alerting, the method comprising the steps of:
registering a plurality of exporters in Consul, wherein Consul is used for realizing service discovery and configuration of a distributed system;
prometaus supports integration with Consul, automatically discovers targets of Exporter examples registered in Consul, and automatically reads indexes of monitoring objects;
the Prometheus monitoring data is displayed through a Grafana GUI component, and the interface of the Prometheus monitoring data realizes multi-dimensional analysis and inquiry;
the alarm function is provided by an alert manager, and the alert function realizes the push of the alarm information through a receiver such as a butt mail receiver, a WeChat receiver and the like.
Example two
Based on the first embodiment, the specific implementation manner is as follows:
according to the method, the large data Hadoop_Exporter is developed through Prometaus docking user definition, and various monitoring target data are collected through the Exporter commonly used by an open source. The invention registers a plurality of exporters in Consul, wherein Consul is used for realizing service discovery and configuration of a distributed system. Prometaus supports integration with Consul, can automatically find the target of an Exporter instance registered in Consul, and can automatically read the index of a monitoring object. Prometheus monitoring data is displayed through a Grafana GUI component, and the interface can realize multi-dimensional analysis and inquiry. The alarm function is provided by an alert manager, and the alert function realizes the push of the alarm information through a receiver such as a butt mail receiver, a WeChat receiver and the like. The whole structure diagram is shown in detail in figure 1.
(1)Exporter
Exors are a program for providing monitoring data, which exposes an access address of http for obtaining current monitoring sample data, and each exor instance is called a target. The access Exporter comprises a Hadoop_exporter, a core service Exporter, a community open source Mysql_ Exporter, node _exporter and the like which are developed in a self-defined mode. The self-lapping hadoop_exporter directory structure includes: wherein the cmd directory is an parsing script for each component, and the common directory is a common index portion for each component. The directory of individual component names is a different field under each jmx and Log directory is the default Log path. Hadoop_exporter.py is the overall program entry. The directory structure is shown in detail in fig. 2.
Multiple exporters are registered in Consul, each Exporter instance being referred to as a target. Prometheus periodically acquires monitoring data samples from these targets by polling and stores in a database. The exporters in the present invention are broadly divided into 4 general categories: host Exporter, base component Exporter, big data component Exporter, core service Exporter.
(2)Consul
Consul is a set of open source distributed service discovery and configuration management system, and provides functions such as service registration/discovery, health check, key/Value storage, multiple data centers, distributed consistency assurance and the like. Monitoring is realized through Prometaus, when a new target needs to change a file_sd_configs configuration file on a server, the corresponding json file needs to be modified by logging in the server, and the method is complex. Prometheus authorities support a variety of automatic service discovery types, with Consul being supported. The required exor information service is registered in Consul by providing an API standard interface, and each exor instance is called a target. Prometheus periodically acquires monitoring data from the targets in a training mode, and then automatically acquires the data.
(3)Prometheus Server
Prometheus Server is primarily responsible for data collection, storage, and providing data query support to the outside. The collection of the actual monitoring sample data is completed by the Exporter. The Exporter can be a process running independently, and exposes an HTTP service for obtaining the monitoring data. Prometheus Server only needs to periodically acquire monitoring data from these Exporter exposed HTTP services.
(4)Alertmanager
Prometheus Server end defines threshold alarm Rules for these monitoring index data already stored in the local store based on PromQL by pulling the target monitoring index data in condul. Prometheus calculates alert rules periodically based on configuration parameters and if an alert condition is met, produces an alert message that is pushed to the alert manager component. The alert manager processes the information such as alarm grouping, deleting repetition and the like and sends the information to a correct receiver through a route; the receiver can send the alarm information to the appointed user according to different rules according to the alarm information, and the alert manager supports the alarm information such as mail, weChat, SMS and the like. The specific flow is shown in figure 3.
(5)Grafana
Prometheus UI provides quick verification of PromQL and temporary visualization support capability, and in most scenarios, the introduction of monitoring systems, it is often also necessary to build a monitoring data visualization panel (Dashboard) that can be used for a long period of time. Third party visualization tools Grafana were introduced that provided a powerful and elegant way to create, share, and browse data. Grafana dashboard shows that all accessed exporters register with the data in the target different monitoring data sources in the condul, is an open source visualization platform and provides complete support for Prometaus.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (6)
1. An enterprise-level big data component and an application monitoring and alarming method are characterized in that: the method comprises the following steps:
registering a plurality of exporters in Consul, wherein Consul is used for realizing service discovery and configuration of a distributed system;
prometaus supports integration with Consul, automatically discovers targets of Exporter examples registered in Consul, and automatically reads indexes of monitoring objects;
the Prometheus monitoring data is displayed through a Grafana GUI component, and the interface of the Prometheus monitoring data realizes multi-dimensional analysis and inquiry;
the alarm function is provided by an alert manager, and the alert function realizes the push of the alarm information through a receiver such as a butt mail receiver, a WeChat receiver and the like.
2. An enterprise-wide data component and application monitoring and alerting method as claimed in claim 1, wherein: registering multiple exporters in Consul, each Exporter instance being referred to as a target from which the Prometaus periodically obtains samples of monitored data by polling, and storing in a database, the exporters are broadly divided into 4 categories: host Exporter, base component Exporter, big data component Exporter, core service Exporter.
3. An enterprise-wide data component and application monitoring and alerting method as claimed in claim 2, wherein: prometheus periodically acquires monitoring data from the targets in a training mode, and then automatically acquires the data.
4. An enterprise-wide data assembly and application monitoring and alerting method as claimed in claim 3, wherein: prometheus Server is mainly responsible for data collection, storage and providing data query support to the outside; the actual collection of the monitoring sample data is completed by the Exporter, which is an independent running process, and an HTTP service for obtaining the monitoring data is exposed to the outside, and Prometheus Server only needs to obtain the monitoring data from the Exporter exposure HTTP service at regular time.
5. An enterprise-wide data component and application monitoring and alerting method as claimed in claim 1, wherein: prometheus Server end defines threshold alarm Rules for these monitoring index data already stored in the local store based on PromQL by pulling the target monitoring index data in condul.
6. An enterprise-wide data component and application monitoring and alerting method as claimed in claim 1, wherein: the alert manager processes the alarm grouping and deleting repeated information and sends the alarm grouping and deleting repeated information to a correct receiver through a route; and the receiver sends the alarm message to the appointed user according to different rules according to the alarm mode, and the alert manager supports mail, weChat and SMS alarm modes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310627769.5A CN116820874A (en) | 2023-05-29 | 2023-05-29 | Enterprise-level big data component and method for monitoring and alarming application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310627769.5A CN116820874A (en) | 2023-05-29 | 2023-05-29 | Enterprise-level big data component and method for monitoring and alarming application |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116820874A true CN116820874A (en) | 2023-09-29 |
Family
ID=88111879
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310627769.5A Pending CN116820874A (en) | 2023-05-29 | 2023-05-29 | Enterprise-level big data component and method for monitoring and alarming application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116820874A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117692164A (en) * | 2023-10-31 | 2024-03-12 | 广西壮族自治区信息中心 | Account intercommunication method based on self-research system and Grafana |
-
2023
- 2023-05-29 CN CN202310627769.5A patent/CN116820874A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117692164A (en) * | 2023-10-31 | 2024-03-12 | 广西壮族自治区信息中心 | Account intercommunication method based on self-research system and Grafana |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741089B1 (en) | Interactive location queries for raw machine data | |
| US11768811B1 (en) | Managing user data in a multitenant deployment | |
| CN110855473B (en) | Monitoring method, device, server and storage medium | |
| US11829330B2 (en) | Log data extraction from data chunks of an isolated execution environment | |
| US11226964B1 (en) | Automated generation of metrics from log data | |
| US11775501B2 (en) | Trace and span sampling and analysis for instrumented software | |
| US20190095478A1 (en) | Information technology networked entity monitoring with automatic reliability scoring | |
| US20190098106A1 (en) | Proxying hypertext transfer protocol (http) requests for microservices | |
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| US11755531B1 (en) | System and method for storage of data utilizing a persistent queue | |
| CN108958959B (en) | Method and device for detecting hive data table | |
| US11676345B1 (en) | Automated adaptive workflows in an extended reality environment | |
| CN110414259B (en) | Method and equipment for constructing data category and realizing data sharing | |
| CN110543512B (en) | Information synchronization method, device and system | |
| CN107463479A (en) | A kind of social data monitoring system | |
| US11663172B2 (en) | Cascading payload replication | |
| CN111046022A (en) | Database auditing method based on big data technology | |
| CN112052134A (en) | Service data monitoring method and device | |
| CN116820874A (en) | Enterprise-level big data component and method for monitoring and alarming application | |
| CN114328981B (en) | Knowledge graph establishing and data acquiring method and device based on mode mapping | |
| CN115333966A (en) | Nginx log analysis method, system and equipment based on topology | |
| CN113885860A (en) | Method and equipment for automatically configuring management page to generate interface service | |
| CN111338888B (en) | Data statistics method and device, electronic equipment and storage medium | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN115766527A (en) | Business analysis system and method based on API gateway inlet and outlet flow under trusted environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |