CN112069049A - Data monitoring management method and device, server and readable storage medium - Google Patents
Data monitoring management method and device, server and readable storage medium Download PDFInfo
- Publication number
- CN112069049A CN112069049A CN202010943918.5A CN202010943918A CN112069049A CN 112069049 A CN112069049 A CN 112069049A CN 202010943918 A CN202010943918 A CN 202010943918A CN 112069049 A CN112069049 A CN 112069049A
- Authority
- CN
- China
- Prior art keywords
- data
- tool
- processing
- source
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 131
- 238000007726 management method Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 122
- 238000000034 method Methods 0.000 claims abstract description 65
- 230000008569 process Effects 0.000 claims abstract description 35
- 230000002159 abnormal effect Effects 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 16
- 238000001914 filtration Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 230000004907 flux Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000004044 response Effects 0.000 description 6
- 230000006399 behavior Effects 0.000 description 5
- 238000013480 data collection Methods 0.000 description 4
- 238000012550 audit Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 239000000523 sample Substances 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 230000001502 supplementing effect Effects 0.000 description 2
- 241000219780 Pueraria Species 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3068—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data format conversion
Abstract
The application provides a data monitoring management method, a data monitoring management device, a server and a readable storage medium, and relates to the technical field of computer data monitoring. The method comprises the following steps: acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source; converting the data in the data set into data in a specified format; according to the processing strategy corresponding to the data type of the data in the specified format, the data in the specified format in the data set is analyzed and processed through the big data processing tool to obtain a processing result, and the problems that the monitored data source is single in type and the monitored data index is limited can be solved.
Description
Technical Field
The invention relates to the technical field of computer data monitoring, in particular to a data monitoring management method, a data monitoring management device, a server and a readable storage medium.
Background
With the technological progress, a plurality of new platforms and systems are gradually put into use, so that the comprehensive service capacity and the operation efficiency of the system are improved. During the operation of the platform and the system, the platform and the system are generally required to be monitored and managed. Currently, Zabbix software tools are generally adopted to collect corresponding data of a system and a platform for monitoring. Zabbix is open source monitoring software which can be used for monitoring application programs, the type of the monitored data source is single, and the monitoring index is limited, thereby influencing the monitoring management of the system.
Disclosure of Invention
The application provides a data monitoring management method, a data monitoring management device, a server and a readable storage medium, which can solve the problems of single type of monitored data source and limited monitoring index.
In order to achieve the above purpose, the technical solutions provided in the embodiments of the present application are as follows:
in a first aspect, an embodiment of the present application provides a data monitoring management method, where the method includes:
acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source;
converting the data in the data set into data in a specified format;
and analyzing and processing the data in the specified format in the data set by a big data processing tool according to a processing strategy corresponding to the data type of the data in the specified format to obtain a processing result.
In the above embodiment, by deploying a tool set corresponding to the monitoring data source, and then using the tool set, data can be acquired from at least two data sources of the front-end data source, the application data source, the infrastructure data source, and the business process data source, and then the acquired data is analyzed and processed, so that the problems of single type of the monitored data source and limited monitoring index can be solved.
With reference to the first aspect, in some optional embodiments, acquiring, by a tool set corresponding to a monitoring data source, a data set of the corresponding monitoring data source includes:
acquiring a first data set of the front-end data source through a first tool set corresponding to the front-end data source;
acquiring a second data set of the application data source through a second tool set corresponding to the application data source;
obtaining a third data set of the infrastructure data source through a third tool set corresponding to the infrastructure data source;
and acquiring a fourth data set of the business process data source through a fourth tool set corresponding to the business process data source.
In the foregoing embodiment, through the first tool set, the second tool set, the third tool set, and the fourth tool set, corresponding data can be acquired from the front-end data source, the application data source, the infrastructure data source, and the business process data source, respectively, so as to improve the problems of single type of the monitored data source and limited monitoring index.
With reference to the first aspect, in some optional embodiments, the first set of tools includes at least one of a Piwik tool for collecting statistics of a website, a collection tool for collecting user operation data, a first application performance management tool for collecting performance data;
the second toolset comprises at least one of a second application performance management tool for collecting performance data, a first Zabbix tool for collecting middleware data, a flux/filebed tool for collecting logs;
the third toolset comprises at least one of a second Zabbix tool for collecting server/database related data, a promemeus tool for collecting container data;
the fourth set of tools includes API tools for collecting business data.
In the above embodiment, the Piwik tool, the operation data collection tool, the Flume/FileBeat tool, the Prometheus tool, etc. can be used to collect different types of data from multiple types of data sources, so as to solve the problems of single collected data source, limited monitoring index, and limited collected data types.
With reference to the first aspect, in some optional implementations, according to a processing policy corresponding to a data category of the data in the specified format, performing analysis processing on the data in the specified format in the data set by using a big data processing tool to obtain a processing result, where the method includes:
when the processing strategy is a filtering strategy, filtering the data in the specified format in the data set through the big data processing tool to obtain filtered data;
when the processing strategy is a classification strategy, storing the data in the specified format in a storage area corresponding to the data type and the source identification through the big data processing tool;
and when the processing strategy is a statistical strategy, counting the quantity of the data of the specified data category in a preset time period by the big data processing tool.
In the above embodiment, the data sets are processed and analyzed, so that the processed and received data can be conveniently output and displayed.
With reference to the first aspect, in some optional embodiments, the method further comprises:
and when the processing result comprises a result of the abnormal data, sending an alarm prompt to a monitoring data source or a user terminal corresponding to the abnormal data.
In the above embodiment, the alarm prompt is sent to the monitoring data source or the user terminal corresponding to the abnormal data, so that the position of the abnormal data is conveniently and quickly located, the abnormal reason can be conveniently and timely checked, and the service loss is reduced.
With reference to the first aspect, in some optional embodiments, the method further comprises:
and sending the processing result to a user terminal through a message queue, wherein the user terminal is used for displaying the processing result according to a preset format, and the preset format comprises at least one of a statistical chart and a table.
In the above embodiment, the processing result is sent in a message queue manner, which is beneficial to orderly outputting and displaying the processing result.
With reference to the first aspect, in some optional embodiments, converting the data in the data set into data in a specific format includes:
and converting the data in the data set into data in a JSON format.
In the foregoing embodiment, the JSON format is a lightweight data format, and data in the data set is converted into the JSON format, which is beneficial to subsequent storage and analysis of the data.
In a second aspect, an embodiment of the present application further provides a data monitoring management apparatus, where the apparatus includes:
the data acquisition unit is used for acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source;
the conversion unit is used for converting the data in the data set into data in a specified format;
and the analysis processing unit is used for analyzing and processing the data in the specified format in the data set through a big data processing tool according to a processing strategy corresponding to the data type of the data in the specified format to obtain a processing result.
In a third aspect, an embodiment of the present application further provides a server, where the server includes a memory and a processor coupled to each other, where the memory stores a computer program, and when the computer program is executed by the processor, the server is caused to perform the method described above.
In a fourth aspect, the present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program runs on a computer, the computer is caused to execute the above method.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below. It is appreciated that the following drawings depict only certain embodiments of the application and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.
Fig. 1 is a schematic diagram of a communication connection between a server and a monitoring data source according to an embodiment of the present application.
Fig. 2 is a schematic flow chart of a data monitoring management method according to an embodiment of the present application.
Fig. 3 is a schematic diagram of a tool set in a server according to an embodiment of the present application.
Fig. 4 is a functional block diagram of a data monitoring management apparatus according to an embodiment of the present application.
Icon: 10-a server; 11-a processing module; 12-a storage module; 13-a communication module; 100-data monitoring management device; 110-a data acquisition unit; 120-a conversion unit; 130-an analysis processing unit.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It should be noted that the terms "first," "second," and the like are used merely to distinguish one description from another, and are not intended to indicate or imply relative importance.
The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Referring to fig. 1, a server 10 is provided for collecting data from multiple types of monitoring data sources. The server 10 may include a processing module 11, a storage module 12 and a communication module 13, wherein the storage module 12 stores a computer program, and when the computer program is executed by the processing module 11, the server 10 may be caused to perform each step of the data monitoring and managing method described below.
In this embodiment, the monitoring data sources that the server 10 can collect include, but are not limited to, front-end data sources, application data sources, infrastructure data sources, and business process data sources.
In this embodiment, the front-end data source may be understood as a source for generating front-end data, and the front-end data source may be a front-end device, and may generate corresponding front-end data through an application program, a browser, or the like. Front-end devices include, but are not limited to, smart phones, Personal Computers (PCs), tablet PCs, Personal Digital Assistants (PDAs), and the like. Front-end data includes, but is not limited to, user behavior performance, crash/stuck, page response time performance, network request performance, interaction performance, page resource loading performance, and the like.
The application data source may be understood as a source from which application data is generated, and the application data source may be a terminal device in which an application program is installed. The terminal device may be, but is not limited to, a smart phone, a personal computer, etc. Application data includes, but is not limited to, request data, three-party interface data, message queue data (producer, consumer), logs, and the like.
An infrastructure data source may be understood as a source that generates infrastructure data, which may be a monitored server. The infrastructure data includes, but is not limited to, relevant data such as CPU, memory, hard disk, container, tablespace of database, session connection number, and table locking situation of the server.
The business process data source can be understood as a source for generating business process data, and the business process data source can be front-end equipment, a monitored server, terminal equipment and the like. The business process data includes, but is not limited to, the amount of orders, the amount of audits, etc. data.
In this embodiment, the server 10 may establish a communication connection with one or more monitoring data sources through the communication module 13, so as to collect monitoring data of the monitoring data sources.
Referring to fig. 2, an embodiment of the present application further provides a data monitoring management method, which can be applied to the server 10, and each step in the method is executed or implemented by the server 10. The method may include steps S210 to S230 as follows:
step S210, acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source;
step S220, converting the data in the data set into data in a specified format;
step S230, analyzing and processing the data in the specified format in the data set by using a big data processing tool according to a processing policy corresponding to the data type of the data in the specified format, so as to obtain a processing result.
In this embodiment, by deploying and installing a tool set corresponding to the monitoring data source on the server 10, and then using the tool set, data can be acquired from at least two data sources of the front-end data source, the application data source, the infrastructure data source, and the business process data source, and then the acquired data is analyzed and processed, so that the problems of single type of the monitored data source and limited monitoring index can be solved, and the realization of data monitoring and management of a full link is facilitated.
The individual steps in the process are explained in detail below, as follows:
in step S210, the tool sets corresponding to the monitoring data sources may be set according to the types of the monitoring data sources, and the tool sets corresponding to different monitoring data sources may be different. The tool set may include one or more software tools for collecting data of the monitoring data source, and the number and types of the tools may be set according to actual situations, and are not limited specifically here. The data included in the data set collected by the server 10 through the tool set may be determined according to actual conditions. The data sets may include, but are not limited to, front end data, application data, infrastructure data, business process data, and the like, as described above.
After the corresponding toolset is deployed on the server 10, the monitoring data source may also deploy tools that interact with the toolset. For example, a probe software tool is provided on a terminal device that monitors the data source. The probe can detect the Ping time, the webpage downloading speed, the webpage loading completion time and other data of the terminal equipment website, then the acquired data are sent to the server 10, and the data uploaded by the probe are collected and summarized by a software tool corresponding to the probe in the server 10. In addition, a buried point may be set for an application program running on the terminal device. Each embedded point can collect data such as logs of an application program, then send the collected data such as logs to the server 10, and collect the data uploaded by the embedded points by a software tool corresponding to the embedded points in the server 10.
It should be noted that, the server 10 can more comprehensively obtain various kinds of monitoring data when obtaining the monitoring data from different monitoring data sources. In addition, when acquiring monitoring data from different monitoring data sources, there is a possibility of repeatedly acquiring the data. For example, when data of the front-end data source and the application data source are acquired, related monitoring data, for example, user behavior data, runtime data, and the like of the application program, for the same application program of the terminal device may be acquired from the front-end data source and the application data source.
Referring to fig. 3, as an alternative implementation, step S210 may include:
acquiring a first data set of the front-end data source through a first tool set corresponding to the front-end data source;
acquiring a second data set of the application data source through a second tool set corresponding to the application data source;
obtaining a third data set of the infrastructure data source through a third tool set corresponding to the infrastructure data source;
and acquiring a fourth data set of the business process data source through a fourth tool set corresponding to the business process data source.
Understandably, the first data set is a collection of monitoring data acquired from a front-end data source. The second data set is a set of monitoring data obtained from the application data source. The third data set is a set of monitoring data obtained from infrastructure data sources. The fourth data set is a set of monitoring data obtained from the business process data source. The tools included in the first tool set, the second tool set, the third tool set and the fourth tool set can be set according to actual conditions.
The first set of tools includes, but is not limited to, at least one of a Piwik tool for collecting statistics of a website, a collection tool for collecting user operational data, a first application performance management tool for collecting performance data.
The Piwik tool is Web statistics software for open source code for Hypertext Preprocessor (PHP) and relational database management system (MySQL). The server 10 may generate a statistical report of the website through Piwik statistics, such as statistics of the number of people browsing the web page, the most visited pages, and the like, as the data in the first data set.
The collection tool (e.g., an operation data collection tool such as a pueraria IO collection tool, a policy data tool, etc.) may collect user operation data. The server 10 may obtain user operation data from the front-end data source through the gio collection tools, where the user operation data may include, but is not limited to, user information, advertisement information, page click volume, user grouping, conversion path, sticky retention, user behavior path, and other data.
The first Application Performance Management tool is an Application Performance Management (APM) tool capable of collecting front-end data sources, and may include a first tool (which may be referred to as APP-APM) for mobile-end Application Performance and a second tool (which may be referred to as browser APM) for browser Performance Management. The server 10 and the terminal device to be monitored may be provided with a first application performance management tool in cooperation with each other. The APP-APM can be used for monitoring and managing the application program performance in the mobile terminal, provides an end-to-end application performance monitoring management analysis platform, and can acquire data such as user behavior, crash/stuck condition, page response time, network request, interaction condition, page resource loading time and the like aiming at the application program.
The browser APM may provide an end-to-end application performance monitoring management analysis platform. The server 10 may collect data such as user behavior data, a crash/stuck condition, page response time, a network request, page resource loading duration, and the like for the browser through the browser APM.
The second set of tools includes, but is not limited to, at least one of a second application performance management tool for collecting performance data, a first Zabbix tool for collecting middleware data, a flux/FileBeat tool for collecting logs.
The second application performance management tool is an APM that can obtain monitoring data from an application data source. The server 10 may collect request data, call three-party interface data, call message queue data (producer, consumer), call database data, call NoSQL data, JVM data, error exception data, and topological relationships between data through a second application performance management tool.
The first Zabbix tool may obtain monitoring data related to the middleware from an application data source. Middleware includes, but is not limited to, WebLogic, JBoss, Tomcat, Druid data sources, and the like. The monitored data of the middleware includes, but is not limited to, a service state, a data source usage rate, a log, a port TCP connection state, etc. of the middleware, and may be determined according to actual situations. The functional roles of the middleware such as WebLogic, JBoss, Tomcat, and Druid data sources are well known to those skilled in the art and will not be described herein.
The flux/FileBeat tool may obtain log data such as logs of applications, logs of systems, etc. from an application data source. For example, the server 10 may collect GC (Garbage Collection) information of the JVM, system batch execution information, and some other log information output by the application system from the application data source through the Flume/filebed tool. Wherein, JVM refers to Java Virtual Machine, and is called Java Virtual Machine in English.
The third set of tools includes, but is not limited to, at least one of a second Zabbix tool for collecting server/database related data, a promemeus tool for collecting container data.
The second Zabbix tool may collect relevant data of the monitored server's CPU, memory, hard disk, container, etc. In addition, the second Zabbix tool may also collect, for the database, related data such as availability, capacity, data synchronization condition, tablespace, session connection number, and table locking condition of the database.
The Prometheus tool may collect data about the container, such as the memory space occupied by the container. The monitored containers may be referred to as collections, and may be used to store other data objects, as is well known to those skilled in the art, and will not be described in detail herein.
The fourth set of tools includes an API (Application Programming Interface) tool for collecting business data. The server 10 may collect the business data such as the order amount, the batch processing task number, the order amount, the audit amount, etc. from the corresponding API through the API tool. Of course, the service data may also be other data, and is not limited herein.
In step S220, the specified format may be set according to actual situations, for example, the specified format may be a JSON format or other formats (such as a TXT format). The data converted into the specified format may be stored in a database. For example, the data can be stored in a database such as Hbase or ClickHouse, etc., so as to facilitate the subsequent analysis processing of the data with a specified format.
For example, step S220 may include: and converting the data in the data set into data in a JSON format. Understandably, the JSON format is a lightweight data exchange format. The data in the JSON format is easy to read and write by people, is easy to analyze and generate by machines, and can effectively improve the network transmission efficiency.
In step S230, the server 10 may identify the data in the designated format, and obtain the data category of the data. The data category may be classified based on the IP address of the source of the data and the data name, and may be set according to actual situations. When the server 10 performs processing analysis on the collected data, a corresponding processing strategy can be selected to perform analysis processing on the data based on the data category of the data. The processing strategies for different data may be the same or different. The manner of processing each data may include one or more processing strategies.
The big data processing tool can be set according to actual conditions. For example, the big data processing tool may be, but is not limited to, a SparkStreaming tool, a Storm tool. Among them, the SparkStreaming tool may analyze and process data within a time period (for example, within n seconds, n is an integer greater than 0). The Storm tool can perform analytical processing on data acquired in real time. The function of the SparkStreaming tool, Storm tool, is well known to those skilled in the art and will not be described herein. In this embodiment, various types of data can be analyzed and processed according to user requirements by using big data processing tools such as a SparkStreaming tool and a Storm tool in combination with corresponding processing strategies.
As an alternative implementation, step S230 may include:
when the processing strategy is a filtering strategy, filtering the data in the specified format in the data set through the big data processing tool to obtain filtered data;
when the processing strategy is a classification strategy, storing the data in the specified format in a storage area corresponding to the data type and the source identification through the big data processing tool;
and when the processing strategy is a statistical strategy, counting the quantity of the data of the specified data category in a preset time period by the big data processing tool.
In this embodiment, the filtering policy, the classifying policy, and the statistical policy may be set according to actual situations. For example, the filtering policy may be to filter duplicate monitoring data. The repeated monitoring data can be understood as data with the same data content and the same equipment and time for generating the data. Or, the filtering policy may delete other data for data of a specified data category in the retained data set; or to delete data of a specified data category, other data is retained. The data of the specified data category may be set according to actual conditions, for example, log data, service data, and the like. Through the filtering strategy, the method is beneficial to flexibly screening corresponding data by a user, filtering repeated data, improving the effectiveness of the filtered data and facilitating the subsequent processing, such as statistical processing, of the filtered data.
In this embodiment, the data collected by the server 10 through the tool set may include data content, a name of the data, and a source identifier, where the source identifier may be a unique identifier of a device generating the data, such as a MAC address. The classification method of the classification strategy may be: classifying data with the same source identification and the same data name into the same class of data, and classifying data with the same source identification and different data names into different classes; alternatively, the data of the same source identifier is classified into a large class. Based on this, the server 10 is convenient to integrate and classify the data of different sources, so as to analyze and process the monitoring data of the same source identification in a targeted manner.
The statistical mode of the statistical strategy can be as follows: and in the corresponding time period, counting the quantity, the total amount of bytes and the like of the data of the same data category by combining with the source identification.
As an optional implementation, the method may further include: and when the processing result comprises a result of the abnormal data, sending an alarm prompt to a monitoring data source or a user terminal corresponding to the abnormal data.
For example, for monitoring data for which a safety threshold range exists, if the data is detected to exceed the safety threshold range, the server 10 may generate a result characterizing the data anomaly. The safety threshold range may be set according to actual conditions, and is not specifically limited herein.
For example, when the APP-APM collects the page response time of the front-end data source, if the response time exceeds the set time range, it indicates that the page response time is abnormal, and at this time, the server 10 may send an alarm prompt to the front-end data source or the user terminal. The set time can be set according to actual conditions. The user terminal may be a smartphone or a personal computer.
As an optional implementation, the method may further include: and sending the processing result to a user terminal through a message queue, wherein the user terminal is used for displaying the processing result according to a preset format, and the preset format comprises at least one of a statistical chart and a table.
Understandably, the server 10 may transmit the processing result to the user terminal to present the processing result of the monitoring data at the user terminal. The display mode may be, but is not limited to, a statistical chart, a table, and the like. The user terminal can be a display screen or a terminal device with the display screen, such as a personal computer, and supports small screen pages integrated through a single point. And the manager can check the monitoring processing result of the full link at any time through the user terminal. In addition, when abnormal data occurs, the abnormal data can be found in time through alarming, and the abnormal data is positioned, so that the abnormal condition can be maintained in time.
In order to facilitate understanding of the implementation process of the data monitoring management method, the implementation process of the data monitoring management method will be described in the following by way of example, as follows:
the server 10 may obtain corresponding monitoring data from the monitored data source in different ways based on the deployed tool set, collect and summarize the monitoring data into a standard JSON format, identify data link information, and push the data link information to Kafka for analysis and calculation by the big data processing tool. The toolset can implement data collection service for interfacing with an API, OGG (the OGG is called Oracle golden Gate in English and is a structured data backup tool based on logs) for interfacing with an Oracle database, Canal for interfacing with a Mysql database, Filebeat for interfacing with logs, and the like. If a new monitoring data source is accessed, a corresponding tool can be selected to be accessed into the Kafka according to the characteristics of the data source, for example:
operation monitoring-orychophragmus violaceus IO: the Zhuge IO collection module in the tool set can call the query API of the Zhuge IO and push the query result to Kafka for the big data analysis and processing tool to analyze and process, such as filtering, classifying, converting, calculating and supplementing information to the data;
the method comprises the steps that front-end and back-end performance monitoring-APM, APM used by background service used by APP, a browser and an application can be the same APM, a server 10 calls a query API of the APM through a tool set and pushes a query result to Kafka;
web statistics-Piwik, wherein the Piwik uses Mysql to store data, and pushes monitoring data in a Piwik database to Kafka through Cannal;
the business systems can also set burying points by themselves, and key business data (such as data of finished-form quantity, batch processing task quantity, order quantity, audit quantity and the like) to be monitored are pushed to the data collection Kafka;
the Zabbix monitors the middleware, the server, the network, the database and the container to obtain corresponding monitoring data, the bottom layer of the Zabbix uses Mysql for data storage, and the monitoring data in the Zabbix database is pushed to Kafka through Cannal;
the Prometheus monitors the container, and the server 10 can remotely read monitoring data by calling a remote _ read interface of the Prometheus and push the data to Kafka;
the Flume/FileBeat can identify and analyze log files in an incremental mode, and can collect monitoring information and error information concerned in application logs and push the monitoring information and the error information to Kafka. Part of the existing monitoring content is stored in Oracle, and the OGG can push the monitoring data in Oracle to Kafka in an incremental manner.
After collecting various monitoring data of each data source, the server 10 may perform real-time or timed calculation, analysis, and statistics on the collected monitoring data, and finally store the result in a database such as Hbase or ClickHouse, and push the result to Kafka for front-end data display. The processing and analyzing mode of the server 10 for the monitoring data through the big data tool may be as follows:
real-time streaming computing: acquiring real-time data from Kafka or Hbase, processing large-scale, high-throughput and fault-tolerant real-time data streams through Spark Streaming, persisting a calculated result into the Hbase or ClickHouse, and pushing the result to the Kafka for a front-end large-screen display to display consumption and alarm when monitoring indexes are abnormal;
batch calculation: acquiring original monitoring data, a monitoring data statistical result and service flow data from Hbase or ClickHouse through Spark according to Rowkey periodically (the periodic time can be set according to actual conditions), performing calculation statistical analysis (such as solving the maximum value within half an hour, solving the total number of data errors within half an hour and the like), and pushing to Kafka for a service system and a front-end large-screen display to show an alarm when consumption and monitoring indexes are abnormal;
splitting data according to data attribution, splitting the data according to data keywords of different data sources when splitting the data, and splitting the data if the Key of data JSON related to application contains IndexName and the Key of data JSON related to Zabbix monitoring contains Table and the like;
data processing: after various types of data are filtered and split, the processing modes of different types of data can be different. The data processing may include the following:
A. and completing the data information. And adding a data source identifier in the monitoring data, and converting the data needing to be converted between Code and Name so as to facilitate the subsequent use of the server 10.
B. And (6) associating the data. And for the case of multi-table association, performing association query of data and supplementing the data content.
C. And (6) integrating data. For the batch execution data acquired from the OGG, two pieces of data need to be combined into one piece for calculation processing (for example, when the execution time consumption of one piece of data is determined, the recording time of the first piece of data needs to be subtracted from the recording time of the second piece of data).
D. And (4) logic calculation. A logical evaluation is required for the partial data. For example, Full GC may be multiple strips. The required values for Full GC are two, one for the number of Full GC minutes and one for the average GC time. The packets are grouped according to the same system on the same IP at the same time, the number of GC times of several pieces of data under the packets is several, and the GC time of each GC is added to obtain the total GC time. The total time divided by the number of times is the average time.
Based on the design, the acquisition, real-time calculation, batch processing calculation, front-end display and output of full-link data can be realized. The server 10 can help system operation and maintenance personnel to rapidly 'ask for a diagnosis' and locate facilities for problems, thereby rapidly solving the abnormity and reducing the service loss; and intelligent early warning is carried out through big data, so that abnormity is avoided, and service loss is avoided.
Referring to fig. 4, an embodiment of the present application further provides a data monitoring management apparatus 100, which can be applied to the server 10 to execute or implement each step of the data monitoring management method. The data monitoring management device 100 includes at least one software functional module which can be stored in the storage module 12 in the form of software or Firmware (Firmware) or solidified in an Operating System (OS) of the server 10. The processing module 11 is used for executing executable modules stored in the storage module 12, such as software functional modules and computer programs included in the data monitoring management device 100.
The data monitoring and management device 100 may include a data acquisition unit 110, a conversion unit 120, and an analysis processing unit 130.
The data obtaining unit 110 is configured to obtain a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, where the monitoring data source includes at least two of a front-end data source, an application data source, an infrastructure data source, and a business process data source.
A converting unit 120, configured to convert the data in the data set into data in a specified format.
An analysis processing unit 130, configured to perform analysis processing on the data in the specified format in the data set by using a big data processing tool according to a processing policy corresponding to the data type of the data in the specified format, so as to obtain a processing result.
Optionally, the data obtaining unit 110 is configured to: acquiring a first data set of the front-end data source through a first tool set corresponding to the front-end data source; acquiring a second data set of the application data source through a second tool set corresponding to the application data source; obtaining a third data set of the infrastructure data source through a third tool set corresponding to the infrastructure data source; and acquiring a fourth data set of the business process data source through a fourth tool set corresponding to the business process data source.
Optionally, the analysis processing unit 130 is configured to: when the processing strategy is a filtering strategy, filtering the data in the specified format in the data set through the big data processing tool to obtain filtered data; when the processing strategy is a classification strategy, storing the data in the specified format in a storage area corresponding to the data type and the source identification through the big data processing tool; and when the processing strategy is a statistical strategy, counting the quantity of the data of the specified data category in a preset time period by the big data processing tool.
Optionally, the data monitoring and managing apparatus 100 may further include an alarm unit, configured to send an alarm prompt to a monitoring data source or a user terminal corresponding to the abnormal data when the processing result includes a result that the abnormal data exists.
Optionally, the data monitoring and managing apparatus 100 may further include a sending unit, configured to send the processing result to a user terminal through a message queue, where the user terminal is configured to display the processing result according to a preset format, and the preset format includes at least one of a statistical chart and a table.
Optionally, the conversion unit 120 may be configured to convert the data in the data set into data in JSON format.
It should be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the server 10 and the data monitoring and managing apparatus 100 described above may refer to the corresponding processes of the steps in the foregoing method, and are not described in detail herein.
In this embodiment, the processing module 11 may be an integrated circuit chip having signal processing capability. The processing module 11 may be a general-purpose processor. For example, the Processor may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Network Processor (NP), or the like; the method, the steps and the logic block diagram disclosed in the embodiments of the present Application may also be implemented or executed by a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
The memory module 12 may be, but is not limited to, a random access memory, a read only memory, a programmable read only memory, an erasable programmable read only memory, an electrically erasable programmable read only memory, and the like. In this embodiment, the storage module 12 may be used for storing monitoring data, processing strategies, and the like. Of course, the storage module 12 may also be used to store a program, and the processing module 11 executes the program after receiving the execution instruction.
The communication module 13 is used for establishing a communication connection between the server 10 and a monitoring data source through a network, and transceiving data through the network.
The embodiment of the application also provides a computer readable storage medium. The readable storage medium has stored therein a computer program that, when run on a computer, causes the computer to execute the data monitoring management method as described in the above embodiments.
From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by hardware, or by software plus a necessary general hardware platform, and based on such understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments of the present application.
In summary, the present application provides a data monitoring management method, an apparatus, a server and a readable storage medium. The method comprises the following steps: acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source; converting the data in the data set into data in a specified format; and analyzing and processing the data in the specified format in the data set by the big data processing tool according to the processing strategy corresponding to the data type of the data in the specified format to obtain a processing result. In the scheme, the tool set corresponding to the monitoring data source is deployed and installed on the server, then the tool set is utilized, data can be acquired from at least two data sources of the front-end data source, the application data source, the infrastructure data source and the business process data source, then the acquired data is analyzed and processed, and the problems that the type of the monitored data source is single and the index of the monitored data is limited can be solved.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus, system, and method may be implemented in other ways. The apparatus, system, and method embodiments described above are illustrative only, as the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A data monitoring and management method is characterized by comprising the following steps:
acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source;
converting the data in the data set into data in a specified format;
and analyzing and processing the data in the specified format in the data set by a big data processing tool according to a processing strategy corresponding to the data type of the data in the specified format to obtain a processing result.
2. The method of claim 1, wherein obtaining the data set of the corresponding monitoring data source through the tool set corresponding to the monitoring data source comprises:
acquiring a first data set of the front-end data source through a first tool set corresponding to the front-end data source;
acquiring a second data set of the application data source through a second tool set corresponding to the application data source;
obtaining a third data set of the infrastructure data source through a third tool set corresponding to the infrastructure data source;
and acquiring a fourth data set of the business process data source through a fourth tool set corresponding to the business process data source.
3. The method of claim 2, wherein the first set of tools includes at least one of a Piwik tool for collecting statistics of a website, a collection tool for collecting user operational data, a first application performance management tool for collecting performance data;
the second toolset comprises at least one of a second application performance management tool for collecting performance data, a first Zabbix tool for collecting middleware data, a flux/filebed tool for collecting logs;
the third toolset comprises at least one of a second Zabbix tool for collecting server/database related data, a promemeus tool for collecting container data;
the fourth set of tools includes API tools for collecting business data.
4. The method of claim 1, wherein analyzing and processing the data in the specified format in the data set by a big data processing tool according to a processing policy corresponding to a data category of the data in the specified format to obtain a processing result comprises:
when the processing strategy is a filtering strategy, filtering the data in the specified format in the data set through the big data processing tool to obtain filtered data;
when the processing strategy is a classification strategy, storing the data in the specified format in a storage area corresponding to the data type and the source identification through the big data processing tool;
and when the processing strategy is a statistical strategy, counting the quantity of the data of the specified data category in a preset time period by the big data processing tool.
5. The method according to any one of claims 1-4, further comprising:
and when the processing result comprises a result of the abnormal data, sending an alarm prompt to a monitoring data source or a user terminal corresponding to the abnormal data.
6. The method according to any one of claims 1-4, further comprising:
and sending the processing result to a user terminal through a message queue, wherein the user terminal is used for displaying the processing result according to a preset format, and the preset format comprises at least one of a statistical chart and a table.
7. The method of claim 1, wherein converting the data in the data set into data in a specified format comprises:
and converting the data in the data set into data in a JSON format.
8. A data monitoring management apparatus, characterized in that the apparatus comprises:
the data acquisition unit is used for acquiring a data set of a corresponding monitoring data source through a tool set corresponding to the monitoring data source, wherein the monitoring data source comprises at least two of a front-end data source, an application data source, an infrastructure data source and a business process data source;
the conversion unit is used for converting the data in the data set into data in a specified format;
and the analysis processing unit is used for analyzing and processing the data in the specified format in the data set through a big data processing tool according to a processing strategy corresponding to the data type of the data in the specified format to obtain a processing result.
9. A server, characterized in that the server comprises a memory, a processor, coupled to each other, in which memory a computer program is stored which, when executed by the processor, causes the server to carry out the method according to any one of claims 1-7.
10. A computer-readable storage medium, in which a computer program is stored which, when run on a computer, causes the computer to carry out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010943918.5A CN112069049A (en) | 2020-09-09 | 2020-09-09 | Data monitoring management method and device, server and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010943918.5A CN112069049A (en) | 2020-09-09 | 2020-09-09 | Data monitoring management method and device, server and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112069049A true CN112069049A (en) | 2020-12-11 |
Family
ID=73663235
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010943918.5A Pending CN112069049A (en) | 2020-09-09 | 2020-09-09 | Data monitoring management method and device, server and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112069049A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112783723A (en) * | 2021-01-20 | 2021-05-11 | 北京首汽智行科技有限公司 | Java virtual machine monitoring method |
| CN113688149A (en) * | 2021-07-20 | 2021-11-23 | 青岛海尔科技有限公司 | Monitoring method and device |
| WO2023245396A1 (en) * | 2022-06-20 | 2023-12-28 | 北京小米移动软件有限公司 | Database connection pool monitoring method and apparatus, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0969376A1 (en) * | 1998-06-30 | 2000-01-05 | Bull S.A. | Performance data service |
| US20060064485A1 (en) * | 2004-09-17 | 2006-03-23 | Microsoft Corporation | Methods for service monitoring and control |
| CN103455526A (en) * | 2012-06-05 | 2013-12-18 | 杭州勒卡斯广告策划有限公司 | ETL (extract-transform-load) data processing method, device and system |
| CN107968840A (en) * | 2017-12-15 | 2018-04-27 | 华北电力大学（保定） | A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system |
-
2020
- 2020-09-09 CN CN202010943918.5A patent/CN112069049A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0969376A1 (en) * | 1998-06-30 | 2000-01-05 | Bull S.A. | Performance data service |
| US20060064485A1 (en) * | 2004-09-17 | 2006-03-23 | Microsoft Corporation | Methods for service monitoring and control |
| CN103455526A (en) * | 2012-06-05 | 2013-12-18 | 杭州勒卡斯广告策划有限公司 | ETL (extract-transform-load) data processing method, device and system |
| CN107968840A (en) * | 2017-12-15 | 2018-04-27 | 华北电力大学（保定） | A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112783723A (en) * | 2021-01-20 | 2021-05-11 | 北京首汽智行科技有限公司 | Java virtual machine monitoring method |
| CN113688149A (en) * | 2021-07-20 | 2021-11-23 | 青岛海尔科技有限公司 | Monitoring method and device |
| WO2023245396A1 (en) * | 2022-06-20 | 2023-12-28 | 北京小米移动软件有限公司 | Database connection pool monitoring method and apparatus, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112612675B (en) | Distributed big data log link tracking method and system under micro-service architecture | |
| US20180365085A1 (en) | Method and apparatus for monitoring client applications | |
| CN112069049A (en) | Data monitoring management method and device, server and readable storage medium | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| US10592327B2 (en) | Apparatus, system, and method for analyzing logs | |
| CN101707632A (en) | Method for dynamically monitoring performance of server cluster and alarming real-timely | |
| CN109684052B (en) | Transaction analysis method, device, equipment and storage medium | |
| CN111240876B (en) | Fault positioning method and device for micro-service, storage medium and terminal | |
| CN101223509A (en) | System and method for adaptively collecting performance and event information | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| CN112307057A (en) | Data processing method and device, electronic equipment and computer storage medium | |
| CN111382023B (en) | Code fault positioning method, device, equipment and storage medium | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN111367760B (en) | Log collection method and device, computer equipment and storage medium | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| US20170132057A1 (en) | Full duplex distributed telemetry system | |
| CN111143286A (en) | Cloud platform log management method and system | |
| CN111651595A (en) | Abnormal log processing method and device | |
| CN111241050A (en) | Linkage analysis system and method for big data platform | |
| CN111800292A (en) | Early warning method and device based on historical flow, computer equipment and storage medium | |
| CN112433908B (en) | Method, system, device and medium for determining interval time of detection server | |
| US9201752B2 (en) | System and method for correlating empirical data with user experience | |
| CN113760652A (en) | Method, system, device and storage medium for full link monitoring based on application | |
| CN110311806B (en) | Mobile application program interface response delay diagnosis method, system and terminal | |
| CN111240936A (en) | Data integrity checking method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201211 |
|
| RJ01 | Rejection of invention patent application after publication |