CN112052134A

CN112052134A - Service data monitoring method and device

Info

Publication number: CN112052134A
Application number: CN202010707313.6A
Authority: CN
Inventors: 乐言; 郭宇
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2020-07-21
Filing date: 2020-07-21
Publication date: 2020-12-08

Abstract

The embodiment of the invention provides a method and a device for monitoring service data, which are applied to a system for monitoring the service data comprising a main control end, a data processing end and a data display end, and comprise the following steps: the main control end acquires a first configuration file, generates a data processing instruction and sends the data processing instruction to the data processing end; the data processing end obtains second target data from the target data source according to the data processing instruction, and the data display end receives the second target data sent by the data processing end and displays the second target data through a visualization tool. In the invention, the user only needs to generate the first configuration file and send the first configuration file to the main control end without learning the bottom logic of each complex component in the monitoring system, so that the monitoring process of the service data can be completed, the user does not need to learn the bottom logic of each component in the monitoring system, the configuration process of the monitoring system is simplified, and the learning cost of the monitoring system is reduced.

Description

Service data monitoring method and device

Technical Field

The present invention relates to the field of information processing, and in particular, to a method and an apparatus for monitoring service data.

Background

With the popularization of internet technology and the continuous improvement of informatization technology, the intelligent analysis and alarm of a service system become an important trend for effectively monitoring the operation condition of the service system in real time and analyzing and alarming hidden dangers in the service system in time.

At present, an open-source log monitoring system is mostly adopted in a service monitoring system, for example, the elasticstack comprises a logstack component, an elasticsearch component and a kibana component, when the elasticstack is used for log monitoring, a data collection and log analysis process needs to be configured through the logstack component, matching rules or regular expressions of some logs are written in the logstack component in advance, the logs generated by the service system are filtered, and formatted log files are generated; in addition, an Application Programming Interface (API) query statement needs to be configured through the elastic search component, so that query and analysis of the formatted log file can be completed according to the API query statement; furthermore, the graph function of the log file needs to be configured through the kibana component, and functions of data storage, real-time monitoring and the like are finally achieved.

However, in the current scheme, each component in the monitoring system needs to be configured respectively to realize data collection, analysis and visual display, and because the complexity of each component is high, developers need to invest in high learning cost to complete the configuration of each component, and special operation and maintenance personnel are needed to perform operation and maintenance, which results in a complex configuration process of the monitoring system and high learning cost.

Disclosure of Invention

The embodiment of the invention aims to provide a method and a device for monitoring business data, so as to solve the problems that in the prior art, when the business data generated by a business system is analyzed and monitored, the configuration of the analysis and monitoring process is carried out through each complex component of a monitoring system, so that the configuration process of the monitoring system is complicated, and the learning cost is high. The specific technical scheme is as follows:

in a first aspect of the present invention, a method for monitoring service data is provided, which is applied to a system for monitoring service data, where the system includes: the method comprises the following steps that:

the main control terminal obtains a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data;

the main control terminal generates a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field;

the main control end sends the data processing instruction to the data processing end;

the data processing end receives the data processing instruction sent by the main control end, and acquires first target data corresponding to a data acquisition rule in the data processing instruction from a target data source;

the data processing terminal calculates the first target data according to the calculation rule to obtain second target data;

the data processing end sends the second target data to the data display end;

the data display end receives the second target data sent by the data processing end;

and the data display end displays the second target data through a visualization tool.

In a second aspect of the present invention, a method for monitoring service data is provided, which is applied to a master control end in a monitoring system for service data, where the monitoring system further includes: the method comprises a data processing end and a data display end, and comprises the following steps:

obtaining a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data;

generating a data processing instruction comprising a data acquisition rule and the calculation rule according to the data acquisition rule field and the data processing rule field;

and sending the data processing instruction to the data processing end so that the data processing end can acquire first target data corresponding to the data acquisition rule from a target data source, calculating the first target data according to the calculation rule to obtain second target data, and further sending the second target data to the data display end so that the data display end can display the second target data through a visualization tool.

In a third aspect of the present invention, a method for monitoring service data is further provided, which is applied to a data processing end in a monitoring system for service data, where the monitoring system further includes: the method comprises the following steps that:

receiving a data processing instruction sent by the main control terminal, and acquiring first target data corresponding to a data acquisition rule in the data processing instruction from a target data source, wherein the data processing instruction comprises the data acquisition rule acquired from a data acquisition rule field in a first configuration file and a calculation rule acquired from a data processing rule field in the first configuration file;

calculating the first target data according to the calculation rule to obtain second target data;

and sending the second target data to the data display end so that the data display end can display the second target data through a visualization tool.

In a fourth aspect of the present invention, a method for monitoring service data is further provided, which is applied to a data display end in a monitoring system for service data, where the monitoring system further includes: the method comprises the following steps that:

receiving second target data sent by the data processing terminal, wherein the second target data is data obtained by acquiring, by the data processing terminal, first target data corresponding to a data acquisition rule in a data processing instruction from a target data source according to the data processing instruction sent by the main control terminal, and performing calculation processing on the first target data according to a calculation rule in the data processing instruction, the data processing instruction is an instruction generated according to a data acquisition rule field and a data processing rule field in a first configuration file, and the data processing rule field includes a calculation rule for the first target data;

and displaying the second target data through a visualization tool.

In a fifth aspect of the present invention, there is further provided a service data monitoring apparatus, which is applied to a service data monitoring system, where the monitoring system includes: the device comprises a main control end, a data processing end and a data display end, and comprises:

a first obtaining module, configured to obtain, by the master control end, a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data;

the first generation module is used for generating a data processing instruction comprising a data acquisition rule and a calculation rule by the main control terminal according to the data acquisition rule field and the data processing rule field;

the first sending module is used for sending the data processing instruction to the data processing end by the main control end;

the first receiving module is used for receiving the data processing instruction sent by the main control end by the data processing end and acquiring first target data corresponding to a data acquisition rule in the data processing instruction from a target data source;

the first calculation module is used for calculating the first target data by the data processing terminal according to the calculation rule to obtain second target data;

the second sending module is used for sending the second target data to the data display end by the data processing end;

the second receiving module is used for the data display end to receive the second target data sent by the data processing end;

and the first display module is used for displaying the second target data through a visualization tool by the data display end.

In a sixth aspect of the present invention, there is further provided a service data monitoring apparatus, which is applied to a master control end in a service data monitoring system, where the monitoring system further includes: data processing end and data show end, the device includes:

a second obtaining module, configured to obtain a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data;

a second generating module, configured to generate a data processing instruction including a data acquisition rule and the calculation rule according to the data acquisition rule field and the data processing rule field;

and the third sending module is used for sending the data processing instruction to the data processing end so that the data processing end can obtain first target data corresponding to the data obtaining rule from a target data source, and can calculate and process the first target data according to the calculating rule to obtain second target data, and further sending the second target data to the data display end so that the data display end can display the second target data through a visualization tool.

In a seventh aspect of the present invention, there is further provided a service data monitoring apparatus, which is applied to a data processing end in a service data monitoring system, where the monitoring system further includes: master control end and data show end, the device includes:

a third receiving module, configured to receive a data processing instruction sent by the master control end, and obtain, from a target data source, first target data corresponding to a data obtaining rule in the data processing instruction, where the data processing instruction includes the data obtaining rule obtained from a data obtaining rule field in a first configuration file and a calculation rule obtained from a data processing rule field in the first configuration file;

the second calculation module is used for calculating the first target data according to the calculation rule to obtain second target data;

and the fourth sending module is used for sending the second target data to the data display end so that the data display end can display the second target data through a visualization tool.

In an eighth aspect of the present invention, there is further provided a service data monitoring apparatus, which is applied to a data display end in a service data monitoring system, where the monitoring system further includes: master control end and data processing end, the device includes:

a fourth receiving module, configured to receive second target data sent by the data processing end, where the second target data is data obtained by the data processing end obtaining, according to a data processing instruction sent by the master control end, first target data corresponding to a data obtaining rule in the data processing instruction from a target data source, and performing calculation processing on the first target data according to a calculation rule in the data processing instruction, where the data processing instruction is an instruction generated according to a data obtaining rule field and a data processing rule field in a first configuration file, and the data processing rule field includes a calculation rule for the first target data;

and the second display module is used for displaying the second target data through a visualization tool.

In another aspect of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

the memory is used for storing a computer program;

the processor is configured to execute any one of the above-described service data monitoring methods when executing the program stored in the memory.

In yet another aspect of the present invention, there is also provided a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements any of the above-mentioned service data monitoring methods.

Compared with the prior art, the service data monitoring method and the service data monitoring device have the following advantages that:

the embodiment of the invention provides a method and a device for monitoring service data, which are applied to a system for monitoring the service data, wherein the system comprises the following components: the main control end, the data processing end and the data display end comprise: the main control end obtains a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data; the main control terminal generates a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field; the main control end sends the data processing instruction to the data processing end; the data processing end receives a data processing instruction sent by the main control end, and acquires first target data corresponding to a data acquisition rule in the data processing instruction from a target data source; the data processing end carries out calculation processing on the first target data according to the calculation rule to obtain second target data; the data processing end sends the second target data to the data display end; the data display end receives second target data sent by the data processing end; and the data display end displays the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

Fig. 1 is a flowchart illustrating interaction steps of a method for monitoring service data according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a monitoring system according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating a first configuration file according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a monitoring chart according to an embodiment of the present invention;

fig. 5 is a flowchart illustrating steps of a method for monitoring service data according to an embodiment of the present invention;

fig. 6 is a flowchart illustrating steps of another method for monitoring service data according to an embodiment of the present invention;

fig. 7 is a flowchart illustrating steps of another method for monitoring service data according to an embodiment of the present invention;

fig. 8 is a flowchart illustrating interaction steps of another method for monitoring service data according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating a second configuration file according to an embodiment of the present invention;

fig. 10 is a flowchart illustrating interaction steps of another method for monitoring service data according to an embodiment of the present invention;

FIG. 11 is a diagram illustrating an alarm message according to an embodiment of the present invention;

fig. 12 is a block diagram of a service data monitoring apparatus according to an embodiment of the present invention;

fig. 13 is a block diagram of another service data monitoring apparatus according to another embodiment of the present invention;

fig. 14 is a block diagram of another service data monitoring apparatus according to another embodiment of the present invention;

fig. 15 is a block diagram of another service data monitoring apparatus according to another embodiment of the present invention;

fig. 16 is a block diagram of an electronic device according to an embodiment of the invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.

Referring to fig. 1, a flowchart illustrating interaction steps of a method for monitoring service data according to an embodiment of the present invention is shown.

The embodiment of the invention provides a method for monitoring service data, which is applied to a system for monitoring service data, and the system comprises: the system comprises a main control end, a data processing end and a data display end.

In the embodiment of the present invention, the main control end may run a main control process through the main control server, the data processing end may run a data processing process through the data processing server, the data display end may run a data display process such as data display and alarm through the data display server, and the main control process, the data processing process, and the data display process may run in the same server or may run in different servers.

Step 101, the master control end obtains a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data.

Referring to fig. 2, which shows a schematic structural diagram of a monitoring system according to an embodiment of the present invention, in this step, a main control end 30 in a monitoring system for business data may obtain a first configuration file through a first configuration loading module, where the first configuration file is a configuration file written by a user for business data that needs to be monitored through a specific language, and the first configuration file may include a data obtaining rule field and a data processing rule field, where the data obtaining rule field may include a rule that a data processing end 40 obtains the business data that needs to be monitored from a target data source, and the data processing rule field may include a calculation rule that the data processing end 40 analyzes and calculates the obtained first target data.

Optionally, the first configuration file may be in another Markup Language (YAML) format or structured Query Language (sql) format.

Specifically, the yaml format is a format with high readability, which is used for expressing data serialization, and can simply express data forms such as lists, hash tables, scalars, and the like, and is particularly suitable for expressing or editing data structures, various configuration files, and the like due to blank symbol indentation and a large number of appearance-dependent features, and the yaml format language is centered on data rather than a mark language, so that various closed symbols are avoided ingeniously, such as: quotation marks, various brackets and the like enable engineers to avoid the problem that the symbols are complex and difficult to recognize when nested structures are used for writing configuration files in the yaml format.

Meanwhile, yaml does not have its own definition of data type, but uses a data type of a realization language, so that it can be a bridge for data flow between different languages, for example, if a monitoring system selected before a business system is an agoss host monitoring system (argus system), and a configuration file applied to the argus monitoring system is a file in JS object Notation (JSON) format, so that the configuration file can be converted into a configuration file in a batch manner, and applied to the monitoring system in the embodiment of the present invention, so that the working efficiency of the monitoring system can be improved, and in addition, for a plurality of configurations with similar configuration contents, a configuration file template can be generated, which can include a fixed corpus and a modified corpus, wherein the fixed corpus is a corpus common in the plurality of configurations with similar configuration contents, for example, for the seat locking service, the seat bitmap acquiring service and the ticket drawing service of the online ticket drawing system, the success rate of the service needs to be monitored, and when the power meets the alarm triggering condition, alarm information is generated, so that the field corresponding to the alarm aggregation function for calculating the ratio of the success times and the total times (namely, the power) can be configured in a configuration file: "date: "aspermcount (groupByNode (exclude { a }, 'success'),6, 'sum')), groupByNode ((exclude { B }, { Node }), 'sum'))" is set as a fixed corpus, and the storage path fields of the third target data a and B required for calculating the success rate are set as: the 'A' count. For another example, for two configurations in which the same first target data is obtained from the target data source according to the same data obtaining rule, the data obtaining rule field in the configuration template is determined as a fixed corpus, the modified corpus is a corpus having differences among a plurality of configurations, for example, different configurations of second target data are obtained according to the same first target data and through different data processing rules, the data processing rule field in the configuration template is determined as a modified corpus, and a technician writes the corresponding data processing rule field according to an actual situation, so that configuration files can be generated in batches, and the working efficiency of the monitoring system is further improved.

The sql language format is a special purpose programming language, a database query and programming language, used to access data and query, update, and manage relational database systems. The structured query language statements can be nested, which allows great flexibility and powerful nesting of functional structured query language statements, which allows great flexibility and powerful functionality.

Referring to fig. 3, a schematic diagram of a first configuration file according to an embodiment of the present invention is shown, as shown in fig. 3, that is, the first configuration file written in yaml language.

Wherein, part A is:

#regx：

'(？<level>NOTICE:？).*idc\[(？p<idc>\w+？)\].*cinema_id\[(？p<cinema_id>\w+？)\].*partner_cost_ms\[(？p<cost>\d+(\.\d+)？)\].*partner\[(？p<pname>\w+？)\].*getSeatInfo\[(？p<tstatus>\w+？)\]'

#regx：

'(？<level>NOTICE:？).*idc\[(？p<idc>\w+？)\].*partner_cost_ms\[(？p<cost>\d+(\.\d+)？)\].*partner\[(？p<pname>\w+？)\].*getSeatInfo\[(？p<tstatus>\w+？)\]'

#regx：

'(？<level>NOTICE:？).*idc\[(？p<idc>\w+？)\].*(partner_cost_ms\[(？p<cost>\d+(\.\d+)？)\])？.*partner\[(？p<pname>\w+？)\].*getSeatInfo\[(？p<tstatus>\w+？)\]'

regx：

'(？<level>NOTICE:？).*idc\[(？p<idc>\w+？)\].*partner\[(？p<pname>\w+？)\].*getSeatInfo\[(？p<tstatus>\w+？)\]'

the part B is as follows:

path_filed:

-idc

-pname

#-cinema_id

-tstatus

the part C is:

#number_filed:

#cost:

#-max

#-min

#-sum

the part D is:

alert_rule:

-rule:1

name: 'seating chart is less than 75% in idc success rate 7 times in succession'

A:'count.*.*.*'

B:'count.*.*.*'

Node：5

date：″asPercent(transformNull(groupByNode(exclude{A}，'success')，6，'sum'))，groupByNode((exclude{B}，{Node})，'sum'))″

operator：'＜'

threshold：'75'

filter：″7/7″

normal_filter：″3/3″

Specifically, the part a is a data acquisition rule field in the first configuration file, where the data acquisition field includes a data acquisition rule for acquiring the first target data from the target data source. Part C in fig. 3 is a data processing rule field in the first configuration file, and the data processing end analyzes and calculates the screened first target data according to a calculation rule for the first target data included in the data processing rule field to obtain an index and a dimension, which are required to be monitored by the user, for the target data source, and obtain second target data, so that the data display end can finally display the second target data by using a visualization tool, and the user can visually and clearly monitor each index of the service data.

And 102, the main control terminal generates a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field.

In this step, the main control end may obtain the data obtaining rule field and the data processing rule field from the first configuration file, and generate the data processing instruction including the data obtaining rule and the calculation rule according to the obtained data obtaining rule field and the data processing rule field.

Referring to fig. 2, the main control end 30 may receive the plurality of first configuration files through the first configuration loading module, and generate a plurality of data processing instructions according to the plurality of first configuration files through the monitoring task generating module, that is, generate a plurality of monitoring tasks for performing multi-process monitoring, so as to monitor the target data source from a plurality of dimensions, and improve the efficiency and the practicability of the monitoring system.

Optionally, the monitoring system may include a plurality of main control ends, one main control end is only responsible for part of the monitoring tasks by default, a data processing instruction is generated for part of the monitoring tasks, and the instruction is issued to the data processing end, so that the monitoring tasks can be distributed, the data processing pressure of each main control end is reduced, and when one main control end fails, the monitoring task responsible for the failed main control end can be completed by using the other main control ends, thereby avoiding that the service system cannot realize monitoring due to failure of the main control end.

And 103, the main control end sends the data processing instruction to the data processing end.

In this step, the main control end sends the generated data processing instruction to the data processing end, so that the data processing end obtains first target data corresponding to the data obtaining rule from a target data source, and performs calculation processing on the first target data according to the calculation rule to obtain second target data.

Referring to fig. 2, the main control terminal 30 sends the generated monitoring task to the data processing terminal 40 through the task distribution module, that is, sends a data processing instruction including a data acquisition rule and a calculation rule to the data processing terminal 40, so that the data processing terminal 40 acquires first target data corresponding to the data acquisition rule from a target data source, and analyzes and calculates the first target data according to the calculation rule through the calculation unit and the second aggregation unit to finally obtain second target data.

And 104, the data processing terminal receives the data processing instruction sent by the main control terminal, and acquires first target data corresponding to a data acquisition rule in the data processing instruction from a target data source.

In the step, the data processing end receives a data processing instruction sent by the main control end, and acquires first target data corresponding to a data acquisition rule from a target data source according to the data acquisition rule in the data processing instruction.

In the embodiment of the present invention, the data acquisition rule field in the first configuration file may be a regular expression, and in the process of acquiring data, the data processing end may screen out data that conforms to the regular expression from the target data source according to the regular expression, and use the data as the first target data, as shown in fig. 3, and according to the regular expression included in part a, the data processing end needs to screen out data that includes fields "idc", "partner", and "getseatnfo" from the target data source, and use the data as the first target data.

Specifically, a high-efficiency regular engine can be adopted to complete the regular matching process, so that the working efficiency of the monitoring system is improved, preferably, a hyperscan regular engine can be selected, the hyperscan is a high-performance regular expression matching library, the hyperscan can support the matching of tens of thousands to hundreds of thousands of rules according to the difference of rule complexity, and excellent matching performance and expansibility are shown.

Alternatively, referring to fig. 2, the target data source is a data consumption module in the data processing end 40, and consumes the obtained data from the kaffa (kafka) in the data summarization module 50.

The kafka, among other things, is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of consumers in websites, which actions (web browsing, searching and other user actions) are a key factor for many social functions on modern networks, and these data are usually solved by handling logs and log aggregations due to throughput requirements.

kafka has the following characteristics: (1) the message persistence capability is provided in a mode that the time complexity is O (1) (the data volume is increased by 1 time, and the time consumption is increased by 1 time), so that the access performance of constant time can be ensured even for data above Terabyte (TB) level; (2) the throughput rate is high, and the single machine can support the transmission of 100K messages per second even on a very cheap commercial machine; (3) supporting message partitioning and distributed consumption among Kafka servers, and simultaneously ensuring the message sequence transmission in each partition (partition); (4) simultaneously, offline data processing and real-time data processing are supported; (5) online horizontal extension is supported.

In this embodiment of the present invention, the service system stores the generated log data in a kafka message system in the data summarization module, where the messages stored in the kafka message system include a message category (topic) and a message content (body), where the topic of each message is the category of the message, and is customizable by a user when writing the kafka data, the messages of physically different topics are stored separately, and each topic physically includes one or more partitions, optionally, the messages that generate the same type within a certain time range may be stored in the same partition according to the generation time of the message, and the body of each message is the log data generated by the service system and stored in kafka.

Referring to fig. 2, log data written to the kafka messaging system is collected by the log data sourcing module 60 and written to the kafka messaging system. Optionally, the log data may include a production service log, a gateway log, an open source frame log, and the like, the production service log may include data generated during each production service in the service system, the gateway log may include important information such as a caller Internet Protocol (IP), a Resource Locator (URL) of a request, response delay, a status code returned, a number of bytes requested, and a response byte in an access record, and the open source frame log may include open source frame data of the service system.

And 105, the data processing terminal calculates the first target data according to the calculation rule to obtain second target data.

In this step, the data processing terminal receives a data processing instruction sent by the main control terminal, and performs calculation processing on the acquired first target data according to a calculation rule in the data processing instruction to obtain second target data.

Specifically, referring to fig. 3, a part C in the figure is a data processing rule field in the first configuration file, and the data processing end analyzes and calculates the screened first target data according to a calculation rule for the first target data included in the data processing rule field to obtain an index and a dimension, which are needed to be monitored by the user, for the target data source, and obtain second target data, so that the second target data can be displayed by using a visualization tool finally, and the user can monitor each index of the service data visually and clearly.

In the embodiment of the present invention, as a default, the calculation rule includes a count number rule, that is, in the target data source, if data meeting the data acquisition rule is detected, the count corresponding to the data is incremented by 1, that is, in the target data source, data including fields "idc", "parater", and "getseatnfo" is detected, the number of data of corresponding "idc" type, "pname" type, and "tstattus" type is incremented by 1.

It should be noted that for a specific online ticketing service, "idc" may indicate an area where data is generated, for example, bj (beijing machine room) or wh (wuhan machine room), the area is written in an "idc" field in a service log, "pname" may indicate a name of a partner, the name of the partner is written in a "partner" field in the service log, and tstatus may indicate a status of the specific service, for example, success/failure of acquiring a seat map is requested, and the status of the service is written in a "getseatnfo" field in the service log.

Optionally, if there is no "#" at the top of each row in the portion C in fig. 3 (the character "#" in the yaml file identifies a comment), the calculation rules further include a maximum value rule, a minimum value rule, and a summation rule.

Specifically, according to the above calculation quantity rule, the calculation logic of the data processing end when calculating the quantity of the first target data is as follows:

def count(num,base):

return num+1

the data processing end sums the first target data according to the summation rule, wherein the computation logic of the data processing end when summing the first target data is as follows:

def sum(num,base):

return num+base

the data processing end calculates the maximum value of the first target data according to the maximum value rule, and the calculation logic is as follows:

def max(num,base):

return num if num>base else base

the data processing end calculates the minimum value of the first target data according to the minimum value rule, and the calculation logic is as follows:

def min(num,base):

return num if num<base else base

and 106, the data processing end sends the second target data to the data display end.

In the step, the data processing end sends the calculated second target data to the data display end, so that the data display end can display the second target data by using a visualization tool.

And 107, the data display end receives the second target data sent by the data processing end.

In this step, the data display end may receive the second target data sent by the data processing end.

And 108, displaying the second target data by the data display end through a visualization tool.

In this step, the data processing end sends the calculated second target data to the data display end, and the data display end may display the second target data by using a visualization tool after receiving the second target data.

Optionally, the second target data includes a first timestamp, the first timestamp includes generation time of the second target data, the data display end may generate a monitoring chart corresponding to the second target data through a visualization tool according to a time sequence of the first timestamp, and display the monitoring chart, where an abscissa of the monitoring chart is time.

Specifically, referring to fig. 4, a schematic diagram of a monitoring chart according to an embodiment of the present invention is shown, as shown in fig. 4, a curve F represents a change of a total number of successful requests of all partners with time for a specific online ticketing service, for example, the total number of successful requests of all partners is 2240 times at 16: 32. Correspondingly, the second target data generated by the data processing end is the total number of successful requests of all the partners and the first timestamp containing the time for generating the second target data, and it should be noted that, if the monitoring system monitors the online ticketing service in real time, the data generated at the generation time is equivalent to the real-time data of the online ticketing service.

Further, the data display end can utilize a visualization tool to generate a chart with the time as a horizontal coordinate and the total successful requests of all the partners as a vertical coordinate according to the time corresponding to the total successful requests of all the partners, and the chart is displayed in the visualization tool so that workers related to the online ticket issuing service can monitor the change condition of the total successful requests of all the partners in real time.

Further, as shown in fig. 4, the curve G represents the change in the total number of requests of all partners over time, i.e., the number of requests including both the case of request failure and the case of request success, for example, the total number of requests of all partners is 2245 times at 16: 32. Correspondingly, the second target data generated by the data processing terminal is the total request number of all the partners, and the first time stamp comprises the time for generating the second target data. Further, the data display end can utilize a visualization tool to generate a chart with the time as an abscissa and the total request number of all the partners as an ordinate according to the time corresponding to the total request number of all the partners, and the chart is displayed in the visualization tool so that workers related to the online ticket issuing service can monitor the change condition of the total request number of all the partners in real time.

Preferably, the visualization tool may be a grapana visualization display tool, and grapana is an open source application written in the golang (go) language, is mainly used for visualizing and displaying large-scale index data, is the most popular time series data display tool in network architecture and application analysis, and currently supports most common time series databases.

In addition, the Grafana visual display tool can finish drawing only by configuring a small number of API variables at the front end of the tool, and meanwhile, the Grafana visual display tool can generate a statistical chart for displaying relevant data of a business system according to historical data of the business system.

In summary, the service data monitoring method provided in the embodiment of the present invention is applied to a service data monitoring system, where the monitoring system includes: the main control end, the data processing end and the data display end comprise: the main control end obtains a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data; the main control terminal generates a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field; the main control end sends the data processing instruction to the data processing end; the data processing end receives a data processing instruction sent by the main control end, and acquires first target data corresponding to a data acquisition rule in the data processing instruction from a target data source; the data processing end carries out calculation processing on the first target data according to the calculation rule to obtain second target data; the data processing end sends the second target data to the data display end; the data display end receives second target data sent by the data processing end; and the data display end displays the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 5, a flowchart illustrating steps of a method for monitoring service data according to an embodiment of the present invention is shown.

The embodiment of the invention provides a method for monitoring service data, which is applied to a main control end in a monitoring system of the service data, and the monitoring system further comprises: the data processing terminal and the data display terminal.

Step 201, obtaining a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data.

This step can refer to step 101, and is not described herein.

Step 202, generating a data processing instruction including a data acquisition rule and the calculation rule according to the data acquisition rule field and the data processing rule field.

This step can be referred to as step 102, and is not described herein.

Step 203, sending the data processing instruction to the data processing end.

In this step, the main control end sends the data processing instruction to the data processing end, so that the data processing end obtains first target data corresponding to the data obtaining rule from a target data source, calculates the first target data according to the calculation rule to obtain second target data, and further sends the second target data to the data display end, so that the data display end displays the second target data through a visualization tool.

This step can refer to step 103, and is not described herein.

In summary, the monitoring method for service data provided in the embodiment of the present invention is applied to a master control end in a monitoring system for service data, and the monitoring system further includes: data processing end and data show end include: acquiring a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data; generating a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field; and sending the data processing instruction to a data processing end so that the data processing end can acquire first target data corresponding to the data acquisition rule from a target data source, and calculating the first target data according to the calculation rule to obtain second target data, and further sending the second target data to a data display end so that the data display end can display the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 6, a flowchart illustrating steps of another method for monitoring service data according to an embodiment of the present invention is shown.

The embodiment of the invention provides a method for monitoring service data, which is applied to a data processing end in a monitoring system of the service data, and the monitoring system further comprises: the main control end and the data display end.

Step 301, receiving a data processing instruction sent by the master control end, and acquiring first target data corresponding to a data acquisition rule in the data processing instruction from a target data source.

This step can refer to step 104, and is not described herein.

Step 302, performing calculation processing on the first target data according to the calculation rule to obtain second target data.

This step can be referred to specifically as step 105, and is not described herein again.

Step 303, sending the second target data to the data display end, so that the data display end displays the second target data through a visualization tool.

This step can be referred to as step 106, and is not described herein.

In summary, the monitoring method for service data provided in the embodiment of the present invention is applied to a data processing end in a monitoring system for service data, and the monitoring system further includes: the master control end and the data display end comprise: receiving a data processing instruction sent by a main control terminal, and acquiring first target data corresponding to a data acquisition rule in the data processing instruction from a target data source, wherein the data processing instruction comprises the data acquisition rule acquired from a data acquisition rule field in a first configuration file and a calculation rule acquired from a data processing rule field in the first configuration file; calculating the first target data according to the calculation rule to obtain second target data; and sending the second target data to a data display end so that the data display end can display the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 7, a flowchart illustrating steps of another method for monitoring service data according to an embodiment of the present invention is shown.

The embodiment of the invention provides a method for monitoring service data, which is applied to a data display end in a monitoring system of the service data, and the monitoring system further comprises: the system comprises a main control end and a data processing end.

Step 401, receiving second target data sent by the data processing end.

This step can be referred to as step 107, and is not described herein.

And 402, displaying the second target data through a visualization tool.

This step can refer to step 108, and is not described herein.

In summary, the monitoring method for service data provided in the embodiment of the present invention is applied to a data display end in a monitoring system for service data, and the monitoring system further includes: the main control end and the data processing end comprise: receiving second target data sent by a data processing end, wherein the second target data is data obtained by the data processing end by acquiring first target data corresponding to a data acquisition rule in a data processing instruction from a target data source according to the data processing instruction sent by a main control end and calculating and processing the first target data according to a calculation rule in the data processing instruction, the data processing instruction is an instruction generated according to a data acquisition rule field and a data processing rule field in a first configuration file, and the data processing rule field comprises a calculation rule for the first target data; and displaying the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 8, a flowchart illustrating interaction steps of another method for monitoring service data according to an embodiment of the present invention is shown.

The business data monitoring method provided by the embodiment of the invention is applied to a business data monitoring system comprising a main control end, a data processing end and a data display end.

Step 501, a data processing end obtains a second configuration file, where the second configuration file includes a target data source type.

In this step, referring to fig. 2, the data processing end 40 may obtain, through the second configuration loading module, a second configuration file, where the second configuration file is a configuration file written by a user for service data that needs to be monitored through a specific language, where the second configuration file may include a type of a target data source, referring to fig. 9, a schematic diagram of a second configuration file according to an embodiment of the present invention is shown, as shown in fig. 9, the second configuration file is the second configuration file written by using yaml language, and "bmovieapp _ topic" defined in the second configuration file is the type of the target data source, because all log data generated by the service system are stored in the kafka message system in the data summarizing module 50, the data volume is large, and if the data processing end receives a data processing instruction sent by the main control end, according to a data obtaining rule in the data processing instruction, the data needing to be monitored is matched and screened out from the kafka message system with large data volume, so that the time consumption of the monitoring process is long, and the timeliness and the efficiency of the monitoring process are reduced.

Since in general, when monitoring a service system, the same type of data (i.e., a target data source) is analyzed and calculated to obtain a final monitoring or alarm result, the data processing end 40 may extract data with the same topic from the kafka message system through the consumption data module before receiving a data processing instruction sent by the main control end 30.

Specifically, the messages stored in the kafka message system include topic and body, where topic of each message is a category of the message, and is customizable by a user when writing kafka data, and the messages of physically different topics are stored separately, and body of each message is log data generated by the business system and stored in kafka.

Step 502, the data processing end acquires, according to the target data source type, a plurality of target data sources matched with the target data source type from a data source, where the target data sources include a third timestamp, and the third timestamp includes generation time of the target data sources.

In the step, the data processing end acquires a plurality of target data sources matched with the target data source types from the data sources stored in the kafka message system through the consumption data module according to the target data source types in the second configuration file.

Further, the target data source may further include a third timestamp including a generation time of the target data source.

In step 503, the data processing end obtains a plurality of preset time ranges.

In this step, the data processing end may obtain a plurality of preset time ranges, for example, the preset time ranges may be set as one preset time range every 1 minute from a fixed time point to a current time point.

Step 504, the data processing end adds the target data source with the third timestamp in the same preset time range to the same sub-target data source to obtain a plurality of sub-target data sources, and stores the plurality of sub-target data sources in a local disk.

In this step, the data processing end adds a plurality of target data sources acquired from the kafka message system to the same sub-target data source according to third timestamps corresponding to the plurality of target data sources, where the third timestamps are within the same preset time range, so as to physically partition the target data sources, that is, subdivide data having the same topic into a plurality of partitions, and store the plurality of sub-target data sources in a local disk of the data processing end.

Step 505, the main control end obtains a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data.

This step can refer to step 101, and is not described herein.

Step 506, the master control end generates a data processing instruction including the data acquisition rule and the calculation rule according to the data acquisition rule field and the data processing rule field.

This step can be referred to as step 102, and is not described herein.

And 507, the main control end sends the data processing instruction to the data processing end so that the data processing end can acquire first target data corresponding to the data acquisition rule from a target data source, and calculates the first target data according to the calculation rule to obtain second target data.

This step can refer to step 103, and is not described herein.

Step 508, the data processing end receives the data processing instruction sent by the master control end, and obtains a plurality of first sub-target data corresponding to the data obtaining rule in the data processing instruction from a plurality of sub-target data sources.

In this step, after the data processing end receives the data processing instruction sent by the main control end, a plurality of threads may be started, and according to the received data processing instruction, a plurality of first sub-target data corresponding to the data acquisition rule in the data processing instruction are acquired from the plurality of sub-target data sources generated in step 504.

In the embodiment of the present invention, the data obtaining rule in the first configuration file may be a regular expression, and in the process of obtaining data, the data processing end may screen out, according to the regular expression, data that meets the regular expression from the plurality of sub-target data sources, and use the data as the first sub-target data, as shown in fig. 3, and according to the regular expression included in part a, the data processing end needs to screen out, from the plurality of sub-target data sources, data that includes fields "idc", "partner", and "getseatnfo" as the first sub-target data.

Specifically, in step 504, the data processing end further divides the target data sources having the same type into a plurality of partitions according to the third timestamp of the target data sources, so that in this step, the data processing end can screen out data matching the regular expression from the plurality of partitions according to the regular expression in the data processing instruction, and use the data as the first sub-target data.

In step 509, the data processing end performs calculation processing on the plurality of first sub-target data through the plurality of calculation units according to the calculation rule in the data processing instruction, so as to obtain a plurality of second sub-target data.

In the embodiment of the invention, the data processing terminal comprises a plurality of computing units and a first aggregation unit, the number of the computing units is the same as that of the sub-target data sources, and the computing units and the sub-target data sources are in one-to-one correspondence.

In this step, after acquiring a plurality of first sub-target data matched with the regular expression in the first configuration file, the data processing terminal may call a corresponding number of computing units according to the number of partitions to which the first sub-target data belongs, and each computing unit analyzes and computes data in one partition according to a computing rule in the data processing instruction, so as to obtain a plurality of second sub-target data.

The process of calculating the first target data by the data processing end can adopt a multi-thread mode, so that the efficiency of monitoring the service system can be improved, meanwhile, if the calculating unit of one thread has errors, the data to be processed by the thread corresponding to the fault calculating unit can be correspondingly calculated by other threads, and the availability and the concurrency of the monitoring system can be improved.

Specifically, the process of performing data processing on data in a partition by the computing unit according to the computing rule in the data processing instruction may refer to the process in step 105, and is not described herein again.

It should be noted that, for one data processing instruction, that is, for the same monitoring task, the multi-thread analysis calculation may be implemented by using multiple computing units in the data processing end, or for the same monitoring task, the multi-thread analysis calculation process may be further completed by using multiple data processing ends and multiple computing units in multiple data processing ends, so as to ensure that if one of the data processing ends fails, the data that needs to be processed by the thread corresponding to the failed data processing end may be correspondingly calculated by using other threads, thereby further improving the availability and concurrency of the monitoring system.

Meanwhile, when multi-thread analysis and calculation are carried out, in order to prevent data confusion among threads and unsafe threads, namely data access protection is not provided, data may be changed by a plurality of threads in sequence to cause that the obtained data is dirty data. In this regard, a locking mechanism may be employed, that is, when one thread accesses certain data of this class, the data is protected, so that other threads cannot access the data until the thread finishes reading the data, and other threads cannot use the data, thereby ensuring that data inconsistency or data pollution does not occur.

And step 510, the data processing end performs aggregation operation on the plurality of second sub-target data through the first aggregation unit according to the first aggregation function to obtain the second target data.

In this embodiment of the present invention, the data processing end includes a plurality of computing units and a first aggregation unit, and the data processing rule field further includes a first aggregation function, where the first aggregation function may be determined according to a specific computing process, for example, if a computing rule according to which the computing process is based is a computation quantity rule or a summation rule, the first aggregation function is only to simply sum up the plurality of second sub-target data, and if the computing rule according to which the computing process is based is a maximum value rule or a minimum value rule, the first aggregation function is to compare the plurality of second sub-target data again to obtain a maximum value or a minimum value in the plurality of second sub-target data.

In this step, the data processing end may perform, by using the first aggregation unit, an aggregation operation on the plurality of second sub-target data calculated by each thread according to the first aggregation function, to obtain final second target data.

And 511, the data processing end sends the second target data to the data display end.

In this step, the data processing terminal sends the calculated second target data to the data display terminal, so that the data display terminal can display the second target data through the visualization tool.

Step 512, the data display end receives the plurality of second target data sent by the plurality of data processing ends.

In this embodiment of the present invention, the monitoring system for the service data includes a plurality of the data processing terminals, the data display terminal includes a second aggregation unit, and the data processing rule field further includes a second aggregation function.

In this step, because a multi-thread analysis calculation process can be performed on one data processing instruction, that is, for the same monitoring task, by using a plurality of data processing terminals and a plurality of calculation units in the plurality of data processing terminals, after a plurality of second sub-target data obtained by the plurality of calculation units in a single data processing terminal are aggregated according to the first aggregation function, each data processing terminal sends a plurality of second target data obtained after aggregation to the data display terminal.

And 513, the data display end performs aggregation operation on the plurality of second target data through the second aggregation unit according to the second aggregation function to obtain aggregated data, and displays the aggregated data through the visualization tool.

In this step, the data display end receives a plurality of second target data generated and sent by a plurality of data processing ends, and for the same monitoring task, the second target data calculated by one data processing end is still the second target data calculated for a part of the service data, so that after receiving the plurality of second target data sent by the plurality of data processing ends, the data display end can use the second aggregation unit to perform aggregation operation on the plurality of second target data again according to the second aggregation function, and the obtained aggregated data is used as final second target data and displayed by the visualization tool.

Therefore, if one data processing end fails, the data to be processed by the thread corresponding to the failed data processing end can be calculated by other threads, and the availability and concurrency of the monitoring system are further improved.

Specifically, each data processing end may write the calculated second target data and the identifier of the corresponding data processing end into a Remote Dictionary service (Remote Dictionary Server, redis) database, so that the data display end reads the corresponding second target data from the redis database, when the data display end reads the second target data, the second target data may be verified, and it is determined whether the read second target data is normal, thereby determining whether the data processing end has successfully performed the calculation and writing process of the data, if the verification fails, the data processing end is removed from the monitoring system, and a part of the first target data processed by the data processing end is processed by other data processing ends, thereby ensuring availability and concurrency of the monitoring system.

In addition, in this step, the process of displaying the aggregated data (i.e., the second target data) through the visualization tool may specifically refer to step 108, which is not described herein again.

In summary, the method for monitoring service data provided in the embodiment of the present invention is applied to a main control end, a data processing end, and a data display end in a monitoring system for service data, and includes: the main control end obtains a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data; the main control terminal generates a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field; the main control end sends the data processing instruction to the data processing end; the data processing end receives a data processing instruction sent by the main control end, and acquires first target data corresponding to a data acquisition rule in the data processing instruction from a target data source; the data processing end carries out calculation processing on the first target data according to the calculation rule to obtain second target data; the data processing end sends the second target data to the data display end; the data display end receives second target data sent by the data processing end; and the data display end displays the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 10, a flowchart illustrating interaction steps of another method for monitoring service data according to an embodiment of the present invention is shown.

Step 601, the data processing end obtains a second configuration file, wherein the second configuration file comprises a target data source type.

This step can be referred to specifically as step 501, and is not described herein again.

Step 602, the data processing end acquires, according to the target data source type, a plurality of target data sources matched with the target data source type from a data source, where the target data sources include a third timestamp, and the third timestamp includes generation time of the target data sources.

This step may refer to step 502, and will not be described herein.

Step 603, the data processing end obtains a plurality of preset time ranges.

This step can refer to step 503, which is not described herein again.

And step 604, adding the target data source with the third timestamp in the same preset time range to the same sub-target data source by the data processing terminal to obtain a plurality of sub-target data sources, and storing the plurality of sub-target data sources in a local disk.

This step can be referred to specifically as step 504, and is not described here again.

Step 605, the main control end obtains a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data.

This step can refer to step 101, and is not described herein.

And 606, generating a data processing instruction comprising a data acquisition rule and a calculation rule by the main control terminal according to the data acquisition rule field and the data processing rule field.

This step can be referred to as step 102, and is not described herein.

Step 607, the main control end sends the data processing instruction to the data processing end, so that the data processing end obtains the first target data corresponding to the data obtaining rule from the target data source, and performs calculation processing on the first target data according to the calculation rule to obtain the second target data.

Optionally, the data obtaining rule included in the first configuration file includes a regular expression corresponding to the first target data, and correspondingly, in this step, the master control end sends the generated data processing instruction to the data processing end, so that the data processing end obtains, from a target data source, first target data matched with the regular expression, and performs calculation processing on the first target data according to the calculation rule to obtain second target data.

Specifically, referring to fig. 2, the main control terminal 30 sends the generated monitoring task to the data processing terminal 40 through the task distribution module, that is, sends a data processing instruction including a data obtaining rule and a calculation rule to the data processing terminal 40, so that the data processing terminal 40 obtains first target data matched with the regular expression from a target data source, and analyzes and calculates the first target data according to the calculation rule through the calculation unit and the first aggregation unit, so as to finally obtain second target data.

In the step, the data processing end receives a data processing instruction sent by the main control end, and obtains first target data matched with a regular expression from a target data source according to the regular expression in the data processing instruction.

Step 608, the data processing end receives the data processing instruction sent by the main control end, and obtains, from a target data source, first target data corresponding to a data obtaining rule in the data processing instruction, where the data processing instruction includes the data obtaining rule obtained from a data obtaining rule field in a first configuration file, and a calculation rule obtained from a data processing rule field in the first configuration file.

Optionally, step 608 specifically includes:

and a substep 6081, in which the data processing terminal receives the data processing instruction sent by the main control terminal, and acquires first target data matched with the regular expression from the target data source.

In the embodiment of the present invention, the data obtaining rule included in the first configuration file includes a regular expression corresponding to the first target data, and correspondingly, in this step, the data processing terminal receives the data processing instruction sent by the main control terminal, and obtains the first target data matched with the regular expression from the target data source according to the regular expression in the data processing instruction.

Specifically, the data acquisition rule field in the first configuration file may be a regular expression, and in the process of acquiring data, the data processing end may screen out data that conforms to the regular expression from the target data source according to the regular expression, and use the data as the first target data, as shown in fig. 3, and according to the regular expression included in part a, the data processing end needs to screen out data that includes fields "idc", "partner", and "getseatnfo" from the target data source, and uses the data as the first target data.

And step 609, the data processing end performs calculation processing on the first target data according to the calculation rule to obtain second target data.

And step 610, the data processing end sends the second target data to the data display end.

This step can be referred to as step 106, and is not described herein.

Step 611, the data display end receives the second target data sent by the data processing end, and writes the second target data into the time sequence database according to the write path and the time sequence of the first timestamp.

In this embodiment of the present invention, the data processing rule field in the first configuration file further includes: a write path when the second target data is written into the time series database; the second target data includes a first timestamp including a generation time of the second target data.

Correspondingly, in this step, referring to fig. 2, after receiving the second target data generated and sent by the data processing end, the data display end 70 may write the second target data into the time sequence database according to the time sequence order of the first timestamps through the time sequence database storage module 20.

Specifically, as shown in fig. 3, a "path _ file" in the B part is a write path when the second target data is written into the time sequence database in the first configuration file.

Preferably, the time-series database may be a graph, which is an open-source real-time graphical system displaying time-series metric data, and the graph does not collect the metric data itself, but rather, like a database, receives the metric data through its back end, and then queries, transforms, and combines the metric data in a real-time manner. Graph supports a built-in World Wide Web interface that allows users to browse metric data and graphs. The architecture of the graph is modular and scalable, has huge communities and wide support, and has a plurality of open source tools which cooperate with the graph, for example, the selected visualization tool Grafana, the structure of the data stored in the graph and the structure of the data required by the Grafana for visualization display are matched with each other, and an additional data structure conversion process is not required.

Furthermore, a carbon is used as a storage middleware in the graph time sequence database, so that different underlying data storage services can be accessed more conveniently.

Preferably, infixtb can be selected as a time sequence database, and the infixtb supports a sql-like language, so that a monitoring system for writing a first configuration file in an sql language format can be supported to perform a data storage process, and user experience of a user using the sql language is improved.

And 612, the data display end extracts the second target data from the time sequence database through the visualization tool, and generates a monitoring chart corresponding to the second target data according to the time sequence of the first timestamp through the visualization tool, and displays the monitoring chart, wherein the abscissa of the monitoring chart is time.

Referring to fig. 2, in this step, the data processing terminal 40 sends the calculated second target data to the data display terminal 70, and the data display terminal 70 may display the second target data by using a visualization tool, such as a Grafana display tool, in the display/alarm module 10 after receiving the second target data.

Specifically, the second target data includes a first timestamp, the first timestamp includes generation time of the second target data, the data display end may generate a monitoring chart corresponding to the second target data according to a time sequence of the first timestamp by using the second target data through a visualization tool, and display the monitoring chart, where an abscissa of the monitoring chart is time, and reference is made to step 108 in the specific step, which is not repeated here.

Step 613, the data display end extracts third target data corresponding to the storage path and a second timestamp corresponding to the third target data from the second target data in the time sequence database according to a preset time interval.

Optionally, the first configuration file further includes: the alarm rule fields, e.g., the portions shown as D and E in FIG. 3, are two different alarm rule fields.

Optionally, the alarm rule field further includes: one or more of an alarm sequence number, an alarm name, and an aggregation level of an alarm aggregation function. Referring to fig. 3, the alarm sequence number in the alarm rule field D is: "rule: 1 ", the alarm name: "name: ' seating chart is lower than 75% '7 times in succession with idc success rate ', and the aggregation level of the alarm aggregation function: "Node: 5". The alarm sequence number is used for numbering and distinguishing a plurality of alarm rule fields when the plurality of alarm rule fields exist in the second configuration file; the alarm name is used for providing intuitive alarm type information for operation and maintenance personnel receiving the alarm information when the alarm information is generated so that the operation and maintenance personnel can intuitively judge the type of the received alarm information; and the aggregation level of the alarm aggregation function is used for aggregating the parameters in the aggregation level in the storage path when the alarm aggregation function is used for acquiring the third target data from the time sequence database according to the storage path, so that the aggregation calculation is accurately performed.

The alarm rule field includes: the alarm aggregation function, the storage path of the third target data corresponding to the alarm aggregation function, and the alarm trigger condition correspond to the alarm rule field D, and the alarm aggregation function is: "date: "asPercent (transmissive Null (exception { A }, 'success'),6, 'sum')), group pByNode ((exception { B }, { Node }), 'sum'))", and the storage path of the third target data corresponding to the alarm aggregation function is: "A:' count. One or more of alarm judgment relationship, alarm threshold, alarm filtering condition and alarm suppression rule, wherein the corresponding alarm judgment relationship is' operator: '<' ", the alarm threshold is" threshold: '75' ″, the alarm filtering condition is "filter: "7/7" ", the alarm suppression rule is" normal _ filter: "3/3".

Referring to fig. 2, in the embodiment of the present invention, the data display end 70 may generate an alarm task according to a plurality of first configuration files through an alarm task generation module, so as to call related data from the time sequence database according to the alarm task, and generate alarm information, so as to alarm operation and maintenance personnel when data of the service system is abnormal, and implement a monitoring and alarm process of the service system.

Optionally, step 613 specifically includes:

in the substep 6131, the data display end generates, according to a preset time interval, a query statement including the storage path according to the storage path of the third target data corresponding to the alarm aggregation function, and extracts, according to the query statement, the third target data and a second timestamp corresponding to the third target data from the time sequence database.

In this step, the data presentation end may generate, according to a storage path of third target data corresponding to the alarm aggregation function, an inquiry statement including the storage path, and extract, according to the inquiry statement and according to a preset time interval, corresponding third target data and a second timestamp corresponding to the third target data from the time sequence database.

Specifically, if the alarm aggregation function is date: "asPercent (transformNull (outer { A }, 'success'),6 'sum')), group pByNode ((outer { B }, { Node }), 'sum'))", and for a specific online ticketing service, the alarm aggregation function calculates the success rate of a partner bitmap request, and correspondingly, the storage path of the third target data corresponding to the alarm aggregation function is A: 'count. > and B:' count. > then the query statement generated is http://10.41.145.186: 8111/render? the target is asPercent (foreign ({ a }, 'success'),6, 'sum')), groupByNode ({ B },6, 'sum')) & from 11:43_20200112& unity ═ 12:13_20200112& maxDataPoints ═ 801, and the third target data required by the corresponding alarm aggregation function can be acquired from the time sequence database according to the query statement.

And 614, calculating the third target data by the data display end according to the alarm aggregation function to obtain alarm service data.

In this step, the data display end may perform aggregation calculation on the third target data acquired from the time sequence base according to the alarm aggregation function, so as to obtain alarm service data, and referring to the example in step 613, the obtained alarm service data is the success rate of the request of the seating chart of a certain partner.

And 615, displaying the alarm service data by the data display end through the visualization tool according to the time sequence of the second timestamp corresponding to the third target data.

In this step, the data display end may display the alarm service data by using a visualization tool after obtaining the alarm service data through calculation.

Specifically, the time stamp of the third target data of the alarm service data, that is, the time sequence of the second time stamp, may be obtained through calculation, and the master control end may generate an alarm chart corresponding to the alarm service data through the visualization tool according to the time sequence of the second time stamp, and display the alarm chart, where the abscissa of the alarm chart is time.

As shown in fig. 4, a curve H represents a change of the total power of all the partners with time, that is, the third target data acquired by the data presentation end from the time sequence library is the total request number of all the partners and the total successful request number of all the partners, and the corresponding alarm aggregation function is a ratio of the total successful request number of all the partners to the total request number of all the partners, so as to calculate the total power of all the partners of the alarm service data.

For example, for the online ticketing service, at 16:32, the total number of requests of all the partners is 2245, the total number of successful requests of all the partners is 2240, the number of failed requests of all the partners is 5, and according to an alarm aggregation function, the total power of all the partners is equal to the total number of successful requests of all the partners/the total number of requests of all the partners, the power of all the partners in a curve G is correspondingly calculated to be at 16:32, and the total power of all the partners in the system is 99.78%.

And 616, generating alarm information by the data display end under the condition that the alarm service data meets the alarm triggering condition.

Referring to fig. 2, in this step, the data display end may determine, through the Clarm alarm module in the display/alarm module 10, whether the calculated alarm service data meets an alarm trigger condition, and generate alarm information under the condition that the alarm service data meets the alarm trigger condition.

Specifically, referring to fig. 3, the alarm triggering condition includes: one or more of alarm judgment relationship, alarm threshold, alarm filtering condition and alarm suppression rule, wherein the corresponding alarm judgment relationship is' operator: '<' ", the alarm threshold is" threshold: '75' ″, the alarm filtering condition is "filter: "7/7", i.e. the corresponding data in the alert service: a certain partner obtains the request success rate of the seating chart, and if the request success rate meets the alarm triggering condition: if the number of times of the operation is less than 75% continuously, an alarm information schematic diagram shown in fig. 11 is generated, that is, for a module newticket in an application bmovie-I of a product line bmovieapp, service data of an acquisition seating chart (getseatinfo) of a partner (pname) J conforms to the alarm triggering condition, and the alarm information is generated.

Optionally, an alarm picture link may be generated under the condition that the alarm service data meets the alarm triggering condition, and after receiving the alarm picture link, the alarm information receiving side may obtain an alarm picture generated according to the alarm service data through the alarm picture link.

Step 617, the data display end sends the alarm information to an alarm information receiver corresponding to the communication information of the alarm information receiver when the alarm service data does not conform to the alarm suppression rule.

Optionally, the communication information of the alarm information receiver corresponds to the alarm suppression rule, in this step, the data display end determines, through the Clarm alarm module, whether the alarm service data conforms to the alarm suppression rule, and if the alarm service data does not conform to the alarm suppression rule, the alarm information generated in step 616 is sent to the alarm information receiver corresponding to the communication information of the alarm information receiver.

Specifically, the warning information receiver may be a maintenance person of the service system, and after the service system generates the warning information through the warning system, the maintenance person may receive the warning information in time, so as to monitor the service system in real time, and when the service system fails, receive the failure information in time, and maintain the service system according to the failure information.

In the embodiment of the invention, a maintainer can receive the warning information in a mode of mail, short message, WeChat and the like, and the corresponding communication information of the warning information receiver is the information of the maintainer such as mail address, mobile phone number, WeChat account number and the like.

Referring to fig. 3, the alarm suppression rule written in the first configuration file is "normal _ filter: "3/3", that is, under the condition that the alarm service data does not satisfy the alarm triggering condition for three consecutive times, it indicates that the service system is recovered to normal, so that the sending of the alarm information can be suppressed.

Step 618, the data display end stops sending the alarm information to the alarm information receiver when the alarm service data conforms to the alarm suppression rule.

In this step, the data display end judges whether the alarm service data conforms to an alarm suppression rule through a Clarm alarm module, if so, the service system is restored to normal, and the alarm information generated in step 616 is stopped being sent to the alarm information receiver corresponding to the alarm information receiver communication information.

In addition, the monitoring system can also be additionally provided with refined configurations such as alarm combination, advanced convergence, a mute (mute) time period, advanced alarm suppression low-level alarm and the like, so that more comprehensive configurations are set for the alarm function of the monitoring system, and the use experience of a user is improved.

On the basis of the above embodiments, the embodiments of the present invention further provide a service data monitoring apparatus.

Referring to fig. 12, a block diagram of a monitoring apparatus 700 for business data, which is applied to a monitoring system for business data according to an embodiment of the present invention, is shown, where the monitoring system includes: the main control terminal, the data processing terminal and the data display terminal may specifically include the following modules:

a first obtaining module 701, configured to obtain, by the master control end, a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data;

a first generating module 702, configured to generate, by the master control end, a data processing instruction including a data acquisition rule and the calculation rule according to the data acquisition rule field and the data processing rule field;

a first sending module 703, configured to send the data processing instruction to the data processing end by the master control end;

a first receiving module 704, configured to receive the data processing instruction sent by the master control end, and obtain, from a target data source, first target data corresponding to a data obtaining rule in the data processing instruction;

a first calculation module 705, configured to perform calculation processing on the first target data by the data processing end according to the calculation rule to obtain second target data;

a second sending module 706, configured to send the second target data to the data display end by the data processing end;

a second receiving module 707, configured to receive, by the data display end, the second target data sent by the data processing end;

the first display module 708 is configured to display the second target data through a visualization tool by the data display end.

To sum up, the monitoring device for service data provided by the present application is applied to a monitoring system for service data, the monitoring system includes: the main control end, the data processing end and the data display end comprise: the main control end obtains a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data; the main control terminal generates a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field; the main control end sends the data processing instruction to the data processing end; the data processing end receives a data processing instruction sent by the main control end, and acquires first target data corresponding to a data acquisition rule in the data processing instruction from a target data source; the data processing end carries out calculation processing on the first target data according to the calculation rule to obtain second target data; the data processing end sends the second target data to the data display end; the data display end receives second target data sent by the data processing end; and the data display end displays the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 13, a block diagram of a structure of a monitoring apparatus 800 for a service data, which is applied to a master control end in a monitoring system for a service data according to an embodiment of the present invention is shown, where the monitoring system further includes: the data processing end and the data display end may specifically include the following modules:

a second obtaining module 801, configured to obtain a first configuration file, where the first configuration file includes: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data;

a second generating module 802, configured to generate a data processing instruction including a data obtaining rule and the calculation rule according to the data obtaining rule field and the data processing rule field;

a third sending module 803, configured to send the data processing instruction to the data processing end, so that the data processing end obtains, from a target data source, first target data corresponding to the data obtaining rule, performs calculation processing on the first target data according to the calculation rule to obtain second target data, and further sends the second target data to the data display end, so that the data display end displays the second target data through a visualization tool.

Optionally, the data obtaining rule includes a regular expression corresponding to the first target data.

Optionally, the first configuration file is in a yaml format or a structured query language format.

To sum up, the monitoring device for service data provided by the present application is applied to a main control end in a monitoring system for service data, and the monitoring system further includes: data processing end and data show end include: acquiring a first configuration file, wherein the first configuration file comprises: a data acquisition rule field and a data processing rule field, the data processing rule field including a calculation rule for the first target data; generating a data processing instruction comprising a data acquisition rule and a calculation rule according to the data acquisition rule field and the data processing rule field; and sending the data processing instruction to a data processing end so that the data processing end can acquire first target data corresponding to the data acquisition rule from a target data source, and calculating the first target data according to the calculation rule to obtain second target data, and further sending the second target data to a data display end so that the data display end can display the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 14, a block diagram of a monitoring apparatus 900 for a service data, which is applied to a data processing end in a monitoring system for a service data according to an embodiment of the present invention, is shown, where the monitoring system further includes: the main control end and the data display end may specifically include the following modules:

a third receiving module 901, configured to receive a data processing instruction sent by the master control end, and obtain, from a target data source, first target data corresponding to a data obtaining rule in the data processing instruction, where the data processing instruction includes the data obtaining rule obtained from a data obtaining rule field in a first configuration file and a calculation rule obtained from a data processing rule field in the first configuration file;

a second calculating module 902, configured to perform calculation processing on the first target data according to the calculation rule to obtain second target data;

a fourth sending module 903, configured to send the second target data to the data display end, so that the data display end displays the second target data through a visualization tool.

Optionally, the data obtaining rule includes a regular expression corresponding to the first target data;

the third receiving module specifically includes:

and the first receiving submodule is used for acquiring first target data matched with the regular expression from the target data source.

Optionally, the apparatus further comprises:

a third obtaining module, configured to obtain a second configuration file, where the second configuration file includes a target data source type;

a fourth obtaining module, configured to obtain, according to the target data source type, multiple target data sources that match the target data source type from a data source, where the target data sources include a third timestamp, and the third timestamp includes a generation time of the target data source;

a fifth obtaining module, configured to obtain a plurality of preset time ranges;

and the storage module is used for adding the target data source with the third timestamp in the same preset time range to the same sub-target data source to obtain a plurality of sub-target data sources and storing the plurality of sub-target data sources in a local disk.

Optionally, the data processing end includes a plurality of computing units and a first aggregation unit, the data processing rule field further includes a first aggregation function, the number of the computing units is the same as the number of the sub-target data sources, and the computing units and the sub-target data sources are in one-to-one correspondence;

the third receiving module specifically includes:

the second receiving submodule is used for receiving the data processing instruction sent by the main control terminal and acquiring a plurality of first sub-target data corresponding to a data acquisition rule in the data processing instruction from a plurality of sub-target data sources;

the second calculation module specifically includes:

the first calculation submodule is used for respectively performing calculation processing on the plurality of first sub-target data according to a calculation rule in the data processing instruction through the plurality of calculation units to obtain a plurality of second sub-target data;

and the first aggregation submodule is used for performing aggregation operation on the plurality of second sub-target data through the first aggregation unit according to the first aggregation function to obtain the second target data.

To sum up, the monitoring device for service data provided by the present application is applied to a data processing end in a monitoring system for service data, and the monitoring system further includes: the master control end and the data display end comprise: receiving a data processing instruction sent by a main control terminal, and acquiring first target data corresponding to a data acquisition rule in the data processing instruction from a target data source, wherein the data processing instruction comprises the data acquisition rule acquired from a data acquisition rule field in a first configuration file and a calculation rule acquired from a data processing rule field in the first configuration file; calculating the first target data according to the calculation rule to obtain second target data; and sending the second target data to a data display end so that the data display end can display the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

Referring to fig. 15, a block diagram of a structure of a monitoring apparatus 1000 for business data at a data presentation end in a monitoring system for business data according to an embodiment of the present invention is shown, where the monitoring system further includes: the main control end and the data processing end may specifically include the following modules:

a fourth receiving module 1001, configured to receive second target data sent by the data processing end, where the second target data is data obtained by the data processing end obtaining, according to a data processing instruction sent by the master control end, first target data corresponding to a data obtaining rule in the data processing instruction from a target data source, and performing calculation processing on the first target data according to a calculation rule in the data processing instruction, where the data processing instruction is an instruction generated according to a data obtaining rule field and a data processing rule field in a first configuration file, and the data processing rule field includes a calculation rule for the first target data;

a second display module 1002, configured to display the second target data through a visualization tool.

Optionally, the second target data includes a first timestamp, and the first timestamp includes a generation time of the second target data;

the second display module specifically includes:

and the first display sub-module is used for generating a monitoring chart corresponding to the second target data according to the time sequence of the first time stamp by the second target data through the visualization tool and displaying the monitoring chart, wherein the abscissa of the monitoring chart is time.

Optionally, the data processing rule field further includes: a write path when the second target data is written into the time series database; the second target data comprises a first timestamp comprising a generation time of the second target data;

the fourth receiving module specifically includes:

the third receiving submodule is used for receiving the second target data sent by the data processing terminal and writing the second target data into the time sequence database according to the writing path and the time sequence of the first timestamp;

the second display module specifically includes:

and the second display sub-module is used for extracting the second target data from the time sequence database through the visualization tool, enabling the second target data to pass through the visualization tool, generating a monitoring chart corresponding to the second target data according to the time sequence of the first time stamp, and displaying the monitoring chart, wherein the abscissa of the monitoring chart is time.

Optionally, the first configuration file further includes: an alarm rule field, the alarm rule field comprising: the alarm aggregation function, a storage path of third target data corresponding to the alarm aggregation function and an alarm triggering condition;

the fourth receiving module further includes:

the extraction submodule is used for extracting third target data corresponding to the storage path and a second timestamp corresponding to the third target data from the second target data in the time sequence database according to a preset time interval;

the second calculation submodule is used for calculating the third target data according to the alarm aggregation function to obtain alarm service data;

the third display submodule is used for displaying the alarm service data according to the time sequence of the second timestamp corresponding to the third target data through the visualization tool;

and the generating submodule is used for generating alarm information under the condition that the alarm service data meets the alarm triggering condition.

Optionally, the extracting sub-module specifically includes:

and the extracting unit is used for generating a query statement containing the storage path according to the storage path of the third target data corresponding to the alarm aggregation function, and extracting the third target data and a second timestamp corresponding to the third target data from the time sequence database according to the query statement.

Optionally, the alarm rule field further includes: the communication information of the alarm information receiver and the alarm suppression rule;

the fourth receiving module further includes:

the first sending submodule is used for sending the alarm information to an alarm information receiver corresponding to the communication information of the alarm information receiver under the condition that the alarm service data does not accord with the alarm suppression rule;

and the second sending submodule is used for stopping sending the alarm information to the alarm information receiving party under the condition that the alarm service data accords with the alarm suppression rule.

Optionally, the alarm rule field further includes: one or more of an alarm sequence number, an alarm name and an aggregation level of an alarm aggregation function;

the alarm triggering conditions include: one or more of alarm judgment relationship, alarm threshold, alarm filtering condition and alarm suppression rule.

Optionally, the monitoring system for the service data includes a plurality of data processing terminals, the data display terminal includes a second aggregation unit, and the data processing rule field further includes a second aggregation function;

the fourth receiving module specifically includes:

a fourth receiving submodule, configured to receive multiple second target data sent by multiple data processing terminals;

the second aggregation sub-module is configured to perform aggregation operation on the plurality of second target data through the second aggregation unit according to the second aggregation function to obtain aggregated data;

the second display module, comprising:

and the fourth display submodule is used for displaying the aggregated data through the visualization tool.

To sum up, the monitoring device for service data provided by the present application is applied to a data display end in a monitoring system for service data, and the monitoring system further includes: the main control end and the data processing end comprise: receiving second target data sent by a data processing end, wherein the second target data is data obtained by the data processing end by acquiring first target data corresponding to a data acquisition rule in a data processing instruction from a target data source according to the data processing instruction sent by a main control end and calculating and processing the first target data according to a calculation rule in the data processing instruction, the data processing instruction is an instruction generated according to a data acquisition rule field and a data processing rule field in a first configuration file, and the data processing rule field comprises a calculation rule for the first target data; and displaying the second target data through a visualization tool. In the embodiment of the invention, related technicians of the service system can complete the configuration process of the monitoring system aiming at the service data only by writing the data acquisition rule field and the data processing rule field into the configuration file, the monitoring system of the service data can utilize the data acquisition rule field and the data processing rule field in the configuration file to acquire, calculate, store and display the service data after loading the configuration file, and the monitoring process of the service data can be realized, because the learning cost of the rule field is lower, and the user controls the service logic for interaction between the main control end and the data processing end, the user does not need to directly configure each component in the monitoring system, thereby avoiding the user from learning the bottom logic of each component in the monitoring system, only by generating the first configuration file, the method can realize the monitoring of the service data, simplify the configuration process of the monitoring system and reduce the learning cost of the monitoring system.

An embodiment of the present invention further provides an electronic device, as shown in fig. 16, including a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104, where the processor 1101, the communication interface 1102, and the memory 1103 complete communication with each other through the communication bus 1104,

a memory 1103 for storing a computer program;

the processor 1101 is configured to implement the following steps when executing the program stored in the memory 1103:

a monitoring method of service data is applied to a monitoring system of service data, and the monitoring system comprises: the method comprises the following steps that:

the data processing end sends the second target data to the data display end;

A monitoring method of service data is applied to a main control end in a monitoring system of service data, and the monitoring system further comprises: the method comprises a data processing end and a data display end, and comprises the following steps:

A monitoring method of service data is applied to a data processing end in a monitoring system of service data, and the monitoring system further comprises: the method comprises the following steps that:

A monitoring method of service data is applied to a data display end in a monitoring system of service data, and the monitoring system further comprises: the method comprises the following steps that:

and displaying the second target data through a visualization tool.

The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the terminal and other equipment.

The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.

In another embodiment of the present invention, a computer-readable storage medium is further provided, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer is caused to execute the method for monitoring service data in any of the above embodiments.

In a further embodiment of the present invention, there is also provided a computer program product containing instructions, which when run on a computer, causes the computer to execute the method for monitoring business data as described in any of the above embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for monitoring service data is applied to a system for monitoring service data, and the system comprises: the method comprises the following steps that:

the data processing end sends the second target data to the data display end;

2. A monitoring method of service data is characterized in that the method is applied to a main control end in a monitoring system of service data, and the monitoring system also comprises: the method comprises a data processing end and a data display end, and comprises the following steps:

3. The method of claim 2, wherein the data acquisition rule comprises a regular expression corresponding to the first target data.

4. The method of claim 2, wherein the first configuration file is in a yaml format or a structured query language format.

5. A monitoring method of service data is characterized in that the method is applied to a data processing end in a monitoring system of service data, and the monitoring system further comprises the following steps: the method comprises the following steps that:

6. The method of claim 5, wherein the data acquisition rule comprises a regular expression corresponding to the first target data;

the step of acquiring, from a target data source, first target data corresponding to a data acquisition rule in the data processing instruction specifically includes:

and acquiring first target data matched with the regular expression from the target data source.

7. The method according to claim 5, wherein before the step of receiving the data processing instruction sent by the master control end and acquiring the first target data corresponding to the data acquisition rule in the data processing instruction from the target data source, the method further comprises:

acquiring a second configuration file, wherein the second configuration file comprises a target data source type;

according to the target data source type, acquiring a plurality of target data sources matched with the target data source type from a data source, wherein the target data sources comprise third timestamps, and the third timestamps comprise the generation time of the target data sources;

acquiring a plurality of preset time ranges;

and adding the target data source with the third timestamp in the same preset time range to the same sub-target data source to obtain a plurality of sub-target data sources, and storing the plurality of sub-target data sources in a local disk.

8. The method according to claim 7, wherein the data processing end comprises a plurality of computing units and a first aggregation unit, the data processing rule field further comprises a first aggregation function, the number of the computing units is the same as the number of the sub-target data sources, and the computing units and the sub-target data sources are in one-to-one correspondence;

the step of receiving the data processing instruction sent by the master control end and acquiring first target data corresponding to a data acquisition rule in the data processing instruction from a target data source specifically includes:

receiving the data processing instruction sent by the master control terminal, and acquiring a plurality of first sub-target data corresponding to a data acquisition rule in the data processing instruction from a plurality of sub-target data sources;

the step of performing calculation processing on the first target data according to the calculation rule to obtain second target data specifically includes:

respectively carrying out calculation processing on the plurality of first sub-target data through the plurality of calculation units according to a calculation rule in the data processing instruction to obtain a plurality of second sub-target data;

and performing aggregation operation on the plurality of second sub-target data through the first aggregation unit according to the first aggregation function to obtain the second target data.

9. A monitoring method of service data is characterized in that the method is applied to a data display end in a monitoring system of service data, and the monitoring system further comprises the following steps: the method comprises the following steps that:

and displaying the second target data through a visualization tool.

10. The method of claim 9, wherein the second target data comprises a first timestamp comprising a generation time of the second target data;

the step of displaying the second target data through the visualization tool specifically includes:

and enabling the second target data to pass through the visualization tool, generating a monitoring chart corresponding to the second target data according to the time sequence of the first time stamps, and displaying the monitoring chart, wherein the abscissa of the monitoring chart is time.

11. The method of claim 9, wherein the data processing rule field further comprises: a write path when the second target data is written into the time series database; the second target data comprises a first timestamp comprising a generation time of the second target data;

the step of receiving the second target data sent by the data processing terminal specifically includes:

receiving the second target data sent by the data processing terminal, and writing the second target data into the time sequence database according to the writing path and the time sequence of the first timestamp;

and extracting the second target data from the time sequence database through the visualization tool, enabling the second target data to pass through the visualization tool, generating a monitoring chart corresponding to the second target data according to the time sequence of the first time stamp, and displaying the monitoring chart, wherein the abscissa of the monitoring chart is time.

12. The method of claim 11, wherein the first configuration file further comprises: an alarm rule field, the alarm rule field comprising: the alarm aggregation function, a storage path of third target data corresponding to the alarm aggregation function and an alarm triggering condition;

after the step of receiving the second target data sent by the data processing terminal and writing the second target data into the time sequence database according to the writing path and the time sequence of the first timestamp, the method further includes:

according to a preset time interval, extracting third target data corresponding to the storage path and a second timestamp corresponding to the third target data from the second target data in the time sequence database;

calculating the third target data according to the alarm aggregation function to obtain alarm service data;

displaying the alarm service data according to the time sequence of the second time stamps corresponding to the third target data through the visualization tool;

and generating alarm information under the condition that the alarm service data meets the alarm triggering condition.

13. The method according to claim 12, wherein the step of extracting third target data corresponding to the storage path and a second timestamp corresponding to the third target data from the second target data in the time-series database specifically includes:

and generating a query statement containing the storage path according to the storage path of the third target data corresponding to the alarm aggregation function, and extracting the third target data and a second timestamp corresponding to the third target data from the time sequence database according to the query statement.

14. The method of claim 12, wherein the alarm rule field further comprises: the communication information of the alarm information receiver and the alarm suppression rule;

after the step of generating the alarm information under the condition that the alarm service data meets the alarm triggering condition, the method further includes:

under the condition that the alarm service data does not accord with the alarm suppression rule, the alarm information is sent to an alarm information receiver corresponding to the communication information of the alarm information receiver;

and under the condition that the alarm service data accords with the alarm suppression rule, stopping sending the alarm information to the alarm information receiver.

15. The method of claim 12, wherein the alarm rule field further comprises: one or more of an alarm sequence number, an alarm name and an aggregation level of an alarm aggregation function;

16. The method according to claim 9, wherein the monitoring system for the business data comprises a plurality of data processing terminals, the data display terminal comprises a second aggregation unit, and the data processing rule field further comprises a second aggregation function;

receiving a plurality of second target data sent by a plurality of data processing terminals;

performing aggregation operation on the plurality of second target data through the second aggregation unit according to the second aggregation function to obtain aggregated data;

the step of displaying the second target data through a visualization tool includes:

displaying the aggregated data through the visualization tool.

17. A monitoring device for service data is applied to a monitoring system for service data, the monitoring system comprises: the device comprises a main control end, a data processing end and a data display end, and comprises:

18. A monitoring device of service data is characterized in that the monitoring device is applied to a main control end in a monitoring system of service data, and the monitoring system further comprises: data processing end and data show end, the device includes:

19. A monitoring device of service data is characterized in that the monitoring device is applied to a data processing end in a monitoring system of service data, and the monitoring system further comprises: master control end and data show end, the device includes:

20. A monitoring device of service data is characterized in that the monitoring device is applied to a data display end in a monitoring system of service data, and the monitoring system further comprises: master control end and data processing end, the device includes:

21. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

the memory is used for storing a computer program;

the processor, when executing the program stored in the memory, implementing the method steps of any of claims 1-16.

22. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-16.