CN112732531A - Monitoring data processing method and device - Google Patents

Monitoring data processing method and device Download PDF

Info

Publication number
CN112732531A
CN112732531A CN202110049853.4A CN202110049853A CN112732531A CN 112732531 A CN112732531 A CN 112732531A CN 202110049853 A CN202110049853 A CN 202110049853A CN 112732531 A CN112732531 A CN 112732531A
Authority
CN
China
Prior art keywords
data
monitoring data
alarm
processing
processed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110049853.4A
Other languages
Chinese (zh)
Inventor
王艳雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
21VIANET GROUP Inc
Original Assignee
21VIANET GROUP Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 21VIANET GROUP Inc filed Critical 21VIANET GROUP Inc
Priority to CN202110049853.4A priority Critical patent/CN112732531A/en
Publication of CN112732531A publication Critical patent/CN112732531A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
    • G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The utility model relates to the technical field of computer and communication, especially, relate to a processing method and device of monitoring data, solve the monitoring data in the different monitored control systems and be difficult to integrate, and current monitored control system adopts an engine, carries out the collection, handles and reports an emergency and asks for help or increased vigilance of monitoring data, influences the problem of the processing efficiency of monitoring data, and the method is: capturing monitoring data by adopting an acquisition engine, then asynchronously taking the monitoring data successfully matched with the alarm triggering rule and the matched alarm triggering rule thereof as data to be processed by adopting a processing engine, further asynchronously determining an alarm strategy corresponding to the alarm triggering rule in each piece of data to be processed by adopting the alarm engine, and executing the corresponding alarm strategy. Therefore, the acquisition engine, the processing engine and the alarm engine are respectively adopted to realize the acquisition and processing of the monitoring data, so that the acquisition process, the processing process and the alarm process of the monitoring data do not have strong correlation any more, and the processing speed and the processing efficiency of the monitoring data are greatly improved.

Description

Monitoring data processing method and device
Technical Field
The present disclosure relates to the field of computer and communication technologies, and in particular, to a method and an apparatus for processing monitoring data.
Background
In order to meet the increasingly complex business processing requirements of a company, a plurality of monitoring environments such as a server room, a containerized server room, a cloud server, etc. may exist in one company, which requires that corresponding monitoring systems be respectively deployed in different monitoring environments to monitor each monitored device existing in different monitoring environments, and commonly used monitoring systems include a promemetus system, a Zabbix system, a Nagios system, etc.
However, for each existing monitoring system, only one engine is used for achieving the functions of monitoring data acquisition, processing and alarming, and the engine in the existing monitoring system cannot be in butt joint with other external systems, that is, the engine in the monitoring system only supports data supply to the outside but does not support data acquisition from the outside, which greatly affects the processing speed and processing quality of the existing engine.
Disclosure of Invention
The embodiment of the disclosure provides a method and a device for processing monitoring data, which are used for solving the problems that monitoring data in different monitoring systems are difficult to integrate in the prior art, and the prior monitoring system adopts an engine to collect, process and alarm the monitoring data, so that the processing efficiency of the monitoring data is influenced.
The specific technical scheme provided by the embodiment of the disclosure is as follows:
in a first aspect, a method for processing monitoring data is provided, including:
capturing monitoring data collected by each data management node by adopting a collection engine, and asynchronously storing each monitoring data associated with a collection timestamp to a first message queue, wherein each data management node is deployed in different monitoring environments and is used for collecting operation information of monitored equipment in each monitoring environment, and the monitoring data comprises identification information of the monitored equipment;
asynchronously adopting a processing engine to match each piece of monitoring data in the first message queue with each stored alarm triggering rule, and storing the successfully matched monitoring data and the matched alarm triggering rule as data to be processed into a second message queue, wherein the data to be processed is associated with a processing time stamp;
and asynchronously adopting an alarm engine to obtain each piece of data to be processed in the second message queue, determining an alarm strategy corresponding to an alarm trigger rule in each piece of data to be processed, and executing a corresponding alarm strategy.
Optionally, the capturing of the monitoring data collected by each data management node by using the collection engine includes any one of the following modes or combinations:
capturing each data management node and collecting monitoring data in each monitoring environment by adopting an acquisition engine through an API (application programming interface), wherein the monitoring data comprises acquired time;
and accessing the databases of the data management nodes by adopting an acquisition engine, and capturing the monitoring data in the corresponding monitoring environment stored in each database, wherein the monitoring data comprises the acquired time.
Optionally, the asynchronously storing each monitoring data associated with the collection timestamp to the first message queue includes:
asynchronously storing each monitoring data associated with the acquisition time stamp to a first message queue and a first storage area;
the step of storing the successfully matched monitoring data and the matched alarm triggering rule thereof as the data to be processed to a second message queue includes:
screening out each monitoring data successfully matched, and determining an alarm triggering rule matched with each monitoring data;
and aiming at each screened monitoring data, determining that the monitoring data is successfully verified when target monitoring data which is the same as the monitoring data exists in the first storage area based on the acquisition timestamp associated with the monitoring data and the identification information of the monitored equipment, generating a piece of data to be processed based on the monitoring data and the alarm triggering rule matched with the monitoring data, and storing the data to be processed in a second message queue.
Optionally, the asynchronously adopting the processing engine to perform matching processing on each piece of monitoring data in the first message queue and each stored alarm triggering rule, including:
and the asynchronous adoption processing engine reads the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamp associated with the monitoring data, and performs matching processing on the read monitoring data and the stored alarm triggering rule, wherein the processing mark is added to the read monitoring data in the first message queue every time one monitoring data is read.
Optionally, the asynchronously acquiring each piece of to-be-processed data in the second message queue by using the alarm engine includes:
and asynchronously adopting an alarm engine, and acquiring the to-be-processed data which is not added with the acquisition mark in the second message queue according to the time sequence of the processing time stamps associated with the to-be-processed data, wherein the acquisition mark is added to the to-be-processed data acquired in the second message queue every time one to-be-processed data is acquired.
Optionally, the executing the corresponding alarm policy includes:
determining monitored equipment corresponding to each piece of data to be processed, and accessing an external system to acquire attribution information and a processing personnel set associated with each piece of monitored equipment, wherein the attribution information is used for assisting a processing personnel in positioning the monitored equipment;
and according to an alarm strategy corresponding to the monitored equipment, respectively sending alarm information at least comprising the attribution information and the data to be processed to the associated equipment of each group of processing personnel in the processing personnel set by taking a set time length as an interval until the pause alarm information fed back by any processing personnel in the processing personnel set based on the received alarm information is received.
Optionally, after the monitored device corresponding to each piece of to-be-processed data executes the corresponding alarm policy, the method further includes:
and storing the alarm triggering rules corresponding to the data to be processed and the execution records of the alarm strategies, and generating a monitoring report according to a set display format based on the stored monitoring data and the execution records of the data to be processed.
Optionally, the collection engine, the processing engine, and the alarm engine are respectively disposed in different containers.
In a second aspect, a device for processing monitoring data is provided, including:
the system comprises a capturing unit, a first message queue and a second message queue, wherein the capturing unit is used for capturing monitoring data collected by each data management node by adopting an acquisition engine and asynchronously storing each monitoring data associated with an acquisition timestamp into the first message queue, each data management node is deployed in different monitoring environments and is used for collecting operation information of monitored equipment in each monitoring environment, and the monitoring data comprises identification information of the monitored equipment;
the matching unit is used for asynchronously adopting a processing engine to match each piece of monitoring data in the first message queue with each stored alarm triggering rule, and storing the successfully matched monitoring data and the matched alarm triggering rule as data to be processed into a second message queue, wherein the data to be processed is associated with a processing time stamp;
and the execution unit is used for asynchronously adopting an alarm engine to acquire each piece of data to be processed in the second message queue, determining an alarm strategy corresponding to the alarm trigger rule in each piece of data to be processed, and executing the corresponding alarm strategy.
Optionally, when the collection engine is used to capture the monitoring data collected by each data management node, the capture unit performs capture of the monitoring data in any one of the following manners or a combination of the following manners:
capturing each data management node and collecting monitoring data in each monitoring environment by adopting an acquisition engine through an API (application programming interface), wherein the monitoring data comprises acquired time;
and accessing the databases of the data management nodes by adopting an acquisition engine, and capturing the monitoring data in the corresponding monitoring environment stored in each database, wherein the monitoring data comprises the acquired time.
Optionally, when the monitoring data associated with the collection timestamp is asynchronously stored in the first message queue, the capturing unit is configured to:
asynchronously storing each monitoring data associated with the acquisition time stamp to a first message queue and a first storage area;
when the successfully matched monitoring data and the matched alarm triggering rule thereof are used as data to be processed and stored in the second message queue, the matching unit is used for:
screening out each monitoring data successfully matched, and determining an alarm triggering rule matched with each monitoring data;
and aiming at each screened monitoring data, determining that the monitoring data is successfully verified when target monitoring data which is the same as the monitoring data exists in the first storage area based on the acquisition timestamp associated with the monitoring data and the identification information of the monitored equipment, generating a piece of data to be processed based on the monitoring data and the alarm triggering rule matched with the monitoring data, and storing the data to be processed in a second message queue.
Optionally, when the asynchronous processing engine matches each piece of monitoring data in the first message queue with each stored alarm triggering rule, the matching unit is configured to:
and the asynchronous adoption processing engine reads the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamp associated with the monitoring data, and performs matching processing on the read monitoring data and the stored alarm triggering rule, wherein the processing mark is added to the read monitoring data in the first message queue every time one monitoring data is read.
Optionally, when the asynchronous alert engine is used to obtain each piece of to-be-processed data in the second message queue, the execution unit is configured to:
and asynchronously adopting an alarm engine, and acquiring the to-be-processed data which is not added with the acquisition mark in the second message queue according to the time sequence of the processing time stamps associated with the to-be-processed data, wherein the acquisition mark is added to the to-be-processed data acquired in the second message queue every time one to-be-processed data is acquired.
Optionally, when the corresponding alarm policy is executed, the execution unit is configured to:
determining monitored equipment corresponding to each piece of data to be processed, and accessing an external system to acquire attribution information and a processing personnel set associated with each piece of monitored equipment, wherein the attribution information is used for assisting a processing personnel in positioning the monitored equipment;
and according to an alarm strategy corresponding to the monitored equipment, respectively sending alarm information at least comprising the attribution information and the data to be processed to the associated equipment of each group of processing personnel in the processing personnel set by taking a set time length as an interval until the pause alarm information fed back by any processing personnel in the processing personnel set based on the received alarm information is received.
Optionally, after the monitored device corresponding to each piece of to-be-processed data executes the corresponding alarm policy, the execution unit is further configured to:
and storing the alarm triggering rules corresponding to the data to be processed and the execution records of the alarm strategies, and generating a monitoring report according to a set display format based on the stored monitoring data and the execution records of the data to be processed.
Optionally, the collection engine, the processing engine, and the alarm engine are respectively disposed in different containers.
In a third aspect, an electronic device is provided, including:
a memory for storing executable instructions;
a processor, configured to read and execute the executable instructions stored in the memory, so as to implement the method for processing monitoring data according to any one of the first aspect.
In a fourth aspect, a computer-readable storage medium is provided, in which instructions, when executed by an electronic device, enable the electronic device to perform the method for processing monitoring data according to any one of the first aspect.
The beneficial effects of this disclosure are as follows:
in the embodiment of the disclosure, an acquisition engine is adopted to capture monitoring data collected by each data management node, and each monitoring data associated with an acquisition timestamp is asynchronously stored in a first message queue, wherein each data management node is deployed in different monitoring environments and is used for collecting operation information of monitored equipment included in each monitoring environment, the monitoring data includes identification information of the monitored equipment, then, a processing engine is adopted asynchronously to perform matching processing on each monitoring data in the first message queue and each stored alarm triggering rule, the successfully matched monitoring data and the matched alarm triggering rule thereof are used as to-be-processed data and stored in a second message queue, wherein the to-be-processed data is associated with a processing timestamp, and then, the alarm engine is adopted asynchronously to obtain each to-be-processed data in the second message queue, and determining an alarm strategy corresponding to the alarm triggering rule in each piece of data to be processed, and executing a corresponding alarm strategy. Thus, the integrated processing of the monitoring data under different monitoring environments can be realized by capturing the monitoring data in different monitoring environments collected by each data management node, and the acquisition and processing of the monitoring data are realized by respectively adopting the acquisition engine, the processing engine and the alarm engine, so that the acquisition process, the processing process and the alarm process of the monitoring data do not have strong association any more, the data processing and the alarm can be asynchronously carried out while the monitoring data are acquired, the processing speed and the processing efficiency of the monitoring data are greatly improved, secondly, the acquisition efficiency of the data can also be improved to a certain extent by storing the monitoring data and the data to be processed into a message queue, moreover, the stored alarm triggering rules and the alarm strategies are uniformly adopted to process the data stored in the message queue, and the repeated configuration of the alarm triggering rules and the alarm strategies in the prior art is avoided, the data reading speed is improved, and the running conditions of each monitored device in different monitoring environments can be effectively obtained.
Drawings
FIG. 1 is a schematic diagram of an interaction architecture for monitoring data processing according to an embodiment of the present disclosure;
FIG. 2 is a schematic flow chart illustrating the process of monitoring data according to an embodiment of the disclosure;
FIG. 3 is a schematic view of a set of process personnel in an embodiment of the disclosure;
FIG. 4 is a schematic diagram of a monitoring report generated in an embodiment of the present disclosure;
FIG. 5 is a system hierarchy diagram of a processing device in an embodiment of the disclosure;
FIG. 6 is a schematic diagram of a logic structure of a processing apparatus for monitoring data according to an embodiment of the present disclosure;
fig. 7 is a schematic physical structure diagram of a processing apparatus for monitoring data according to an embodiment of the present disclosure.
Detailed Description
In order to make the purpose, technical solution and beneficial effects of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the disclosure and are not intended to limit the disclosure.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
In order to solve the problem that monitoring data in different monitoring systems are difficult to integrate in the prior art, and the prior monitoring system adopts an engine to collect, process and alarm the monitoring data, which affects the processing efficiency of the monitoring data, the disclosure provides a processing method of the monitoring data, which adopts an acquisition engine to capture the monitoring data collected by each data management node and asynchronously store each monitoring data associated with an acquisition timestamp into a first message queue, wherein each data management node is deployed in different monitoring environments and is used for collecting the operation information of the monitored equipment included in each monitoring environment, the monitoring data includes the identification information of the monitored equipment, then, each monitoring data in the first message queue is asynchronously matched with each stored alarm triggering rule by the processing engine, and taking the successfully matched monitoring data and the matched alarm triggering rules thereof as data to be processed, storing the data to be processed into a second message queue, wherein a processing time stamp is associated with the data to be processed, asynchronously adopting an alarm engine to acquire each piece of data to be processed in the second message queue, determining an alarm strategy corresponding to the alarm triggering rule in each piece of data to be processed, and executing a corresponding alarm strategy.
In the embodiment of the present disclosure, reference is made to fig. 1, which is a schematic diagram of an interaction architecture of monitoring data processing in the implementation of the present disclosure. The interactive architecture at least comprises a data management node and a processing device, wherein,
the monitoring system comprises data management nodes, a monitoring device and a monitoring system, wherein the data management nodes are deployed in a monitoring environment and used for collecting operation data of each monitored device in the monitoring environment, specifically, one data management node is deployed in one monitoring environment and used for collecting monitoring data of the monitored device reported by each acquisition node managed in the corresponding monitoring environment, the acquisition nodes are deployed on the monitored device, one acquisition node is deployed on one monitored device, and the monitoring data comprises the acquired operation data of the monitored device and acquisition time of the operation data.
It should be noted that in the implementation of the present disclosure, the data management node may specifically be a server of an existing monitoring system tool, such as a server of a Prometheus monitoring tool, a server of a Zabbix monitoring tool, a server of a Nagios monitoring tool, a server of an openface monitoring tool, and the like. The existing monitoring tool generally comprises a collection end and a service end, wherein the service end is used for collecting monitoring data collected by each collection end in the monitoring environment where the service end is located.
The processing equipment is used for realizing the functions by an acquisition engine, a processing engine and an alarm engine which are independent from each other, wherein the acquisition engine is used for capturing monitoring data collected by each data management node in real time or periodically and asynchronously storing the collected monitoring data into a first message queue; the processing engine acquires the monitoring data in the first message queue in an asynchronous processing mode with the acquisition engine, matches the monitoring data with the stored alarm triggering rules, takes the monitoring data meeting the alarm triggering rules and the matched alarm triggering rules as data to be processed and stores the data to the second message queue; and the alarm engine asynchronously acquires each piece of data to be processed in the second message queue, determines an alarm strategy corresponding to the data to be processed, and executes the corresponding alarm strategy.
It should be noted that, in the embodiment of the present disclosure, according to actual configuration needs, the acquisition engine, the processing engine, and the alarm engine may be respectively disposed in different containers to implement automatic capacity reduction and capacity expansion by using the characteristics of the containers themselves, and in some embodiments, the alarm engine may also interface with an external system by using an API interface to obtain data in the external system.
Preferred embodiments of the disclosed embodiments are described in further detail below with reference to the accompanying drawings:
referring to fig. 2, which is a schematic view of a processing flow of monitoring data in an embodiment of the present disclosure, a description is given below, with reference to fig. 2, of a processing process of monitoring data by a processing device in an implementation of the present disclosure.
Step 201: and capturing the monitoring data collected by each data management node by adopting an acquisition engine, and asynchronously storing each monitoring data associated with an acquisition timestamp to a first message queue.
The processing device captures the monitoring data collected by each data management node by using an acquisition engine, and specifically, may capture the monitoring data by using any one or a combination of the following manners including, but not limited to.
The method comprises the steps of adopting an acquisition engine to capture each data management node through an API (application programming interface) interface and collecting monitoring data in each monitoring environment, wherein the monitoring data comprises acquired time.
The processing device captures the monitoring data collected by each data management node through an Application Programming Interface (API) capable of acquiring the monitoring data collected by the data management nodes by using a collection engine. The data management nodes are deployed in different monitoring environments and used for collecting operation information of monitored equipment included in the respective monitoring environments, the monitoring data include identification information of the monitored equipment, and the API (application programming interface) is an access interface provided by each data management node, so that the processing equipment adopts an acquisition engine and can capture the monitoring data collected by each data management node through the access API.
For example, suppose that a server room and a containerized server exist inside a company X, and a monitoring environment of the server room is monitored by using a Zabbix monitoring tool, and a monitoring environment of the containerized server is monitored by using a promemeus monitoring tool, taking the monitoring of promemeus as an example, each acquisition end of promemeus is deployed on each monitored server in the containerized server and reports the acquired monitoring data to a service end of promemeus, and the promemeus monitoring tool is externally provided with an API interface, so that a processing device can acquire the monitoring data collected by promemeus by using an acquisition engine, where the monitoring data includes acquired time.
And accessing the databases of the data management nodes by adopting an acquisition engine, and capturing the monitoring data in the corresponding monitoring environment stored in each database, wherein the monitoring data comprises the acquired time.
The processing device accesses the databases of the data management nodes by adopting the acquisition engine, and captures the monitoring data in the respective monitoring environments stored in the databases, that is, the processing device can directly access the databases of the data management nodes to acquire the monitoring data stored in the databases.
Therefore, the monitoring data can be obtained by accessing the API interface or the database, which is equivalent to obtaining the monitoring data in different monitoring environments, and the integration of the monitoring data collected in different monitoring environments is realized.
It should be noted that, in the embodiment of the present disclosure, the collection engines may be disposed in the container, so that the number of the collection engines in the container can be adjusted by means of the characteristics of the container according to the actual processing requirement.
In some embodiments of the present disclosure, the acquisition engine may be used to capture the monitoring data periodically, specifically, assuming that the set time length is used as a period to capture the monitoring data collected by each data processing node, the associated acquisition time of the monitoring data captured each time is between the last capture period and the capture period, for example, assuming that the set time length is 15min, that is, the capture of the monitoring data is performed every 15min, and the currently captured monitoring data is generated within the past 15 min. In other embodiments of the present disclosure, the collection engine may capture the monitoring data collected by each data management node in real time, so as to implement real-time processing.
Further, the processing device asynchronously stores each monitoring data associated with the collection timestamp to the first message queue, that is, the processing device asynchronously performs capturing operation and storing operation on the monitoring data, so that the capturing and storing of the monitoring data do not have strong correlation, and different operations on the monitoring data do not interfere with each other.
It should be noted that, in the embodiment of the present disclosure, each piece of monitoring data associated with the collection timestamp may be asynchronously stored in the first message queue and the first storage area, that is, while the monitoring data is asynchronously written into the first message queue, the monitoring data may be synchronously written into the first storage area, which is equivalent to storing the monitoring data to a location capable of persistently storing, for example, in a magnetic disk, so as to facilitate subsequent backtracking of the monitoring data of the monitored device.
Therefore, the captured monitoring data is asynchronously stored in the message queue, the strong association between the capturing and the storing of the monitoring data is eliminated, the data acquisition efficiency is improved to a certain extent, the data reading speed can be improved to a certain extent by storing the monitoring data in the message queue, and support is provided for the rapid processing of the monitoring data.
Step 202: and asynchronously adopting a processing engine to match each piece of monitoring data in the first message queue with each stored alarm triggering rule, and storing the successfully matched monitoring data and the matched alarm triggering rule thereof as data to be processed to a second message queue, wherein the data to be processed is associated with a processing time stamp.
Specifically, the processing device asynchronously adopts the processing engine to read the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamp associated with the monitoring data, and performs matching processing on the read monitoring data and the stored alarm triggering rule, wherein each time one monitoring data is read, the processing mark is added to the read monitoring data in the first message queue.
That is to say, when the processing device stores the monitoring data in the first message queue, the monitoring data in the first message queue is asynchronously read, in other words, the operation of storing the monitoring data in the first message queue and the operation of reading the monitoring data stored in the first message queue are asynchronously performed, and there is no relationship between them, and the processing device can read the data in the first message queue and process the data while continuously storing the data in the first message queue. When the monitoring data is read, the processing device reads the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamps associated with the monitoring data in the first message queue, that is, the monitoring data with the acquisition time before the acquisition time corresponding to the acquisition time stamp is read first, and in order to ensure that the monitoring data in the first message queue is not read repeatedly, each time one monitoring data is read, the processing mark is added to the monitoring data read in the first message queue, and each piece of the read monitoring data is matched with each stored alarm triggering rule.
In the embodiment of the present disclosure, the processing engine may be disposed in the container, so that the number of the processing engines in the container may be adjusted by using the characteristics of the container according to an actual processing requirement, in the present disclosure, according to an actual processing requirement, a plurality of processing engines may be used to perform matching processing on each piece of monitoring data in the first message queue and each stored alarm triggering rule in parallel, where a container in which the processing engine is disposed is different from a container in which the acquisition engine is disposed.
In the embodiment of the present disclosure, the related alarm triggering rules include, but are not limited to: the memory occupancy rate exceeds each set threshold value, the CPU utilization rate reaches each set threshold value, and the like, the alarm trigger rule adopted in the present disclosure may follow the alarm trigger rule existing in the existing monitoring system, and the present disclosure is not specifically limited herein.
Further, the processing device adopts a processing engine to store the successfully matched monitoring data and the matched alarm triggering rule thereof as to-be-processed data to a second message queue, wherein the to-be-processed data is associated with a processing time stamp.
Specifically, the processing device performs matching processing on each piece of monitoring data by using a processing engine, and when it is determined that the monitoring data matches a stored alarm triggering rule, the monitoring data and the alarm triggering rule matching the monitoring data serve as data to be processed, and the data to be processed associated with a processing timestamp is stored in a second message queue, wherein the processing device may store a plurality of alarm triggering rules, and for an obtained piece of monitoring data, when it is determined that the monitoring data matches any one of the alarm triggering rules successfully, it is determined that the monitoring data matches each stored alarm triggering rule successfully.
It should be noted that, in the embodiment of the present disclosure, when determining a matched alarm trigger rule, a closest alarm trigger rule in the same type may be used as the matched alarm trigger rule, for example, if the alarm trigger rule includes conditions that a CPU usage rate reaches 50%, a CPU usage rate reaches 60%, a CPU usage rate reaches 70%, and the like, if the CPU usage rate in the current monitoring data is 85%, it is known that the current monitoring data satisfies three alarm trigger rules in the alarm type of the CPU usage rate, and when determining the matched alarm trigger rule, it is considered that the CPU usage rate reaches 70% as the matched alarm trigger rule.
In some embodiments of the present disclosure, based on the second message queue storing the data to be processed, the processing device may analyze, according to the actual processing needs, the condition that the executed monitored device triggers an alarm within a certain time, which is helpful for performing failure analysis and troubleshooting on the monitored device.
Therefore, the processing engine different from the acquisition engine is adopted, the monitoring data and each stored alarm triggering rule can be matched, the acquisition process and the processing process of the monitoring data can be asynchronously carried out, no sequential strong association relation exists between the acquisition process and the processing process, and the problem of low processing efficiency caused by the same engine for the acquisition and processing functions of the monitoring data in the prior art is solved.
It should be noted that, in some embodiments of the present disclosure, in a case that monitoring data is stored in the first message queue and the first storage area, after the processing device determines that the monitoring data and the alarm trigger rule are successfully matched, and before the processing device stores the monitoring data and the alarm trigger rule successfully matched therewith as data to be processed in the second message queue, the processing device may screen out each monitoring data successfully matched, determine the alarm trigger rule respectively matched with each monitoring data, determine, for each screened monitoring data, when it is determined that target monitoring data identical to the monitoring data exists in the first storage area based on the acquisition timestamp associated with the monitoring data and the identification information of the monitored device included in the monitoring data, determine that the monitoring data is successfully verified, and generate a piece of data to be processed based on the monitoring data and the alarm trigger rule matched therewith, and storing the message to the second message queue.
Therefore, the reliability of the monitoring data can be ensured to a certain extent, errors of the data to be processed stored in the second message queue are avoided, and effective processing of the monitoring data is ensured.
Step 203: and asynchronously adopting an alarm engine to obtain each piece of data to be processed in the second message queue, determining an alarm strategy corresponding to an alarm trigger rule in each piece of data to be processed, and executing a corresponding alarm strategy.
The processing equipment adopts the processing engine to match each stored alarm triggering rule, stores the successfully matched monitoring data and the matched alarm triggering rule thereof in the second message queue, and synchronously adopts the alarm engine to acquire each piece of data to be processed in the second message queue asynchronously, and determines the alarm strategy corresponding to the alarm triggering rule in each piece of data to be processed.
Specifically, the processing device asynchronously adopts an alarm engine to acquire the to-be-processed data, to which the acquisition marker is not added, in the second message queue according to the time sequence of the processing time stamp associated with the to-be-processed data, wherein each time one to-be-processed data is acquired, the acquisition marker is added to the to-be-processed data acquired in the second message queue.
That is to say, while continuously storing the to-be-processed data in the second message queue, the processing device asynchronously reads the to-be-processed data in the second message queue by using the alarm engine, wherein in the process of reading the to-be-processed data, the to-be-processed data which is not added with the acquisition mark in the second message queue is read according to the time sequence of the processing time stamp associated with the to-be-processed data, that is, the to-be-processed data which corresponds to the processing time stamp is acquired first, and in order to ensure that the to-be-processed data in the second message queue is not repeatedly acquired, each time one to-be-processed data is acquired, the to-be-processed data acquired in the second message queue is added with the acquisition mark.
Further, the processing device determines an alarm policy corresponding to an alarm trigger rule in each piece of data to be processed by using an alarm engine, and executes a corresponding alarm policy.
In some embodiments of the present disclosure, rule identification information (Identity, ID) may be configured for each alarm trigger rule, and a corresponding policy ID may be configured for each alarm policy, and the alarm trigger rule stored in the information to be processed is embodied in a rule ID form, so that the processing device may subsequently determine a corresponding policy ID according to the rule ID, and determine a corresponding alarm policy according to the obtained policy ID.
It should be noted that, in the embodiment of the present disclosure, the alarm engines may be deployed in the container, so that the number of the alarm engines in the container can be adjusted by using the characteristics of the container according to the actual processing requirement.
When the processing device executes a corresponding alarm policy for each piece of to-be-processed data by using the alarm engine, in some embodiments of the present disclosure, corresponding alarm information may be generated for a monitored device corresponding to each piece of to-be-processed data, and the alarm information may be presented to the stored relevant personnel. In other embodiments of the present disclosure, the processing device determines, by using an alarm engine, a monitored device corresponding to each piece of data to be processed, and accesses an external system to obtain attribution information and a processing staff set associated with each monitored device, where the attribution information is used to assist a processing staff in positioning the monitored device, and then sends alarm information at least including the attribution information and the data to be processed to associated devices of each group of processing staff in the processing staff set at intervals of a set time length according to an alarm policy corresponding to the monitored device, until receiving suspended alarm information fed back by any processing staff in the processing staff set based on the received alarm information.
That is to say, in the present disclosure, the processing device may establish data interaction with an external system through an alarm engine, and acquire the attribution information of the monitored device provided by the external system and a processing staff set associated with the monitored device, so that a processing staff receiving the attribution information can accurately locate the position of the monitored device, where the processing staff set includes various sets of processing staff, where an intersection exists between different sets of processing staff, and the external system capable of performing interaction includes a system capable of providing the position information of the monitored device and a user system capable of providing the processing staff set associated with the monitored device.
Referring to fig. 3, which is a schematic diagram of a set of processing persons in the embodiment of the present disclosure, a set of data persons includes different groups of processing persons, where the different groups include different processing persons, a group including the smallest number of processing persons is first notified when sending an alarm message, and as the time required for an alarm is detected to increase, when it is determined that no response of any processing person is received, a notification range of the processing person is gradually increased according to the set processing persons, for example, each processing person in a first set of processing persons is first notified, if no response of a relevant processing person is received within a certain time, each processing person in a second set of processing persons is further notified, and if no response of a relevant person is received within a certain time, each processing person in a third set of processing persons is notified, the second processing personnel set comprises a first processing personnel set, and the third processing personnel set comprises a second processing personnel set. Correspondingly, after receiving notification suspension information indicated by the notified processing personnel, stopping sending the warning information to the processing personnel, wherein the warning information at least comprises the attribution information and the data to be processed.
Further, in the embodiment of the present disclosure, the processing device stores the alarm triggering rules and the execution records of the alarm policies corresponding to each piece of to-be-processed data, and generates the monitoring report according to the set display format based on the stored monitoring data and the execution records of the to-be-processed data.
Specifically, the processing device stores, for each piece of to-be-processed data, a corresponding alarm trigger rule and an execution record of an alarm policy, and performs statistics and analysis according to a set display format based on the stored monitoring data and the execution record of each piece of to-be-processed data to generate a monitoring report.
For example, referring to fig. 4, which is a schematic diagram of a monitoring report generated in the embodiment of the present disclosure, a processing device may analyze obtained monitoring data and an execution record of an alarm policy, determine each monitored device meeting an alarm trigger rule, and after performing statistics, may set the monitored device meeting a certain number of alarm trigger rules as a serious alarm according to the number of alarm trigger rules met by the monitored device, meanwhile, the alarm trigger rules for the alarm conditions within a certain time, such as CPU conditions, network conditions, memory conditions, storage conditions, hard disk conditions and the like, moreover, the processing device analyzes the index targeted by the alarm triggering rule, such as an exemplary monitored device presenting the host alarm 5 top, or an optional monitored device presenting the CPU utilization 5 top.
Based on the same inventive concept, referring to fig. 5, which is a system layered architecture diagram of a processing device in the embodiment of the present disclosure, the processing device may be deployed hierarchically, and is a data acquisition layer where an acquisition engine is located, a data processing layer where the processing engine is located, and an alarm notification layer where an alarm engine is located, where each layer supports containerization and distributed deployment operations, and high-concurrency and fast processing of mass data is ensured by using fast capacity expansion and capacity contraction characteristics of a container, and operations among different acquisition engines, processing engines, and alarm engines are asynchronous, that is, operations among the various engines are mutually decoupled, so that efficiency of monitoring data processing and timeliness of alarm are ensured.
In the system illustrated in fig. 5, the collecting engine obtains the monitoring data collected by each data management node, asynchronously writes each monitoring data associated with the collecting timestamp into the first storage area and the first message queue, further the processing device asynchronously adopts the processing engine to obtain the stored alarm rule and the monitoring data in the first message queue, further stores the successfully matched alarm trigger rule and the corresponding monitoring data as the data to be processed into the second message queue, and asynchronously adopts the alarm engine to obtain the data to be processed in the second message queue, and determines the alarm policy matched with the alarm trigger rule in each data to be processed, at the same time, the location information of the monitored device and the associated processing personnel information can be obtained through the external system, further the alarm policy can be executed in a targeted manner, and the alarm records and the execution records are stored, and further, a monitoring report meeting the requirements can be generated in a background task mode according to the stored alarm records and the execution records.
Based on the same inventive concept, referring to fig. 6, which is a schematic diagram of a logical structure of a processing apparatus for monitoring data in the embodiment of the present disclosure, the processing apparatus includes a capturing unit 601, a matching unit 602, and an executing unit 603, wherein,
a capturing unit 601, configured to capture, by using an acquisition engine, monitoring data collected by each data management node, and asynchronously store each monitoring data associated with an acquisition timestamp in a first message queue, where each data management node is deployed in different monitoring environments and is configured to collect operation information of a monitored device included in each monitoring environment, and the monitoring data includes identification information of the monitored device;
a matching unit 602, configured to asynchronously adopt a processing engine to perform matching processing on each piece of monitoring data in the first message queue and each stored alarm trigger rule, and store, as to-be-processed data, the monitoring data that is successfully matched and the alarm trigger rule that is matched therewith, to a second message queue, where the to-be-processed data is associated with a processing timestamp;
the executing unit 603 is configured to asynchronously acquire each piece of to-be-processed data in the second message queue by using an alarm engine, determine an alarm policy corresponding to an alarm trigger rule in each piece of to-be-processed data, and execute a corresponding alarm policy.
Optionally, when the collection engine is used to capture the monitoring data collected by each data management node, the capture unit 601 performs capture of the monitoring data by using any one of the following manners or a combination of the following manners:
capturing each data management node and collecting monitoring data in each monitoring environment by adopting an acquisition engine through an API (application programming interface), wherein the monitoring data comprises acquired time;
and accessing the databases of the data management nodes by adopting an acquisition engine, and capturing the monitoring data in the corresponding monitoring environment stored in each database, wherein the monitoring data comprises the acquired time.
Optionally, when the monitoring data associated with the collection timestamp is asynchronously stored in the first message queue, the capturing unit 601 is configured to:
asynchronously storing each monitoring data associated with the acquisition time stamp to a first message queue and a first storage area;
when the monitoring data successfully matched and the alarm triggering rule matched therewith are used as data to be processed and stored in the second message queue, the matching unit 602 is configured to:
screening out each monitoring data successfully matched, and determining an alarm triggering rule matched with each monitoring data;
and aiming at each screened monitoring data, determining that the monitoring data is successfully verified when target monitoring data which is the same as the monitoring data exists in the first storage area based on the acquisition timestamp associated with the monitoring data and the identification information of the monitored equipment, generating a piece of data to be processed based on the monitoring data and the alarm triggering rule matched with the monitoring data, and storing the data to be processed in a second message queue.
Optionally, when the asynchronous processing engine matches each piece of monitoring data in the first message queue with each stored alarm triggering rule, the matching unit 602 is configured to:
and the asynchronous adoption processing engine reads the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamp associated with the monitoring data, and performs matching processing on the read monitoring data and the stored alarm triggering rule, wherein the processing mark is added to the read monitoring data in the first message queue every time one monitoring data is read.
Optionally, when the asynchronous alert engine is used to obtain each piece of to-be-processed data in the second message queue, the execution unit 603 is configured to:
and asynchronously adopting an alarm engine, and acquiring the to-be-processed data which is not added with the acquisition mark in the second message queue according to the time sequence of the processing time stamps associated with the to-be-processed data, wherein the acquisition mark is added to the to-be-processed data acquired in the second message queue every time one to-be-processed data is acquired.
Optionally, when the corresponding alarm policy is executed, the executing unit 603 is configured to:
determining monitored equipment corresponding to each piece of data to be processed, and accessing an external system to acquire attribution information and a processing personnel set associated with each piece of monitored equipment, wherein the attribution information is used for assisting a processing personnel in positioning the monitored equipment;
and according to an alarm strategy corresponding to the monitored equipment, respectively sending alarm information at least comprising the attribution information and the data to be processed to the associated equipment of each group of processing personnel in the processing personnel set by taking a set time length as an interval until the pause alarm information fed back by any processing personnel in the processing personnel set based on the received alarm information is received.
Optionally, after the monitored device corresponding to each piece of to-be-processed data executes the corresponding alarm policy, the executing unit 603 is further configured to:
and storing the alarm triggering rules corresponding to the data to be processed and the execution records of the alarm strategies, and generating a monitoring report according to a set display format based on the stored monitoring data and the execution records of the data to be processed.
Optionally, the collection engine, the processing engine, and the alarm engine are respectively disposed in different containers.
Based on the same inventive concept, referring to fig. 7, it is a schematic block diagram of a processing apparatus for monitoring data according to an embodiment of the present disclosure, which includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732 for storing instructions, such as an application program, executable by the processing component 722. The application programs stored in memory 732 may include one or more modules that each correspond to a set of instructions. Further, the processing component 722 is configured to execute instructions to perform the above-described methods.
The apparatus 700 may also include a power component 726 configured to perform power management of the apparatus 700, a wired or wireless network interface 750 configured to connect the apparatus 700 to a network, and an input output (I/O) interface 758. The apparatus 700 may operate based on an operating system stored in memory 732, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.
Based on the same inventive concept, embodiments of the present disclosure provide a storage medium in which instructions are executed by an electronic device, so that the electronic device can perform any one of the above methods.
To sum up, in the embodiment of the present disclosure, an acquisition engine is used to capture monitoring data collected by each data management node, and each monitoring data associated with an acquisition timestamp is asynchronously stored in a first message queue, where each data management node is deployed in different monitoring environments and is used to collect operation information of a monitored device included in each monitoring environment, the monitoring data includes identification information of the monitored device, then, a processing engine is used asynchronously to perform matching processing on each monitoring data in the first message queue and each stored alarm trigger rule, and the monitoring data that is successfully matched and the alarm trigger rule that is matched are used as to-be-processed data and stored in a second message queue, where the to-be-processed data is associated with a processing timestamp, and then, an alarm engine is used asynchronously to obtain each to-be-processed data in the second message queue, and determining an alarm strategy corresponding to the alarm triggering rule in each piece of data to be processed, and executing a corresponding alarm strategy. Thus, the integration processing of the monitoring data under different monitoring environments can be realized by capturing the monitoring data in different monitoring environments collected by each data management node, and the acquisition and processing of the monitoring data are realized by respectively adopting the acquisition engine, the processing engine and the alarm engine, so that the acquisition and processing of the monitoring data do not have strong correlation with the acquisition process, the processing process and the alarm process of the monitoring data, the data processing and the alarm can be asynchronously carried out while the monitoring data are acquired, the processing speed and the processing efficiency of the monitoring data are greatly improved, secondly, the acquisition efficiency of the data can be improved to a certain extent by storing the monitoring data and the data to be processed into a message queue, moreover, the data stored in the message queue are processed by uniformly adopting the stored alarm triggering rules and the alarm strategies, and the repeated configuration of the alarm rules and the alarm strategies under the prior art is avoided, the data reading speed is improved, and the running conditions of each monitored device in different monitoring environments can be effectively obtained.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present disclosure have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the disclosure.
It will be apparent to those skilled in the art that various changes and modifications may be made to the disclosed embodiments without departing from the spirit and scope of the disclosed embodiments. Thus, if such modifications and variations of the embodiments of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to encompass such modifications and variations.

Claims (18)

1. A method for processing monitoring data, comprising:
capturing monitoring data collected by each data management node by adopting a collection engine, and asynchronously storing each monitoring data associated with a collection timestamp to a first message queue, wherein each data management node is deployed in different monitoring environments and is used for collecting operation information of monitored equipment in each monitoring environment, and the monitoring data comprises identification information of the monitored equipment;
asynchronously adopting a processing engine to match each piece of monitoring data in the first message queue with each stored alarm triggering rule, and storing the successfully matched monitoring data and the matched alarm triggering rule as data to be processed into a second message queue, wherein the data to be processed is associated with a processing time stamp;
and asynchronously adopting an alarm engine to obtain each piece of data to be processed in the second message queue, determining an alarm strategy corresponding to an alarm trigger rule in each piece of data to be processed, and executing a corresponding alarm strategy.
2. The method of claim 1, wherein the capturing the monitoring data collected by each data management node by using the collection engine comprises any one or a combination of the following:
capturing each data management node and collecting monitoring data in each monitoring environment by adopting an acquisition engine through an API (application programming interface), wherein the monitoring data comprises acquired time;
and accessing the databases of the data management nodes by adopting an acquisition engine, and capturing the monitoring data in the corresponding monitoring environment stored in each database, wherein the monitoring data comprises the acquired time.
3. The method of claim 1, wherein asynchronously storing the respective monitoring data associated with the acquisition time stamps to the first message queue comprises:
asynchronously storing each monitoring data associated with the acquisition time stamp to a first message queue and a first storage area;
the step of storing the successfully matched monitoring data and the matched alarm triggering rule thereof as the data to be processed to a second message queue includes:
screening out each monitoring data successfully matched, and determining an alarm triggering rule matched with each monitoring data;
and aiming at each screened monitoring data, determining that the monitoring data is successfully verified when target monitoring data which is the same as the monitoring data exists in the first storage area based on the acquisition timestamp associated with the monitoring data and the identification information of the monitored equipment, generating a piece of data to be processed based on the monitoring data and the alarm triggering rule matched with the monitoring data, and storing the data to be processed in a second message queue.
4. The method of claim 1, wherein the asynchronous matching each piece of monitoring data in the first message queue with each stored alarm triggering rule by the asynchronous processing engine comprises:
and the asynchronous adoption processing engine reads the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamp associated with the monitoring data, and performs matching processing on the read monitoring data and the stored alarm triggering rule, wherein the processing mark is added to the read monitoring data in the first message queue every time one monitoring data is read.
5. The method of any of claims 1-4, wherein said asynchronously employing an alert engine to obtain each piece of pending data in the second message queue comprises:
and asynchronously adopting an alarm engine, and acquiring the to-be-processed data which is not added with the acquisition mark in the second message queue according to the time sequence of the processing time stamps associated with the to-be-processed data, wherein the acquisition mark is added to the to-be-processed data acquired in the second message queue every time one to-be-processed data is acquired.
6. The method of any of claims 1-4, wherein the executing the respective alert policy comprises:
determining monitored equipment corresponding to each piece of data to be processed, and accessing an external system to acquire attribution information and a processing personnel set associated with each piece of monitored equipment, wherein the attribution information is used for assisting a processing personnel in positioning the monitored equipment;
and according to an alarm strategy corresponding to the monitored equipment, respectively sending alarm information at least comprising the attribution information and the data to be processed to the associated equipment of each group of processing personnel in the processing personnel set by taking a set time length as an interval until the pause alarm information fed back by any processing personnel in the processing personnel set based on the received alarm information is received.
7. The method of claim 6, wherein after executing the corresponding alarm policy for the monitored device corresponding to each piece of data to be processed, further comprising:
and storing the alarm triggering rules corresponding to the data to be processed and the execution records of the alarm strategies, and generating a monitoring report according to a set display format based on the stored monitoring data and the execution records of the data to be processed.
8. The method of any of claims 1-4, wherein the gathering engine, the processing engine, and the alert engine are deployed in different containers, respectively.
9. A device for processing monitoring data, comprising:
the system comprises a capturing unit, a first message queue and a second message queue, wherein the capturing unit is used for capturing monitoring data collected by each data management node by adopting an acquisition engine and asynchronously storing each monitoring data associated with an acquisition timestamp into the first message queue, each data management node is deployed in different monitoring environments and is used for collecting operation information of monitored equipment in each monitoring environment, and the monitoring data comprises identification information of the monitored equipment;
the matching unit is used for asynchronously adopting a processing engine to match each piece of monitoring data in the first message queue with each stored alarm triggering rule, and storing the successfully matched monitoring data and the matched alarm triggering rule as data to be processed into a second message queue, wherein the data to be processed is associated with a processing time stamp;
and the execution unit is used for asynchronously adopting an alarm engine to acquire each piece of data to be processed in the second message queue, determining an alarm strategy corresponding to the alarm trigger rule in each piece of data to be processed, and executing the corresponding alarm strategy.
10. The apparatus according to claim 9, wherein when the collecting engine is used to capture the monitoring data collected by each data management node, the capturing unit is used to perform capturing of the monitoring data in any one or a combination of the following manners:
capturing each data management node and collecting monitoring data in each monitoring environment by adopting an acquisition engine through an API (application programming interface), wherein the monitoring data comprises acquired time;
and accessing the databases of the data management nodes by adopting an acquisition engine, and capturing the monitoring data in the corresponding monitoring environment stored in each database, wherein the monitoring data comprises the acquired time.
11. The apparatus of claim 9, wherein the fetch unit, when asynchronously storing respective monitoring data associated with a collection timestamp in a first message queue, is to:
asynchronously storing each monitoring data associated with the acquisition time stamp to a first message queue and a first storage area;
when the successfully matched monitoring data and the matched alarm triggering rule thereof are used as data to be processed and stored in the second message queue, the matching unit is used for:
screening out each monitoring data successfully matched, and determining an alarm triggering rule matched with each monitoring data;
and aiming at each screened monitoring data, determining that the monitoring data is successfully verified when target monitoring data which is the same as the monitoring data exists in the first storage area based on the acquisition timestamp associated with the monitoring data and the identification information of the monitored equipment, generating a piece of data to be processed based on the monitoring data and the alarm triggering rule matched with the monitoring data, and storing the data to be processed in a second message queue.
12. The apparatus of claim 9, wherein when the asynchronous processing engine matches each piece of monitoring data in the first message queue with each saved alarm triggering rule, the matching unit is configured to:
and the asynchronous adoption processing engine reads the monitoring data which is not added with the processing mark in the first message queue according to the time sequence of the acquisition time stamp associated with the monitoring data, and performs matching processing on the read monitoring data and the stored alarm triggering rule, wherein the processing mark is added to the read monitoring data in the first message queue every time one monitoring data is read.
13. The apparatus of any of claims 9-12, wherein, when the asynchronous employs an alert engine to obtain each piece of pending data in the second message queue, the execution unit is to:
and asynchronously adopting an alarm engine, and acquiring the to-be-processed data which is not added with the acquisition mark in the second message queue according to the time sequence of the processing time stamps associated with the to-be-processed data, wherein the acquisition mark is added to the to-be-processed data acquired in the second message queue every time one to-be-processed data is acquired.
14. The apparatus according to any of claims 9-12, wherein the execution unit, when executing the corresponding alert policy, is configured to:
determining monitored equipment corresponding to each piece of data to be processed, and accessing an external system to acquire attribution information and a processing personnel set associated with each piece of monitored equipment, wherein the attribution information is used for assisting a processing personnel in positioning the monitored equipment;
and according to an alarm strategy corresponding to the monitored equipment, respectively sending alarm information at least comprising the attribution information and the data to be processed to the associated equipment of each group of processing personnel in the processing personnel set by taking a set time length as an interval until the pause alarm information fed back by any processing personnel in the processing personnel set based on the received alarm information is received.
15. The apparatus of claim 14, wherein after the corresponding alarm policy is executed for the monitored device corresponding to each piece of data to be processed, the execution unit is further configured to:
and storing the alarm triggering rules corresponding to the data to be processed and the execution records of the alarm strategies, and generating a monitoring report according to a set display format based on the stored monitoring data and the execution records of the data to be processed.
16. The apparatus of any of claims 9-12, wherein the gathering engine, the processing engine, and the alert engine are each deployed in different containers.
17. An electronic device, comprising:
a memory for storing executable instructions;
a processor for reading and executing executable instructions stored in the memory to implement the method of processing monitoring data according to any one of claims 1 to 8.
18. A computer-readable storage medium, wherein instructions in the storage medium, when executed by an electronic device, enable the electronic device to perform the method of processing monitoring data of any one of claims 1 to 8.
CN202110049853.4A 2021-01-14 2021-01-14 Monitoring data processing method and device Pending CN112732531A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110049853.4A CN112732531A (en) 2021-01-14 2021-01-14 Monitoring data processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110049853.4A CN112732531A (en) 2021-01-14 2021-01-14 Monitoring data processing method and device

Publications (1)

Publication Number Publication Date
CN112732531A true CN112732531A (en) 2021-04-30

Family

ID=75593126

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110049853.4A Pending CN112732531A (en) 2021-01-14 2021-01-14 Monitoring data processing method and device

Country Status (1)

Country Link
CN (1) CN112732531A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113783890A (en) * 2021-09-24 2021-12-10 国网山西省电力公司电力科学研究院 Intelligent Internet of things system Internet of things terminal safety monitoring system based on edge calculation

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7448048B1 (en) * 2003-05-27 2008-11-04 International Business Machines Corporation Method for performing real-time analytics using a business rules engine on real-time heterogeneous materialized data views
CN102625349A (en) * 2012-03-09 2012-08-01 浪潮通信信息系统有限公司 Method for processing data under alarm storm
CN107729214A (en) * 2017-10-13 2018-02-23 福建富士通信息软件有限公司 A kind of visual distributed system monitors O＆M method and device in real time
CN110362455A (en) * 2019-07-15 2019-10-22 北京奇艺世纪科技有限公司 A kind of data processing method and data processing equipment
CN111786833A (en) * 2020-07-01 2020-10-16 浪潮云信息技术股份公司 Alarm matching processing implementation method based on cloud service platform

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7448048B1 (en) * 2003-05-27 2008-11-04 International Business Machines Corporation Method for performing real-time analytics using a business rules engine on real-time heterogeneous materialized data views
CN102625349A (en) * 2012-03-09 2012-08-01 浪潮通信信息系统有限公司 Method for processing data under alarm storm
CN107729214A (en) * 2017-10-13 2018-02-23 福建富士通信息软件有限公司 A kind of visual distributed system monitors O＆M method and device in real time
CN110362455A (en) * 2019-07-15 2019-10-22 北京奇艺世纪科技有限公司 A kind of data processing method and data processing equipment
CN111786833A (en) * 2020-07-01 2020-10-16 浪潮云信息技术股份公司 Alarm matching processing implementation method based on cloud service platform

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113783890A (en) * 2021-09-24 2021-12-10 国网山西省电力公司电力科学研究院 Intelligent Internet of things system Internet of things terminal safety monitoring system based on edge calculation

Similar Documents

Publication Publication Date Title
US7467067B2 (en) Self-learning integrity management system and related methods
KR20180108446A (en) System and method for management of ict infra
CN106101130B (en) A kind of network malicious data detection method, apparatus and system
CN115809183A (en) Method for discovering and disposing information-creating terminal fault based on knowledge graph
US20160055044A1 (en) Fault analysis method, fault analysis system, and storage medium
CN114189430A (en) Three-dimensional log full-link monitoring system, method, medium and equipment
CN111339175B (en) Data processing method, device, electronic equipment and readable storage medium
CN109164780A (en) A kind of industrial field device control method based on edge calculations, apparatus and system
CN106940677A (en) One kind application daily record data alarm method and device
CN113949652B (en) User abnormal behavior detection method and device based on artificial intelligence and related equipment
CN106021613A (en) Bridge health monitoring system based on Hadoop
CN108809734A (en) Network alarm root-cause analysis method, system, storage medium and computer equipment
CN111563022A (en) Centralized storage monitoring method and device
CN103095821A (en) Continuous auditing system based on virtual machine migration recognition
CN108055152B (en) Communication network information system abnormity detection method based on distributed service log
CN113342608B (en) Method and device for monitoring tasks of streaming computing engine
CN112732531A (en) Monitoring data processing method and device
CN110363381A (en) A kind of information processing method and device
CN114598719A (en) Smart city Internet of things event management method, device and readable medium
CN111901172B (en) Application service monitoring method and system based on cloud computing environment
CN105897498A (en) Business monitoring method and device
CN113760689A (en) Interface fault alarm method, device, equipment and storage medium
CN107515864B (en) Method and equipment for monitoring workflow
CN108829563B (en) Alarm method and alarm device
CN109614330A (en) Storage system service test method, device, system, storage control and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination