CN110855527A - Early warning monitoring system - Google Patents
Early warning monitoring system Download PDFInfo
- Publication number
- CN110855527A CN110855527A CN201911145903.8A CN201911145903A CN110855527A CN 110855527 A CN110855527 A CN 110855527A CN 201911145903 A CN201911145903 A CN 201911145903A CN 110855527 A CN110855527 A CN 110855527A
- Authority
- CN
- China
- Prior art keywords
- event
- module
- warning
- data
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 49
- 238000004364 calculation method Methods 0.000 claims abstract description 26
- 238000004891 communication Methods 0.000 claims abstract description 13
- 230000002688 persistence Effects 0.000 claims description 11
- 230000007246 mechanism Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 4
- 230000008878 coupling Effects 0.000 abstract description 4
- 238000010168 coupling process Methods 0.000 abstract description 4
- 238000005859 coupling reaction Methods 0.000 abstract description 4
- 238000004801 process automation Methods 0.000 abstract description 3
- 238000007726 management method Methods 0.000 description 7
- 238000000034 method Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000002955 isolation Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000006378 damage Effects 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0681—Configuration of triggering conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/535—Tracking the activity of the user
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides an early warning monitoring system which comprises an event module, a calculation module, a rule module and a bus module. The system is based on event flow, the communication among all modules is based on an event bus, the coupling requirement is low, and the full-process automation is realized.
Description
Technical Field
The invention belongs to the technical field of monitoring, and particularly relates to an early warning monitoring system.
Background
In the context of big data, some larger companies monitor business data, and further develop a series of monitoring platforms. The existing monitoring platform has the following disadvantages:
1. data communication between components is protocol-based, and the coupling between components is high.
2. Most of the methods only support data offline calculation and have poor expansibility.
3. The existing scheme only supports simple early warning.
4. The operation of service personnel is complicated, the script needs to be operated manually, and the operability is poor.
5. Different business early warnings have different logics, and the later maintenance cost is extremely high.
If a set of efficient, perfect, highly available and extensible monitoring and early warning solution can be designed to solve the problems, the method has strong practical value and wide market prospect.
Disclosure of Invention
In order to solve the problems, the invention discloses an early warning monitoring system which is provided with an event module, a calculation module, a rule module and a bus module;
the bus module is configured to: as a communication channel, receiving service data issued by an external monitored service end and data issued by other modules, and as a communication channel, realizing communication among the other modules;
the event module is configured to: an event configuration interface is arranged for a user to define the early warning event and to issue the defined early warning event and the message created by the event to the bus module;
the rules module is configured to: the system comprises a plurality of event buses, a rule configuration interface, a monitoring bus and a warning bus, wherein a plurality of predefined rules and/or rule sets are stored, the rule configuration interface is arranged, the use of the rules and/or rule sets can be realized, predefined warning data events can be selected, the monitoring rules or the monitoring rule sets can be customized, and customized rule events are formed and are issued to an event bus; the rule module also monitors the bus module, carries out rule judgment on the obtained early warning data event and carries out corresponding early warning operation according to the judgment result;
the computing module is configured to: the monitoring bus module acquires corresponding data for real-time or off-line calculation after monitoring the event creating message and issues a calculation result to the bus module; an offline task and a real-time task can be defined, and task global management is carried out;
the data persistence module is configured to: the bus module is used for carrying out persistence processing on the data which is transferred in the bus module.
Further, the items of the custom rule event include a monitoring name, a monitoring object, a monitoring range, a monitoring index and an alarm threshold.
Further, the bus module employs a message queue.
Furthermore, the content of the self-defined early warning event comprises an event definition, an event and data dependency relationship definition and an event completion rule definition.
Furthermore, the types of the self-defined early warning events comprise control events and data events.
Further, the early warning operation is classified into grading early warning, and/or duty early warning, and/or time-interval early warning.
Further, the data persistence module adopts one or more of MySQ, HBase, Hive, Spark, ES and Kylin.
Furthermore, the event module adopts a tenant mechanism to configure operable data sources and events for each service end.
Further, the creation of the offline task is realized by manually creating a timed task.
Further, the offline calculation is to perform batch calculation on the data in a specified time period before the predetermined time point at the predetermined time point.
The invention has the beneficial effects that:
1. the invention adopts a communication mechanism based on event stream, the communication among all the components is based on an event bus, and the coupling is low.
2. And off-line calculation and real-time calculation of data are supported, and further delay monitoring early warning and real-time early warning are supported.
3. The invention realizes full process automation, including configuration of early warning information, data extraction, data layering, data storage and data early warning.
4. The invention has high expansibility, can effectively integrate all service early warnings and has low later maintenance cost.
Drawings
FIG. 1 is a block diagram of the system of the present invention.
Detailed Description
The design concept of the invention is as follows: a system is provided that is event flow based, that communication between all components is event bus based, that coupling requirements are low, and that full process automation is achieved.
The following description will be made in detail with reference to fig. 1.
The invention comprises an event module, a calculation module, a rule module and a bus module. The following are introduced separately:
firstly, the method comprises the following steps: event module
The event module is configured to: and an event configuration interface is arranged for a service party to configure a self-defined early warning event, and the self-defined early warning event is issued to the bus module. The self-defined early warning events are divided into control type events (namely protocols) and data type events (namely examples).
Specifically, the event module also provides 3 main functions of permission isolation, event self-service operation and metadata management.
a. And (4) permission isolation:
1. a tenant mechanism is employed. The tenant mechanism virtually partitions the data and configuration of the event module, so that each tenant (namely a business side) can use a single instance, and each tenant can individually configure the leased instance according to the requirement of the tenant.
2. And a permission isolation measure is provided, and data sources and events which can be operated by different tenants are configured in advance by a platform administrator.
b. Self-service operation of self-defined event setting
And when the service side considers that the conventional early warning event can not meet the requirement and needs to be defined by users, event definition, event and data dependency relationship definition and event completion rule definition are carried out in an event configuration interface set by an event module by self.
When self-defined event self-service operation is carried out, the event module issues an 'event creation' message on the bus module, and the computing module can monitor the message. The destruction of events is performed without the need for pre-warning.
c. Metadata management
The event module provides metadata management capability for customizing the whole life cycle of the early warning event, metadata persistence capability (storing configuration information metadata into a relational database through a persistence module), metadata version snapshot capability (backing the event to a specific version), maintenance of mapping relation between the event and specific implementation thereof, support of control type events (namely protocols) and data type events (namely instances), and support of corresponding event and data dependency relation definition of a business party.
Second, calculating module
The computing module includes a task manager configured to: and monitoring an event data bus, acquiring a generation script corresponding to the custom early warning data after monitoring the event creation message, and then starting a task to calculate related data.
The calculation is divided into off-line calculation and real-time calculation. The off-line calculation adopts the technical means that the batch calculation is carried out on the data in the appointed time period before the preset time point at the preset time point through a HIVE module, a YARN module and an HDFS module of the HADOOP and through a task scheduling system. In the present embodiment, the off-line calculation is a calculation of data before the morning zero point of the day. The function of off-line calculation supports delay early warning. Offline task generation supports manual creation of timed tasks, addition to a data platform scheduling system for data dependency management and unified scheduling execution. The data platform scheduling system may employ existing tools, such as HERA,
and the real-time task generation support is to create a new real-time task according to the generation specification based on the event content. The calculated data is sent back to the event bus, and is subjected to persistence processing, monitoring is performed by the rule module, and rule hit refreshing is performed. The calculation processing of the early warning data event comprises the steps of generating an offline task and a real-time task by utilizing data.
The task manager monitors the control protocol events meeting the conditions through a daemon process, and performs task global management according to the control protocol events, such as 'creation event', 'destruction event' and the like. The event creating is that the computing module monitors the rule of the event bus to obtain the early warning event so as to create an offline or real-time task (namely, the task created by the YARN cluster of the HADOOP), and the event destroying is that the task in the HADOOP cluster is killed
If the event is accompanied by the task generated by the data (namely the data event), an 'offline task' or a 'real-time task' is newly established according to the configuration information in the early warning data event.
Standardizing data sources, output forms and the like of the offline tasks and the real-time tasks according to the metadata information;
rule module
The rules module is configured to: the system is provided with a rule configuration interface, a user can select a predefined early warning data event and can customize a monitoring rule or a monitoring rule set to form a customized rule event. The custom rule event is published to an event bus. The rule module also monitors the bus module, carries out rule judgment on the obtained early warning data event and carries out corresponding early warning operation according to the judgment result. The early warning operation comprises grading early warning, on-duty early warning and time interval early warning. The meaning of the grading early warning is as follows: setting different early warning levels according to different employee levels and corresponding alarm severity levels, such as CTO level early warning, master supervision level early warning, general level expert early warning and the like;
the meaning of the on-duty early warning is as follows: the company can set the on-duty personnel every day, and the early warning information every day can be sent to the on-duty personnel;
the meaning of the time interval early warning is as follows: forewarning according to the grade time interval of forewarning, for example: the ordinary early warning is only the notification from eight early points to eight late points, and the severe early warning is performed in the whole time period.
The self-defined monitoring rule comprises a rule definition, a rule threshold value setting and a rule disposal action setting. The rule handling action settings include, but are not limited to, mail alerts, nail alerts, phone alerts, and generation of new events.
The custom rule set is a logical operation of the monitoring rules (and, or, not, etc.).
The rule module also supports the enabling, suspending of rules, rule sets.
The following exemplary rules may specifically include the following:
1. monitoring a name;
2. monitoring the object: such as city, media, channel, account, plan, unit, keyword, creative, search term;
3. monitoring range: account, channel, keyword type (including consumption-thread amount, daily average show amount-daily average click rate, thread cost-thread amount);
4. monitoring indexes: such as: exposure amount, click amount, consumption, exposure amount, consumption amount, clue cost, user rate, jump rate, abnormal keyword state, non-approved passing, too low search amount, and newly added or removed keywords.
5. Alarm threshold value: the method comprises upper limit reminding and lower limit reminding. If, the upper bound is reminded to: present day display amount > average over the past N days; the lower limit reminder is: present day display < average over the past N days. Can also set alarm information receiver
The rules module also supports the following functions:
1. metadata management capabilities that support the entire lifecycle of a rule, rule set.
2. The self-defined adding capability to the rule base, the knowledge base and the fact base is provided.
3. Providing metadata persistence capabilities.
4. Metadata version snapshot capability is provided that can roll back rules to a particular version.
5. If a rule handles an action as "generating a new event," then the mapping between the new event and its specific implementation needs to be maintained.
Bus module
The bus module is configured to receive service data issued by an external monitored service end and events, data and messages issued by the other modules, and the received service data, the events, the data and the messages are taken as circulation channels of the bus module to decouple the other modules and realize communication among the modules.
The bus module may be any message communication mechanism, if message queues are employed, the RabbitMQ, Kafka, etc. tools may be used.
Data persistence module
The module is mainly used for persisting events, rules, batch calculation data results, real-time calculation results and the like. The adopted infrastructure can be one or a combination of several of MySQL, HBase, Hive, Spark, ES, Kylin and other tools.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited to the above embodiments, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present invention, and all those persons skilled in the art should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. An early warning monitoring system is characterized by comprising an event module, a calculation module, a rule module and a bus module;
the bus module is configured to: as a communication channel, receiving service data issued by an external monitored service end and data issued by other modules, and as a communication channel, realizing communication among the other modules;
the event module is configured to: an event configuration interface is arranged for a user to define the early warning event and to issue the defined early warning event and the message created by the event to the bus module;
the rules module is configured to: the system comprises a plurality of event buses, a rule configuration interface, a monitoring bus and a warning bus, wherein a plurality of predefined rules and/or rule sets are stored, the rule configuration interface is arranged, the use of the rules and/or rule sets can be realized, predefined warning data events can be selected, the monitoring rules or the monitoring rule sets can be customized, and customized rule events are formed and are issued to an event bus; the rule module also monitors the bus module, carries out rule judgment on the obtained early warning data event and carries out corresponding early warning operation according to the judgment result;
the computing module is configured to: the monitoring bus module acquires corresponding data for real-time or off-line calculation after monitoring the event creating message and issues a calculation result to the bus module; an offline task and a real-time task can be defined, and task global management is carried out;
the data persistence module is configured to: the bus module is used for carrying out persistence processing on the data which is transferred in the bus module.
2. The warning monitoring system of claim 1, wherein the items of the custom rule event include a monitoring name, a monitoring object, a monitoring range, a monitoring index, and an alarm threshold.
3. The warning monitoring system of claim 1, wherein the bus module employs a message queue.
4. The warning monitoring system of claim 1, 2 or 3 wherein the content of the customized warning event includes event definition, event and data dependency definition, and event completion rule definition.
5. The warning monitoring system of claim 1, wherein the category of the customized warning event includes a control type event and a data type event.
6. The warning monitoring system as claimed in claim 1 or 5, wherein the warning operation is classified into a grading warning, and/or an on-duty warning, and/or a period warning.
7. The warning monitoring system of claim 1, wherein the data persistence module employs one or more of MySQ, HBase, Hive, Spark, ES, and Kylin.
8. The warning monitoring system of claim 1, wherein the event module employs a tenant mechanism to configure operational data sources and events for each service end.
9. The warning monitoring system of claim 1 wherein the creation of the offline task is accomplished by manually creating a timed task.
10. The warning monitoring system of claim 1, wherein the offline calculation is a batch calculation of data at a predetermined point in time and within a specified period of time prior to the predetermined point in time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911145903.8A CN110855527A (en) | 2019-11-21 | 2019-11-21 | Early warning monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911145903.8A CN110855527A (en) | 2019-11-21 | 2019-11-21 | Early warning monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110855527A true CN110855527A (en) | 2020-02-28 |
Family
ID=69603099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911145903.8A Pending CN110855527A (en) | 2019-11-21 | 2019-11-21 | Early warning monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110855527A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8856807B1 (en) * | 2011-01-04 | 2014-10-07 | The Pnc Financial Services Group, Inc. | Alert event platform |
| CN106250290A (en) * | 2016-08-03 | 2016-12-21 | 广州唯品会信息科技有限公司 | The analysis method and device of abnormal information |
| CN109086894A (en) * | 2018-07-06 | 2018-12-25 | 西安热工研究院有限公司 | A kind of warning message centring system of facing area genco |
| CN110175451A (en) * | 2019-04-23 | 2019-08-27 | 国家电网公司华东分部 | A kind of method for safety monitoring and system based on electric power cloud |
-
2019
- 2019-11-21 CN CN201911145903.8A patent/CN110855527A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8856807B1 (en) * | 2011-01-04 | 2014-10-07 | The Pnc Financial Services Group, Inc. | Alert event platform |
| CN106250290A (en) * | 2016-08-03 | 2016-12-21 | 广州唯品会信息科技有限公司 | The analysis method and device of abnormal information |
| CN109086894A (en) * | 2018-07-06 | 2018-12-25 | 西安热工研究院有限公司 | A kind of warning message centring system of facing area genco |
| CN110175451A (en) * | 2019-04-23 | 2019-08-27 | 国家电网公司华东分部 | A kind of method for safety monitoring and system based on electric power cloud |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110868338B (en) | Early warning monitoring method | |
| CN111459763B (en) | Cross-kubernetes cluster monitoring system and method | |
| WO2023142054A1 (en) | Container microservice-oriented performance monitoring and alarm method and alarm system | |
| US20100070981A1 (en) | System and Method for Performing Complex Event Processing | |
| CN104407964A (en) | Centralized monitoring system and method based on data center | |
| CN110210705A (en) | A kind of data analysing method and system | |
| CN107995006A (en) | Real-time Billing System based on message trigger under a kind of cloud environment | |
| CN109885453A (en) | Big data platform monitoring system based on flow data processing | |
| CN110221947A (en) | Warning information method for inspecting, system, computer installation and readable storage medium storing program for executing | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| CN109840141A (en) | Thread control method, device, electronic equipment and storage medium based on cloud monitoring | |
| CN113377626A (en) | Visual unified alarm method, device, equipment and medium based on service tree | |
| CN109067000A (en) | One kind matching electricity consumption intelligence load monitoring system | |
| CN109446204A (en) | A kind of date storage method of instant messaging, device, electronic equipment and medium | |
| CN103986607A (en) | Voice-sound-light alarm monitoring system for intelligent data center | |
| CN109858807A (en) | A kind of method and system of enterprise operation monitoring | |
| CN110855527A (en) | Early warning monitoring system | |
| CN111625410B (en) | Information processing method, apparatus, and computer storage medium | |
| CN114756301B (en) | Log processing method, device and system | |
| WO2020010531A1 (en) | Fault detection method and device | |
| CN108122093B (en) | Multi-platform multi-channel task management system based on deep learning | |
| CN109120439B (en) | Distributed cluster alarm output method, device, equipment and readable storage medium | |
| CN113722141A (en) | Method and device for determining delay reason of data task, electronic equipment and medium | |
| CN109766238B (en) | Session number-based operation and maintenance platform performance monitoring method and device and related equipment | |
| CN113781133A (en) | Order data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210825 Address after: 100000 1025, floor 1, No. 1, Kaifa Road, Shangdi Information Industry base, Haidian District, Beijing Applicant after: Beijing tulayan Technology Co.,Ltd. Address before: 100089 08a01, block D, 8 / F, No. a 28, information road, Haidian District, Beijing Applicant before: Beijing Gaoyin Technology Co.,Ltd. |
|
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200228 |