CN109344037A - A kind of business monitoring method based on real-time statistics and alarm formula - Google Patents
A kind of business monitoring method based on real-time statistics and alarm formula Download PDFInfo
- Publication number
- CN109344037A CN109344037A CN201811178025.5A CN201811178025A CN109344037A CN 109344037 A CN109344037 A CN 109344037A CN 201811178025 A CN201811178025 A CN 201811178025A CN 109344037 A CN109344037 A CN 109344037A
- Authority
- CN
- China
- Prior art keywords
- variable
- formula
- statistics
- module
- subsystem
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of based on real-time statistics and alerts the business monitoring method of formula, it is related to technical field of data processing, the present invention includes the statistics of variable subsystem run parallel and formula alarm subsystem, pass through caching shared system shared data between statistics of variable subsystem and formula alarm subsystem, business monitoring method includes the following steps: S1: statistics of variable subsystem is according to pre-set time granularity, the value of each variable is periodically counted parallel, and the statistical value of each variable is stored in caching shared system；S2: formula alerts subsystem according to the pre-set period, the statistical value that formula calculates associated variable is concurrently extracted from caching shared system, and concurrently find out the functional value of each associated variable, formula is periodically substituted into be calculated, when calculated result be it is true, then issue alarm, the present invention being capable of flexible configuration variable and custom formula, by it is a kind of it is more flexible accurately in a manner of monitoring system, be adapted to flexible and changeable multidimensional business monitoring demand.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of based on real-time statistics and alarm formula
Business monitoring method.
Background technique
The prior art usually uses database and carries statistical function or SQL (Structured Query
Language, structured query language) to execute statistics, and by the way of hard coded judge whether statistical value reaches some
Threshold value and issue alarm.
And existing statistical has the following problems:
1, query statistic directly is carried out using SQL, data cannot reuse SQL, variable number of the inquiry times directly with statistics
Linear correlation, if wanting statistical history data (as daily counted), inquiry times can be more, when measuring bigger, inquiry times
Slow database can be excessively dragged, service operation is influenced, Statistical Speed also can be slack-off；
2, statistics is all based on single dimension calculating, cannot utilize the value of each dimension, such as there is statistics of male in system
The SQL of the number and SQL of statistics women number, will be intended to statistics of male number ratio, it is necessary to modify code, lack flexible
Property.
Summary of the invention
It is an object of the invention to: query statistic, inquiry time are carried out in order to solve existing data statistics mode using SQL
Number directly with the variable number of statistics linear correlation, when queries is larger, drags slow database, causes Statistical Speed slack-off, count
Real-time reduces, and causes the problem of alerting not in time, and the present invention provides a kind of business prison based on real-time statistics and alarm formula
Prosecutor method.
The present invention specifically uses following technical scheme to achieve the goals above:
A kind of business monitoring method based on real-time statistics and alarm formula, it is characterised in that: the business monitoring method
Based on Concurrent monitor device, Concurrent monitor device communication connection has caching sharing means, and the Concurrent monitor device includes parallel
The statistics of variable subsystem and formula of operation alert subsystem, and caching sharing means include caching shared system, the variable system
It counts between subsystem and formula alarm subsystem through caching shared system shared data, the business monitoring method includes as follows
Step:
S1: statistics of variable subsystem periodically counts the value of each variable according to pre-set time granularity parallel, and will
In the statistical value deposit caching shared system of each variable；
S2: formula alerts subsystem according to the pre-set period, extracts formula parallel from caching shared system and calculates
The statistical value of associated variable, and the functional value of each associated variable is concurrently found out, it periodically substitutes into formula and is calculated, work as meter
Calculating result is very, then to issue alarm.
Further, the caching shared system includes variable list module and cache module, the variable list module
In be stored with preconfigured variable list, cache module be used to store data, the intermediate result of statistic processes of each variable with
And the temporally statistical value of each variable of grain size statistics.
Further, the statistics of variable subsystem includes single thread access and statistics of variable scheduler task module and multi-thread
Journey statistics task Queue module, in the S1, the execution process of statistics of variable subsystem includes the following steps:
S1.1: the period to be less than time granularity, segmentation obtain data from data source and unite to single thread access with variable
Count scheduler task module；
S1.2: single thread access obtains preconfigured change with statistics of variable scheduler task module from variable list module
Measure list；
S1.3: the data that single thread access will acquire with statistics of variable scheduler task module are submitted to multi-thread with variable list
Journey statistics task Queue module；
S1.4: multithreading statistics task Queue module takes out the intermediate result of each variable last time caching from cache module；
S1.5: the intermediate result that multithreading statistics task Queue module was cached according to the data of acquisition and each variable last time
To each variable, temporally granularity carries out data statistics parallel, obtains the statistical value of each variable；
S1.6: by the statistical value deposit cache module of each variable, primary statistics is completed, S1.1 is returned and is united next time
Meter.
Further, the formula alarm subsystem includes single thread formula scheduler task subsystem module, multithreading public affairs
Formula alarming assignment executes Queue module, alerts listing formulas and warning system, and in the S2, formula alerts the execution stream of subsystem
Journey includes the following steps:
S2.1: single thread formula scheduler task subsystem module is according to the pre-set period, from alarm listing formulas
Obtain listing formulas；
S2.2: single thread formula scheduler task subsystem module submits listing formulas and formula alarming assignment to multithreading public affairs
Formula alarming assignment executes Queue module；
S2.3: multithreading formula alarming assignment executes Queue module from the public affairs obtained in listing formulas in variable list module
Variable information involved in formula；
S2.4: multithreading formula alarming assignment executes the time that Queue module is recorded from cache module according to variable information
Range extracts the interval statistics value of each variable parallel, and concurrently finds out the functional value of each variable；
S2.5: the functional value of each variable is substituted into formula and is calculated；
S2.6: it if formula calculated result is that very, warning system issues alarm, returns to S2.1 and is judged next time.
Further, since real-time statistics are based on time granularity, the data that are stored in the cache module can be with
Temporally remove stale data and intermediate result.
Beneficial effects of the present invention are as follows:
1, periodical pulling data is counted small lot of the present invention to local memory at times, statistics in need change
Amount is multiplexed same part data, reduces data base querying number, and periodical pulling data can not only mitigate database pressure, also
It can reduce delay, reduce network bandwidth, save memory source, reduce the influence to operation system, improve the available of system
Property, meanwhile, multithreading statistics task Queue module parallel counts each variable, improves system execution efficiency, variable system
Meter subsystem and formula alarm subsystem decoupled are run parallel, are improved the real-time of statistics and alarm, are avoided two subsystems
System mutually obstruction.
2, alarm formula of the invention can be customized, and various variables are freely combined and are calculated, hard compared to traditional
Coding mode is more flexible changeable, and variable is multiplexed while also improving system performance, and various variables is enabled to pass through formula
Mode combines, by it is more flexible accurately in a manner of be monitored, be suitable for flexible and changeable multidimensional business monitoring demand, formula
Middle multiplexing variable, has saved computing resource, has improved system performance.
Detailed description of the invention
Fig. 1 is realization logical flow chart of the invention.
Fig. 2 is integral frame structure figure of the invention.
Specific embodiment
In order to which those skilled in the art better understand the present invention, with reference to the accompanying drawing with following embodiment to the present invention
It is described in further detail.
Embodiment 1
As depicted in figs. 1 and 2, the present embodiment provides a kind of based on real-time statistics and alerts the business monitoring method of formula,
The business monitoring method is based on Concurrent monitor device, and Concurrent monitor device communication connection has caching sharing means, described parallel
Monitoring device includes the statistics of variable subsystem run parallel and formula alarm subsystem, and caching sharing means include that caching is shared
System passes through caching shared system shared data, the variable between the statistics of variable subsystem and formula alarm subsystem
Statistics sub system includes single thread access and statistics of variable scheduler task module and multithreading statistics task Queue module, and formula is accused
Alert subsystem includes single thread formula scheduler task subsystem module, multithreading formula alarming assignment execution Queue module, alarm
Listing formulas and warning system, the caching shared system include variable list module and cache module, the variable list mould
Preconfigured variable list is stored in block, cache module is used to store data, the intermediate result of statistic processes of each variable
And the temporally statistical value of each variable of grain size statistics, since real-time statistics are the cache modules based on time granularity
Middle stored data can temporally remove stale data and intermediate result；
The present embodiment is to carry out real-time statistics and alarm with Logout Events to certain app registration, first to variable list module
It is configured with alarm listing formulas:
Variable registEventSum configuration:
| ParamName (parameter name) | registEventCount |
| AppId (application numbers) | app1 |
| EventType (event type) | login |
| TimaSpan (time range) | 3 days |
Variable unregistEventSum configuration:
Formula 1 is alerted to configure:
Formula 2 is alerted to configure:
The business monitoring method includes the following steps:
S1: statistics of variable subsystem periodically counts the value of each variable according to pre-set time granularity parallel, and will
In the statistical value deposit caching shared system of each variable, the time granularity in the present embodiment is 1 day:
S1.1: single thread access is less than the period of time granularity, the present embodiment with statistics of variable scheduler task module
In be ten minutes, segmentation from data source obtain data to single thread access with statistics of variable scheduler task module；
S1.2: single thread access obtains above-mentioned be pre-configured with statistics of variable scheduler task module from variable list module
Variable list；
S1.3: the data that single thread access will acquire with statistics of variable scheduler task module are submitted to multi-thread with variable list
Journey statistics task Queue module；
S1.4: multithreading statistics task Queue module takes out the intermediate result of each variable last time caching from cache module；
S1.5: the intermediate result that multithreading statistics task Queue module was cached according to the data of acquisition and each variable last time
To each variable, temporally granularity carries out data statistics parallel, obtains the statistical value of each variable；
S1.6: by the statistical value deposit cache module of each variable, primary statistics is completed, S1.1 is returned and is united next time
Meter；
S2: formula alerts subsystem according to the pre-set period, extracts formula parallel from caching shared system and calculates
Then the statistical value of associated variable concurrently finds out the functional value of each associated variable, functional value is substituted into formula, periodically right
Formula is calculated, when calculated result be it is true, then issue alarm, specifically:
S2.1: single thread formula scheduler task subsystem module is according to the pre-set period, from alarm listing formulas
Obtain listing formulas；
S2.2: single thread formula scheduler task subsystem module submits listing formulas and formula alarming assignment to multithreading public affairs
Formula alarming assignment executes Queue module；
S2.3: multithreading formula alarming assignment executes Queue module from the public affairs obtained in listing formulas in variable list module
Variable information involved in formula, the variable information in the present embodiment include parameter name in above-mentioned variable configuration, application numbers, thing
Part type and time range；
S2.4: multithreading formula alarming assignment executes Queue module according to the time range in variable information, from caching mould
The interval statistics value of each variable is extracted in block parallel, and concurrently finds out the functional value of the associated each variable of formula；
S2.5: the functional value of each variable is substituted into formula and is calculated；
S2.6: it if formula calculated result is that very, warning system issues alarm, returns to S2.1 and is judged next time.
The present embodiment configures various variables using configuration mode, then the value of these variables of real-time statistics, finally using certainly
The formula of definition calculates these statistical variables, result be it is true, just issue alarm, remind there are the problem of or risk, be suitable for real
When, various dimensions comprehensive statistics business monitoring alarm scene in, can the various statistical variables of flexible configuration and alarm formula, with more complete
Face, more accurately mode are to system implementing monitoring, and the statistics of custom variable and the execution of alarm formula are mutually indepedent, to allow
Certain alarm formula will not because the statistics of uncorrelated variables do not complete and occlusion alarm, improve the stability and alarm of system
Real-time, during statistics of variable, using single task be segmented fetch, with multitask composite statistics strategy reduce Netowrk tape
Wide and memory consumption improves statistic property.
The above, only presently preferred embodiments of the present invention, are not intended to limit the invention, patent protection model of the invention
It encloses and is subject to claims, it is all to change with equivalent structure made by specification and accompanying drawing content of the invention, similarly
It should be included within the scope of the present invention.
Claims (5)
1. a kind of business monitoring method based on real-time statistics and alarm formula, it is characterised in that: the business monitoring method base
In Concurrent monitor device, Concurrent monitor device communication connection has caching sharing means, and the Concurrent monitor device includes parallel fortune
Capable statistics of variable subsystem and formula alert subsystem, and caching sharing means include caching shared system, the statistics of variable
By caching shared system shared data between subsystem and formula alarm subsystem, the business monitoring method includes following step
It is rapid:
S1: statistics of variable subsystem periodically counts the value of each variable according to pre-set time granularity parallel, and by each change
In the statistical value deposit caching shared system of amount；
S2: formula alerts subsystem according to the pre-set period, extracts formula calculating parallel from caching shared system and is closed
The statistical value of the variable of connection, and the functional value of each associated variable is concurrently found out, it periodically substitutes into formula and is calculated, tied when calculating
Fruit is very, then to issue alarm.
2. a kind of business monitoring method based on real-time statistics and alarm formula according to claim 1, it is characterised in that:
The caching shared system includes variable list module and cache module, is stored in the variable list module preconfigured
Variable list, cache module are used to store the data of each variable, the intermediate result of statistic processes and temporally grain size statistics
The statistical value of each variable.
3. a kind of business monitoring method based on real-time statistics and alarm formula according to claim 2, it is characterised in that:
The statistics of variable subsystem includes single thread access and statistics of variable scheduler task module and multithreading statistics task queue mould
Block, in the S1, the execution process of statistics of variable subsystem includes the following steps:
S1.1: the period to be less than time granularity, segmentation obtain data to single thread access and statistics of variable tune from data source
Spend task module；
S1.2: single thread access obtains preconfigured variable column with statistics of variable scheduler task module from variable list module
Table；
S1.3: the data and variable list that single thread access will acquire with statistics of variable scheduler task module are submitted to multithreading system
Count task queue module；
S1.4: multithreading statistics task Queue module takes out the intermediate result of each variable last time caching from cache module；
S1.5: multithreading statistics task Queue module is parallel according to the intermediate result that the data of acquisition and each variable last time cache
To each variable, temporally granularity carries out data statistics, obtains the statistical value of each variable；
S1.6: by the statistical value deposit cache module of each variable, primary statistics is completed, S1.1 is returned and is counted next time.
4. a kind of business monitoring method based on real-time statistics and alarm formula according to claim 2 or 3, feature exist
In: the formula alarm subsystem includes single thread formula scheduler task subsystem module, the execution of multithreading formula alarming assignment
Queue module, alarm listing formulas and warning system, in the S2, the execution process that formula alerts subsystem includes following step
It is rapid:
S2.1: single thread formula scheduler task subsystem module is obtained from alarm listing formulas according to the pre-set period
Listing formulas；
S2.2: single thread formula scheduler task subsystem module submits listing formulas and formula alarming assignment to accuse to multithreading formula
Alert task execution Queue module；
S2.3: multithreading formula alarming assignment executes Queue module from the formula institute obtained in listing formulas in variable list module
The variable information being related to；
S2.4: multithreading formula alarming assignment executes the time range that Queue module is recorded from cache module according to variable information
The interval statistics value of each variable is extracted parallel, and concurrently finds out the functional value of each variable；
S2.5: the functional value of each variable is substituted into formula and is calculated；
S2.6: it if formula calculated result is that very, warning system issues alarm, returns to S2.1 and is judged next time.
5. a kind of business monitoring method based on real-time statistics and alarm formula according to claim 2, it is characterised in that:
The data stored in the cache module temporally remove stale data and intermediate result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811178025.5A CN109344037B (en) | 2018-10-10 | 2018-10-10 | Service monitoring method based on real-time statistics and alarm formula |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811178025.5A CN109344037B (en) | 2018-10-10 | 2018-10-10 | Service monitoring method based on real-time statistics and alarm formula |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109344037A true CN109344037A (en) | 2019-02-15 |
| CN109344037B CN109344037B (en) | 2022-02-11 |
Family
ID=65309315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811178025.5A Active CN109344037B (en) | 2018-10-10 | 2018-10-10 | Service monitoring method based on real-time statistics and alarm formula |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344037B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110147269A (en) * | 2019-05-09 | 2019-08-20 | 腾讯科技（上海）有限公司 | A kind of event-handling method, device, equipment and storage medium |
| CN113538026A (en) * | 2020-04-15 | 2021-10-22 | 北京京东振世信息技术有限公司 | Traffic calculation method and device |
| CN113791589A (en) * | 2021-08-12 | 2021-12-14 | 北京寄云鼎城科技有限公司 | Statistical calculation method and device for statistical process control, computer equipment and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605664A (en) * | 2013-10-22 | 2014-02-26 | 芜湖大学科技园发展有限公司 | Massive dynamic data fast query method meeting different time granularity requirements |
| CN104933190A (en) * | 2015-07-10 | 2015-09-23 | 上海新炬网络信息技术有限公司 | SQL statement execution frequency dynamic regulation method |
| CN106776811A (en) * | 2016-11-23 | 2017-05-31 | 李天� | data index method and device |
| CN106909117A (en) * | 2017-03-28 | 2017-06-30 | 重庆市通信建设有限公司 | data real-time monitoring system and method |
| CN107229623A (en) * | 2016-03-23 | 2017-10-03 | 泰康保险集团股份有限公司 | Data query processing method and processing device |
-
2018
- 2018-10-10 CN CN201811178025.5A patent/CN109344037B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605664A (en) * | 2013-10-22 | 2014-02-26 | 芜湖大学科技园发展有限公司 | Massive dynamic data fast query method meeting different time granularity requirements |
| CN104933190A (en) * | 2015-07-10 | 2015-09-23 | 上海新炬网络信息技术有限公司 | SQL statement execution frequency dynamic regulation method |
| CN107229623A (en) * | 2016-03-23 | 2017-10-03 | 泰康保险集团股份有限公司 | Data query processing method and processing device |
| CN106776811A (en) * | 2016-11-23 | 2017-05-31 | 李天� | data index method and device |
| CN106909117A (en) * | 2017-03-28 | 2017-06-30 | 重庆市通信建设有限公司 | data real-time monitoring system and method |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110147269A (en) * | 2019-05-09 | 2019-08-20 | 腾讯科技（上海）有限公司 | A kind of event-handling method, device, equipment and storage medium |
| CN113538026A (en) * | 2020-04-15 | 2021-10-22 | 北京京东振世信息技术有限公司 | Traffic calculation method and device |
| CN113538026B (en) * | 2020-04-15 | 2023-11-03 | 北京京东振世信息技术有限公司 | Service amount calculation method and device |
| CN113791589A (en) * | 2021-08-12 | 2021-12-14 | 北京寄云鼎城科技有限公司 | Statistical calculation method and device for statistical process control, computer equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109344037B (en) | 2022-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344037A (en) | A kind of business monitoring method based on real-time statistics and alarm formula | |
| CN105204971B (en) | A kind of dynamic supervision interval method of adjustment based on Naive Bayes Classification technology | |
| CN102340415B (en) | Server cluster system and monitoring method thereof | |
| CN110245158A (en) | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology | |
| CN108415789A (en) | Node failure forecasting system and method towards extensive mixing heterogeneous storage system | |
| CN104917627B (en) | A kind of log cluster for large server cluster scans and analysis method | |
| CN110532152A (en) | A kind of monitoring alarm processing method and system based on Kapacitor computing engines | |
| CN103678402B (en) | A kind of method of data real-time statistics under mass data | |
| CN106940677A (en) | One kind application daily record data alarm method and device | |
| CN109165133B (en) | Data monitoring method, device, equipment and storage medium | |
| CN113297183B (en) | Alarm analysis method and device for time window | |
| EP3324602A1 (en) | Sensor data generation and response handling stack | |
| CN113986595A (en) | Abnormity positioning method and device | |
| CN112347163A (en) | High-dispersion SQL dynamic baseline warning method and system | |
| WO2015033126A1 (en) | Analysis of parallel processing systems | |
| CN104518913A (en) | Cloud service abnormality detection method based on artificial immunity | |
| CN111241074B (en) | Steel enterprise data center application system based on time sequence data and relation data | |
| Zhao et al. | Analysis and prediction of big stream data in real-time water quality monitoring system | |
| CN114070718B (en) | Alarm method, alarm device and storage medium | |
| CN115225986A (en) | Adaptive OSU bandwidth adjustment method and device | |
| CN108737164A (en) | A kind of telecommunication network Real-time Alarm filter method and device | |
| CN110659681A (en) | Time sequence data prediction system and method based on pattern recognition | |
| CN113360564A (en) | ETL-based data stream processing method, system, device and readable storage medium | |
| CN102024053B (en) | Approximate circle matching method for isomorphic and symmetric publish-subscribe system | |
| Da Silva et al. | Clusmaster: A clustering approach for sampling data streams in sensor networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |