CN110515810A - A kind of data monitoring log collection and retroactive method based on time series database - Google Patents
A kind of data monitoring log collection and retroactive method based on time series database Download PDFInfo
- Publication number
- CN110515810A CN110515810A CN201910828718.2A CN201910828718A CN110515810A CN 110515810 A CN110515810 A CN 110515810A CN 201910828718 A CN201910828718 A CN 201910828718A CN 110515810 A CN110515810 A CN 110515810A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- data
- time series
- series database
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 107
- 238000000034 method Methods 0.000 title claims abstract description 29
- 239000000523 sample Substances 0.000 claims abstract description 16
- 238000012360 testing method Methods 0.000 claims abstract description 11
- 238000012800 visualization Methods 0.000 claims description 6
- 238000013079 data visualisation Methods 0.000 claims description 4
- 238000011161 development Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- WURBVZBTWMNKQT-UHFFFAOYSA-N 1-(4-chlorophenoxy)-3,3-dimethyl-1-(1,2,4-triazol-1-yl)butan-2-one Chemical compound C1=NC=NN1C(C(=O)C(C)(C)C)OC1=CC=C(Cl)C=C1 WURBVZBTWMNKQT-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000001932 seasonal effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Computer Hardware Design (AREA)
- Fuzzy Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention is more particularly directed to a kind of data monitoring log collection and retroactive method based on time series database.The data monitoring log collection and retroactive method based on time series database, monitoring data is acquired using probe in real time, and collected monitoring data is actively pushed into time series database, time series database records monitoring data from time dimension, and early warning and warning information are then fed back to testing staff in time using based on the corresponding prediction policy of monitoring data matching.The data monitoring log collection and retroactive method based on time series database, by being monitored to business whole process, it can find the problem in time, positioning in time, it early processes, greatly alleviates the emergency cases such as operation system snowslide, interruption of government department's application system when accessing peak, improve society, the satisfaction of warden or guard, improve online work efficiency, excellent change service quality.
Description
Technical field
The present invention relates to informatization technical field, in particular to a kind of data monitoring log based on time series database
Collection and retroactive method.
Background technique
In recent years, global generation information technology development is advanced by leaps and bounds, mobile Internet, big data, cloud computing, Internet of Things
The breakthrough of the information technologies such as net, artificial intelligence and fusion development promote the fast development of digital economy.E-government is built
If simplifying government administration office workflow, functional government departments' Efficiency Construction is enhanced, improves the ability of the service common people.But
It is that the continuous development of e-government construction, working business demand is continuously increased on the net for the following government department, business system
System amount of access is also skyrocketed through, especially in some seasonal and policy peak periods, often to can operation system, middleware,
Server etc. generates larger, sudden, mighty torrent formula pressure, is easier that excessive system load, failure, business is caused to be forced to interrupt
Situations such as.
And at this stage, in order to simplify business handling process, people's satisfaction is improved, it is very multiple services to handle all gradually by line
Under be transferred on line, big and small operation system, which is such as emerged rapidly in large numbersBamboo shoots after a spring rain, to be continued to bring out, therefore the pressure of government services systems face
Power is being skyrocketed through.Each operation system how has been managed, has accomplished that problem is prevented in advance, discovery in time, positioned in time, in time
It solves, prevents business handling obstruction from becoming the project that government pays close attention to.
In tradition monitoring, we can only be obtained most running situations by way of reading using log, shape
The system log of shape and color color is dispersed in each corner of server, and more, not intuitive, the difficult positioning of log are to implementation personnel for problem point
Analysis causes very big puzzlement.
Based on the above situation, in order to alleviate operation system snowslide of government department's application system when accessing peak, interrupt
Etc. emergency cases, promote society, the satisfaction of warden or guard, promote online work efficiency, optimize service quality, this
Invention proposes a kind of data monitoring log collection and retroactive method based on time series database.
Summary of the invention
In order to compensate for the shortcomings of the prior art, the present invention provides a kind of data based on time series database being simple and efficient
Monitor log collection and retroactive method.
The present invention is achieved through the following technical solutions:
A kind of data monitoring log collection and retroactive method based on time series database, it is characterised in that: real using probe
When acquire monitoring data, and collected monitoring data is actively pushed into time series database, time series database is from time dimension
Monitoring data is recorded, then matches corresponding prediction policy in time by early warning and warning information using based on monitoring data
Testing staff is fed back to, to realize the operating system from the bottom, up arrives middleware again, then arrive operation system itself,
And operation system relies on the overall monitor of data.
The present invention is based on the data monitoring log collections and retroactive method of time series database, comprising the following steps:
The first step acquires monitoring data
For different monitored object, using different structure and different language probe, the monitoring log of collection monitoring object
Data provide data supporting；
Second step stores monitoring data
Collected monitoring data is centrally stored in time series database, and stamps time mark to every monitoring information
Label, establish monitoring data base library；
Third step, monitoring visualization
Based on monitoring data base library, pattern visualization is carried out for Testing index and is showed；
4th step, early warning mechanism
Metrics-thresholds model is established, warning information is calculated based on monitoring data base library data, if during monitoring
Data target reaches threshold value, and monitoring system responds rapidly, and early warning and warning information are fed back to testing staff.
In the first step, the form of probe includes but is not limited to war packet, jar packet or executable file.
In the first step, the monitoring range of probe covers entire deployed environment, including operating system CPU, memory and deposits
Utilization rate is stored up, middleware memory usage and Thread Count, operation system resources occupation rate, database maximum number of connections and memory make
With rate.
In the second step, monitoring data is recorded from time dimension using time series database prometheus, 24
The hour monitoring data of uninterrupted all standing enables administrative staff to trace back to any time, the monitoring content of any position.
In the third step, using data visualization tool grafana to host monitor information, middleware monitoring information and
The progress of database monitoring information intuitively shows in real time.
In 4th step, indices threshold value is set in metrics-thresholds model, if a certain index reaches during monitoring
To threshold value, system issues warning information to administrative staff；If a certain index is more than threshold value during monitoring, system is i.e. to management
Personnel issue warning information.
The mode for issuing early warning or warning information include but is not limited to show by large-size screen monitors, short message and/or mail lead to
Know testing staff.
The beneficial effects of the present invention are: being somebody's turn to do data monitoring log collection and retroactive method based on time series database, pass through
Business whole process is monitored, can be found the problem in time, positions, early processes in time, greatly alleviates government department's application
The emergency cases such as operation system snowslide, interruption of the system when accessing peak improve society, warden or guard expires
Meaning degree, improves online work efficiency, excellent change service quality.
Specific embodiment
In order to which technical problems, technical solutions and advantages to be solved are more clearly understood, tie below
Embodiment is closed, the present invention will be described in detail.It should be noted that specific embodiment described herein is only to explain
The present invention is not intended to limit the present invention.
The data monitoring log collection and retroactive method based on time series database acquire monitoring number using probe in real time
According to, and collected monitoring data is actively pushed into time series database, time series database remembers monitoring data from time dimension
Record is got off, and early warning and warning information are then fed back to detection people in time using based on the corresponding prediction policy of monitoring data matching
Member up arrives middleware, then arrive operation system itself and operation system to realize the operating system from the bottom again
Rely on the overall monitor of data.
The data monitoring log collection and retroactive method based on time series database, comprising the following steps:
The first step acquires monitoring data
For different monitored object, using different structure and different language probe, the monitoring log of collection monitoring object
Data provide data supporting；
Second step stores monitoring data
Collected monitoring data is centrally stored in time series database, and stamps time mark to every monitoring information
Label, establish monitoring data base library；
Third step, monitoring visualization
Based on monitoring data base library, pattern visualization is carried out for Testing index and is showed；
4th step, early warning mechanism
Metrics-thresholds model is established, warning information is calculated based on monitoring data base library data, if during monitoring
Data target reaches threshold value, and monitoring system responds rapidly, and early warning and warning information are fed back to testing staff.
In the first step, the form of probe includes but is not limited to war packet, jar packet or executable file.
In the first step, the monitoring range of probe covers entire deployed environment, including operating system CPU, memory and deposits
Utilization rate is stored up, middleware memory usage and Thread Count, operation system resources occupation rate, database maximum number of connections and memory make
With rate.
In the second step, monitoring data is recorded from time dimension using time series database prometheus, 24
The hour monitoring data of uninterrupted all standing enables administrative staff to trace back to any time, the monitoring content of any position.
In the third step, using data visualization tool grafana to host monitor information, middleware monitoring information and
The progress of database monitoring information intuitively shows in real time.
In 4th step, indices threshold value is set in metrics-thresholds model, if a certain index reaches during monitoring
To threshold value, system issues warning information to administrative staff；If a certain index is more than threshold value during monitoring, system is i.e. to management
Personnel issue warning information.
The mode for issuing early warning or warning information include but is not limited to show by large-size screen monitors, short message and/or mail lead to
Know testing staff.
Compared with prior art, it is somebody's turn to do data monitoring log collection and retroactive method based on time series database, is had following
Feature:
1, the probe supported for the operation system of format various kinds, middleware, database personalized exploitation, it can be ensured that
Each monitored object is within monitoring range, realizes system monitoring all standing；
2, probe+time series database prometheus mode more lightweight, data storage is more intuitive, and probe is real-time
Active push monitoring data to time series database, time series database records monitoring data from time dimension, this process accounts for
It is few with resource, former operation system is influenced also relatively small；
3, it is different from traditional database storage mode, novelty has used the concept of time series database in monitoring system,
It virtually joined time attribute to each monitoring log, facilitate the retrospect in later period；
4, uninterrupted all standing monitoring in 24 hours, can not only record the monitoring data of comprehensive full period, moreover it is possible to mention
It is shown inquiry for data visualization tool, reason is traced for follow-up business system exception problem and data foundation is provided；
5, Threshold Alerts mechanism is provided, once some index reaches threshold value and can notify relevant tube in time during monitoring
Reason personnel avoid the state of affairs from continuing to spread, and accomplish to reduce the incidence for preventing emergency case in advance, emergency event is sent out in time in thing
It is existing, business reasons are accurately positioned afterwards, ensure the normal operation of operation system；
6, the burst feelings such as operation system snowslide, the interruption of government department's application system when accessing peak are greatly alleviated
Condition improves society, the satisfaction of warden or guard, improves online work efficiency, excellent change service quality.
Embodiment described above, only one kind of the specific embodiment of the invention, those skilled in the art is in this hair
The usual variations and alternatives carried out in bright technical proposal scope should be all included within the scope of the present invention.
Claims (8)
1. a kind of data monitoring log collection and retroactive method based on time series database, it is characterised in that: real-time using probe
Monitoring data is acquired, and collected monitoring data is actively pushed into time series database, time series database will from time dimension
Monitoring data is recorded, then in time that early warning and warning information is anti-using corresponding prediction policy is matched based on monitoring data
Feed testing staff, to realize the operating system from the bottom, up arrives middleware again, then arrive operation system itself, with
And operation system relies on the overall monitor of data.
2. the data monitoring log collection and retroactive method, feature according to claim 1 based on time series database exists
In, comprising the following steps:
The first step acquires monitoring data
For different monitored object, using different structure and different language probe, the monitoring daily record data of collection monitoring object,
Data supporting is provided；
Second step stores monitoring data
Collected monitoring data is centrally stored in time series database, and stamps time tag to every monitoring information, is built
Vertical monitoring data base library；
Third step, monitoring visualization
Based on monitoring data base library, pattern visualization is carried out for Testing index and is showed；
4th step, early warning mechanism
Metrics-thresholds model is established, warning information is calculated based on monitoring data base library data, if data during monitoring
Index reaches threshold value, and monitoring system responds rapidly, and early warning and warning information are fed back to testing staff.
3. the data monitoring log collection and retroactive method, feature according to claim 2 based on time series database exists
In: in the first step, the monitoring range of probe covers entire deployed environment, including operating system CPU, memory and storage use
Rate, middleware memory usage and Thread Count, operation system resources occupation rate, database maximum number of connections and memory usage.
4. the data monitoring log collection and retroactive method, feature according to claim 3 based on time series database exists
In: in the first step, the form of probe includes but is not limited to war packet, jar packet or executable file.
5. the data monitoring log collection and retroactive method according to claim 2 or 4 based on time series database, feature
It is: in the second step, is recorded monitoring data from time dimension using time series database prometheus, 24 hours
The monitoring data of uninterrupted all standing enables administrative staff to trace back to any time, the monitoring content of any position.
6. the data monitoring log collection and retroactive method, feature according to claim 2 based on time series database exists
In: in the third step, using data visualization tool grafana to host monitor information, middleware monitoring information and data
Monitoring information progress in library intuitively shows in real time.
7. the data monitoring log collection and retroactive method, feature according to claim 2 based on time series database exists
In: in the 4th step, indices threshold value is set in metrics-thresholds model, if a certain index reaches threshold during monitoring
Value, system issue warning information to administrative staff；If a certain index is more than threshold value during monitoring, system is i.e. to administrative staff
Issue warning information.
8. the data monitoring log collection and retroactive method, feature according to claim 7 based on time series database exists
In: the mode for issuing early warning or warning information include but is not limited to show by large-size screen monitors, short message and/or mail notification detect
Personnel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910828718.2A CN110515810A (en) | 2019-09-03 | 2019-09-03 | A kind of data monitoring log collection and retroactive method based on time series database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910828718.2A CN110515810A (en) | 2019-09-03 | 2019-09-03 | A kind of data monitoring log collection and retroactive method based on time series database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110515810A true CN110515810A (en) | 2019-11-29 |
Family
ID=68630585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910828718.2A Pending CN110515810A (en) | 2019-09-03 | 2019-09-03 | A kind of data monitoring log collection and retroactive method based on time series database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110515810A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115023A (en) * | 2020-09-02 | 2020-12-22 | 厦门安胜网络科技有限公司 | Acquisition method and system for data link |
| CN112435074A (en) * | 2020-12-07 | 2021-03-02 | 浙江大学 | New retail real-time data tracking and feedback method and system |
| CN112434063A (en) * | 2020-11-03 | 2021-03-02 | 中国南方电网有限责任公司 | Monitoring data processing method based on time sequence database |
| CN112486789A (en) * | 2020-11-30 | 2021-03-12 | 建信金融科技有限责任公司 | Log analysis system, method and device |
| CN112632127A (en) * | 2020-12-29 | 2021-04-09 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN113051125A (en) * | 2019-12-27 | 2021-06-29 | 微民保险代理有限公司 | Monitoring view drawing method and device for self-defined monitoring indexes and computer equipment |
| CN113485887A (en) * | 2021-06-29 | 2021-10-08 | 上海众言网络科技有限公司 | Database monitoring method and device, electronic equipment and storage medium |
| CN113672474A (en) * | 2021-08-27 | 2021-11-19 | 工银科技有限公司 | Monitoring data processing method, device, equipment, medium and program product of cloud database |
| CN114677066A (en) * | 2022-05-30 | 2022-06-28 | 广州信诚信息科技有限公司 | Electricity inspection operation sequence monitoring model based on time sequence data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491354A (en) * | 2013-10-10 | 2014-01-01 | 国家电网公司 | System operation monitoring and controlling visual platform |
| CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
| CN109034521A (en) * | 2018-06-07 | 2018-12-18 | 国电南瑞科技股份有限公司 | A kind of intelligent O&M architecture design method of dispatching of power netwoks control system |
-
2019
- 2019-09-03 CN CN201910828718.2A patent/CN110515810A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491354A (en) * | 2013-10-10 | 2014-01-01 | 国家电网公司 | System operation monitoring and controlling visual platform |
| CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
| CN109034521A (en) * | 2018-06-07 | 2018-12-18 | 国电南瑞科技股份有限公司 | A kind of intelligent O&M architecture design method of dispatching of power netwoks control system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113051125A (en) * | 2019-12-27 | 2021-06-29 | 微民保险代理有限公司 | Monitoring view drawing method and device for self-defined monitoring indexes and computer equipment |
| CN112115023A (en) * | 2020-09-02 | 2020-12-22 | 厦门安胜网络科技有限公司 | Acquisition method and system for data link |
| CN112434063A (en) * | 2020-11-03 | 2021-03-02 | 中国南方电网有限责任公司 | Monitoring data processing method based on time sequence database |
| CN112486789A (en) * | 2020-11-30 | 2021-03-12 | 建信金融科技有限责任公司 | Log analysis system, method and device |
| CN112435074A (en) * | 2020-12-07 | 2021-03-02 | 浙江大学 | New retail real-time data tracking and feedback method and system |
| CN112632127A (en) * | 2020-12-29 | 2021-04-09 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN112632127B (en) * | 2020-12-29 | 2022-07-15 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN113485887A (en) * | 2021-06-29 | 2021-10-08 | 上海众言网络科技有限公司 | Database monitoring method and device, electronic equipment and storage medium |
| CN113672474A (en) * | 2021-08-27 | 2021-11-19 | 工银科技有限公司 | Monitoring data processing method, device, equipment, medium and program product of cloud database |
| CN114677066A (en) * | 2022-05-30 | 2022-06-28 | 广州信诚信息科技有限公司 | Electricity inspection operation sequence monitoring model based on time sequence data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110515810A (en) | A kind of data monitoring log collection and retroactive method based on time series database | |
| US9652723B2 (en) | Electrical transformer failure prediction | |
| US11258814B2 (en) | Methods and systems for using embedding from Natural Language Processing (NLP) for enhanced network analytics | |
| US8516112B2 (en) | Performance monitoring of a computer resource | |
| CN101854277B (en) | Method for monitoring mobile communication operation analysis system | |
| CN104362757B (en) | Transformer substation on-line monitoring system based on IEC61850 | |
| US20150073608A1 (en) | System and method for managing and monitoring renewable energy power generation | |
| CA2747484C (en) | Time tracking technology | |
| CN106372975B (en) | Invoice management system based on SOAP technology and ActiveX control technology | |
| CN105373899A (en) | Server asset management method and apparatus | |
| CN105512790A (en) | Integrated operation and maintenance management system | |
| CN103745322A (en) | Province-city secondary system integrated comprehensive monitoring and process management system in power dispatching and implementation method for system | |
| WO2013097318A1 (en) | Intelligent power utilization system and method for parks | |
| CN101296115B (en) | Telecommunication network multidimensional monitoring method | |
| US9269062B2 (en) | Methods for optimizing energy consumption and devices thereof | |
| CN105656698A (en) | Intelligent monitoring structure and method for network application system | |
| CN103488793A (en) | User behavior monitoring method based on information retrieval | |
| CN108229154A (en) | Sensitive data operation log recording method, device, storage medium and equipment | |
| CN110598051A (en) | Power industry monitoring system, method and device | |
| CN109800133A (en) | A kind of method, one-stop monitoring alarm platform and the system of unified monitoring alarm | |
| CN117453137A (en) | Cloud intelligent operation and maintenance system data management system | |
| CN117632897A (en) | Dynamic capacity expansion and contraction method and device | |
| CN102571413A (en) | Method for resource management under cluster environment | |
| CN113592210A (en) | Internet of things integrated management platform for water supply non-negative-pressure secondary water supply facility | |
| CN116151787A (en) | IT operation and maintenance management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191129 |