CN113138891A - Service monitoring system based on log - Google Patents
Service monitoring system based on log Download PDFInfo
- Publication number
- CN113138891A CN113138891A CN202010062950.2A CN202010062950A CN113138891A CN 113138891 A CN113138891 A CN 113138891A CN 202010062950 A CN202010062950 A CN 202010062950A CN 113138891 A CN113138891 A CN 113138891A
- Authority
- CN
- China
- Prior art keywords
- log
- early warning
- logs
- configurator
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 15
- 238000005192 partition Methods 0.000 claims description 18
- 238000012423 maintenance Methods 0.000 description 3
- 238000000034 method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a service monitoring system based on logs, which comprises a log collector, a log storage, a log analyzer, an early warning configurator and a message notifier, wherein the log collector is used for collecting log data; the log collector is used for collecting logs of a plurality of service systems; the log storage is used for storing the logs collected by the log collector; the log analyzer is used for analyzing the logs collected by the log collector according to the early warning rules configured by the early warning configurator; the early warning configurator is used for configuring a plurality of corresponding early warning rules according to a plurality of service systems; and the message notifier is used for sending the early warning message to the service system corresponding to the log when the log analyzer analyzes that a certain log triggers the early warning rule. The invention can avoid finding out the problem only by checking a plurality of service systems one by one.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to a service monitoring system based on logs.
Background
Under the scenes of small size of a large system of a website and wide use of micro services, problems among the services are complex to find, debugging is difficult, and the problems are always known after the knowledge. Especially, when a plurality of systems work in cooperation with each other, a problem is caused, and a technician can guess the problem caused by which system only by experience, sometimes needs to check the plurality of systems one by one to find the problem, and needs to spend a lot of manpower and material resources.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a log-based service monitoring system, which can monitor a plurality of service systems in real time, and timely warn operation and maintenance personnel when a problem occurs, so as to avoid finding out the problem only when a plurality of service systems are examined one by one.
In order to solve the technical problems, the invention adopts the technical scheme that: a business monitoring system based on logs comprises a log collector, a log memory, a log analyzer, an early warning configurator and a message notifier; the log collector is used for collecting logs of a plurality of service systems; the log storage is used for storing the logs collected by the log collector; the log analyzer is used for analyzing the logs collected by the log collector according to the early warning rules configured by the early warning configurator; the early warning configurator is used for configuring a plurality of corresponding early warning rules according to a plurality of service systems; and the message notifier is used for sending the early warning message to the service system corresponding to the log when the log analyzer analyzes that a certain log triggers the early warning rule.
In the service monitoring system based on the log, the log storage includes a plurality of storage partitions, each storage partition corresponds to one service system, and logs of different service systems are stored in different storage partitions.
In the above service monitoring system based on logs, when the log analyzer analyzes the logs collected by the log collector according to the early warning rule configured by the early warning configurator, the log analyzer includes: calling an early warning rule from an early warning configurator; finding out a service system identifier corresponding to the early warning rule; according to the found service system identification, finding out a corresponding storage partition from the log memory for log calling; and analyzing the log according to the early warning rule.
When the log storage stores the log, the log storage adds the source service system identifier and the standardized log format to the log, and the standardized log format comprises the service system identifier, the log time, the log level and the log content.
Compared with the prior art, the invention has the following advantages: according to the invention, the logs of the plurality of service systems are collected, and the collected logs are analyzed by using the configured early warning, so that the real-time monitoring of each service system is realized, the operation and maintenance personnel are warned in time when a problem is found, and the problem can be found only by carrying out one-by-one troubleshooting on the plurality of service systems.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
Fig. 1 is a schematic block diagram of the present invention.
FIG. 2 is a flow chart of a method for analyzing logs according to the present invention.
Description of reference numerals:
1-log collector; 2-log storage; 3-log analyzer;
4-early warning configurator; 5-a message notifier; 6-service system.
Detailed Description
As shown in fig. 1, a log-based traffic monitoring system includes a log collector 1, a log storage 2, a log analyzer 3, an early warning configurator 4 and a message notifier 5;
the log collector 1 is used for collecting logs of a plurality of service systems 6;
the log collector 1 performs log collection by the syslog protocol.
The log storage 2 is used for storing the logs collected by the log collector 1;
the log memory 2 comprises a plurality of memory partitions, each memory partition corresponds to one service system 6, and logs of different service systems 6 are stored in different memory partitions.
When the log memory 2 stores the log, the log comprises a source service system identifier and a standardized log format, wherein the standardized log format comprises a service system identifier, log time, log level and log content.
The log storage 2 may be one of a hard disk, DVD, WORM, tape, RDBMS, log specific storage cloud. The log may be stored in a text-based, binary, or compressed format.
The log analyzer 3 is configured to analyze the logs collected by the log collector 1 according to the early warning rule configured by the early warning configurator 4;
the log analyzer 3, when analyzing the log collected by the log collector 1 according to the early warning rule configured by the early warning configurator 4, includes:
s1, calling the early warning rule from the early warning configurator 4;
s2, finding out the service system mark corresponding to the early warning rule;
s3, finding out the corresponding storage partition from the log memory 2 according to the found service system identification to call the log; after the logs in the storage partition are called, the logs are deleted from the storage partition, so that the storage space is prevented from being occupied;
and S4, analyzing the log according to the early warning rule.
The early warning configurator 4 is used for configuring a plurality of corresponding early warning rules according to a plurality of service systems 6;
early warning rules, such as log filtering rules, filter fields in the log and trigger early warning when a particular field is filtered. In actual implementation, the early warning rule is configured by a program worker according to the condition of the service system 6, and the configured early warning rule should have an identifier corresponding to the service system 6.
And the message notifier 5 is configured to send an early warning message to the service system 6 corresponding to a log when the log analyzer 3 analyzes that a certain log triggers an early warning rule.
When the invention is actually used, the log collector 1 collects logs of a plurality of service systems 6, the collected logs are stored in the log memory 2 through a standardized format process, and the logs generated by each service system 6 are stored in a corresponding storage partition, namely, the logs stored in each storage partition in the log memory 2 are from the same service system 6, the log analyzer 3 calls the early warning rules in the early warning configurator 4 in a periodic calling manner, for example, the log analyzer 3 calls the early warning rules corresponding to the service system a from the early warning configurator 4, the logs generated by the service system a are stored in the storage partition a, the log analyzer 3 analyzes the logs in the storage partition a by using the called early warning rules, when the log analyzer 3 analyzes that a certain log triggers the early warning rules, the trigger message notifier 5 sends an early warning message to the service system 6 corresponding to the log. Therefore, real-time monitoring of each service system 6 is achieved, when the service system 6 has problems, operation and maintenance personnel can be warned in time, and the problem is avoided being eliminated and found after a customer complaint comes.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and all simple modifications, changes and equivalent structural changes made to the above embodiment according to the technical spirit of the present invention still fall within the protection scope of the technical solution of the present invention.
Claims (4)
1. A service monitoring system based on logs is characterized by comprising a log collector, a log memory, a log analyzer, an early warning configurator and a message notifier;
the log collector is used for collecting logs of a plurality of service systems;
the log storage is used for storing the logs collected by the log collector;
the log analyzer is used for analyzing the logs collected by the log collector according to the early warning rules configured by the early warning configurator;
the early warning configurator is used for configuring a plurality of corresponding early warning rules according to a plurality of service systems;
and the message notifier is used for sending the early warning message to the service system corresponding to the log when the log analyzer analyzes that a certain log triggers the early warning rule.
2. The log-based traffic monitoring system of claim 1, wherein the log memory comprises a plurality of memory partitions, each memory partition corresponding to a traffic system, and logs for different traffic systems are stored in different memory partitions.
3. The log-based traffic monitoring system of claim 2, wherein the log analyzer analyzes the log collected by the log collector according to the pre-warning rule configured by the pre-warning configurator, and comprises:
calling an early warning rule from an early warning configurator;
finding out a service system identifier corresponding to the early warning rule;
according to the found service system identification, finding out a corresponding storage partition from the log memory for log calling;
and analyzing the log according to the early warning rule.
4. A log-based traffic monitoring system according to claim 2 or 3, wherein the log storage stores the log by adding a source service system identifier and a standardized log format to the log, the standardized log format including the service system identifier, the log time, the log level and the log content.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010062950.2A CN113138891A (en) | 2020-01-19 | 2020-01-19 | Service monitoring system based on log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010062950.2A CN113138891A (en) | 2020-01-19 | 2020-01-19 | Service monitoring system based on log |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113138891A true CN113138891A (en) | 2021-07-20 |
Family
ID=76809902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010062950.2A Pending CN113138891A (en) | 2020-01-19 | 2020-01-19 | Service monitoring system based on log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113138891A (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009053992A (en) * | 2007-08-28 | 2009-03-12 | Jiec Co Ltd | Log collection system |
| US20090171879A1 (en) * | 2007-12-28 | 2009-07-02 | Software Ag | Systems and/or methods for prediction and/or root cause analysis of events based on business activity monitoring related data |
| CN106385331A (en) * | 2016-09-08 | 2017-02-08 | 努比亚技术有限公司 | Method and system for monitoring alarm based on log |
| CN107357804A (en) * | 2017-05-24 | 2017-11-17 | 上海你我贷互联网金融信息服务有限公司 | The analysis system and method for internet finance massive logs |
| CN107612730A (en) * | 2017-09-18 | 2018-01-19 | 山东浪潮云服务信息科技有限公司 | A kind of log collection analysis method, device and system |
| CN107682351A (en) * | 2017-10-20 | 2018-02-09 | 携程旅游网络技术（上海）有限公司 | Method, system, equipment and the storage medium of network security monitoring |
| CN107766208A (en) * | 2017-10-27 | 2018-03-06 | 深圳市中润四方信息技术有限公司 | A kind of method, system and device of monitoring business system |
| CN108011925A (en) * | 2017-11-01 | 2018-05-08 | 北京神州绿盟信息安全科技股份有限公司 | A kind of operating audit system and method |
| CN108170580A (en) * | 2017-11-22 | 2018-06-15 | 链家网（北京）科技有限公司 | A kind of rule-based log alarming method, apparatus and system |
| CN108170538A (en) * | 2017-12-08 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of information processing method, device and electronic equipment |
| CN108322350A (en) * | 2018-02-27 | 2018-07-24 | 阿里巴巴集团控股有限公司 | Business monitoring method and device and electronic equipment |
| CN109189736A (en) * | 2018-08-01 | 2019-01-11 | 中国联合网络通信集团有限公司 | A kind of generation method and device of alarm association rule |
| CN109344243A (en) * | 2018-10-23 | 2019-02-15 | 北京天安智慧信息技术有限公司 | A kind of real-time stream calculation alarm analysis method and system |
| CN110333984A (en) * | 2019-06-05 | 2019-10-15 | 阿里巴巴集团控股有限公司 | Interface method for detecting abnormality, device, server and system |
-
2020
- 2020-01-19 CN CN202010062950.2A patent/CN113138891A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009053992A (en) * | 2007-08-28 | 2009-03-12 | Jiec Co Ltd | Log collection system |
| US20090171879A1 (en) * | 2007-12-28 | 2009-07-02 | Software Ag | Systems and/or methods for prediction and/or root cause analysis of events based on business activity monitoring related data |
| CN106385331A (en) * | 2016-09-08 | 2017-02-08 | 努比亚技术有限公司 | Method and system for monitoring alarm based on log |
| CN107357804A (en) * | 2017-05-24 | 2017-11-17 | 上海你我贷互联网金融信息服务有限公司 | The analysis system and method for internet finance massive logs |
| CN107612730A (en) * | 2017-09-18 | 2018-01-19 | 山东浪潮云服务信息科技有限公司 | A kind of log collection analysis method, device and system |
| CN107682351A (en) * | 2017-10-20 | 2018-02-09 | 携程旅游网络技术（上海）有限公司 | Method, system, equipment and the storage medium of network security monitoring |
| CN107766208A (en) * | 2017-10-27 | 2018-03-06 | 深圳市中润四方信息技术有限公司 | A kind of method, system and device of monitoring business system |
| CN108011925A (en) * | 2017-11-01 | 2018-05-08 | 北京神州绿盟信息安全科技股份有限公司 | A kind of operating audit system and method |
| CN108170580A (en) * | 2017-11-22 | 2018-06-15 | 链家网（北京）科技有限公司 | A kind of rule-based log alarming method, apparatus and system |
| CN108170538A (en) * | 2017-12-08 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of information processing method, device and electronic equipment |
| CN108322350A (en) * | 2018-02-27 | 2018-07-24 | 阿里巴巴集团控股有限公司 | Business monitoring method and device and electronic equipment |
| CN109189736A (en) * | 2018-08-01 | 2019-01-11 | 中国联合网络通信集团有限公司 | A kind of generation method and device of alarm association rule |
| CN109344243A (en) * | 2018-10-23 | 2019-02-15 | 北京天安智慧信息技术有限公司 | A kind of real-time stream calculation alarm analysis method and system |
| CN110333984A (en) * | 2019-06-05 | 2019-10-15 | 阿里巴巴集团控股有限公司 | Interface method for detecting abnormality, device, server and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112612675B (en) | Distributed big data log link tracking method and system under micro-service architecture | |
| CN112422344A (en) | Log abnormity warning method and device, storage medium and electronic device | |
| CN111176879A (en) | Fault repairing method and device for equipment | |
| CN106371986A (en) | Log treatment operation and maintenance monitoring system | |
| CN101997925A (en) | Server monitoring method with early warning function and system thereof | |
| CN108737170A (en) | A kind of batch daily record abnormal data alarm method and device | |
| CN103425750A (en) | Cross-platform and cross-application log collecting system and collecting managing method thereof | |
| CN104699759A (en) | Method for maintaining automatic operation of database | |
| US11322013B2 (en) | Monitoring method of MES, monitoring device, and readable storage medium | |
| CN109034423B (en) | Fault early warning judgment method, device, equipment and storage medium | |
| CN110224865A (en) | A kind of log warning system based on Stream Processing | |
| CN113298486A (en) | Big data-based government affair supervision and supervision method and system | |
| CN111752808A (en) | Method for implementing data sharing exchange service operation monitoring system | |
| CN114356499A (en) | Kubernetes cluster alarm root cause analysis method and device | |
| CN113918412A (en) | Real-time abnormal log analysis method and system | |
| CN110784352B (en) | Data synchronous monitoring and alarming method and device based on Oracle golden gate | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN112929202A (en) | Early warning system of distributed data node abnormal behavior | |
| CN113138891A (en) | Service monitoring system based on log | |
| CN108449212B (en) | MAS message transmission method based on event association | |
| CN112087320A (en) | Abnormity positioning method and device, electronic equipment and readable storage medium | |
| CN115934464A (en) | Information platform monitoring and collecting system | |
| CN213579446U (en) | Fault detection, operation and maintenance reporting and monitoring system | |
| CN114257414A (en) | Intelligent network security duty method and system | |
| CN114265669A (en) | Destroyed container instance identification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210720 |