CN112214459A - Resource processing flow log collection system based on event mechanism - Google Patents
Resource processing flow log collection system based on event mechanism Download PDFInfo
- Publication number
- CN112214459A CN112214459A CN202011082574.XA CN202011082574A CN112214459A CN 112214459 A CN112214459 A CN 112214459A CN 202011082574 A CN202011082574 A CN 202011082574A CN 112214459 A CN112214459 A CN 112214459A
- Authority
- CN
- China
- Prior art keywords
- log
- resource
- log information
- information
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 29
- 230000007246 mechanism Effects 0.000 title claims abstract description 14
- 230000008859 change Effects 0.000 claims abstract description 14
- 238000005111 flow chemistry technique Methods 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000012423 maintenance Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a resource processing flow log collection system based on an event mechanism, and belongs to the field of log query. The invention comprises the following steps: the resource flow processing module is used for grouping the log information of the resources according to the service assembly or the program flow generated by the log; the resource storage module is used for storing resource data, generating a resource state change event when the resource state changes, and notifying the change event to the log collection module in the form of log information; the log collection module is used for sending the log information to a log server when the log information is received; and the log server is used for summarizing and analyzing the received log information according to the grouping condition of the log information of the resources, and when the log server inquires the log information, the processing flow condition of the state change condition of the resources according to the time sequence can be inquired. The invention can carry out quick positioning when the system has a fault.
Description
Technical Field
The invention relates to the field of log query, in particular to a resource processing flow log collection system based on an event mechanism.
Background
The log is an important component of the system, records the operation condition of the system, and when the system fails, the operation and maintenance and development personnel of the system can quickly know the errors in the operation process of the system through the log. With the increasing system scale and the increasing log scale, the statistics and retrieval of the logs become more and more difficult, so that when a fault occurs, a user can find out which process the system is in, and often needs to spend a lot of time to inquire and analyze the fault reason from the huge log amount. In a distributed system, the traditional log checking means needs an operation and maintenance manager to check a corresponding log file on each server, which is a task that is almost impossible to implement in the distributed system. In a traditional log query method, an operation and maintenance manager needs to roughly locate a log, for example, find a log file of a certain service, or obtain time information of an error and then obtain log information according to a time range.
Disclosure of Invention
The invention aims to provide a resource processing flow log collecting system based on an event mechanism, which can carry out quick positioning when a system fails.
The invention solves the technical problem, and adopts the technical scheme that:
the resource processing flow log collecting system based on the event mechanism comprises: the system comprises a resource flow processing module, a resource storage module, a log collection module and a log server;
the resource flow processing module is used for grouping the log information of the resources according to the service components or program flows generated by the logs;
the resource storage module is used for storing resource data, generating a resource state change event when the resource state changes, and notifying the change event to the log collection module in the form of log information;
the log collection module is used for sending the log information to a log server when the log information is received;
the log server is used for collecting and analyzing the received log information according to the grouping condition of the log information of the resources, and when the log server inquires the log information, the processing flow condition of the state change condition of the resources according to the time sequence can be inquired.
Further, it is assumed that each resource must have a status field, and the resource is in different status at different processing stages, and the program flow proceeds to a stage, and the system will send the log information of the resource at the stage.
Further, all resources and corresponding processing flows can send log information through the system when going to a certain stage, and after receiving the log, the log collection module sends the log information to the log server through remote calling.
Further, after receiving the log information and performing summary analysis on the log information, the log server stores the log information into the key value database.
Further, log information is queried through a query interface of the log server, and when the log information is queried, the log information generated by the corresponding service component or the business process is queried according to the system component or the specific resource in the cloud platform.
The invention has the advantages that a series of service processing flows are regarded as a resource processing through the resource processing flow log collection system based on the event mechanism, the processed resources are classified, the change state of the resources is recorded and is stored in the database according to the occurrence time sequence of the events when the resources change, the processing flow of the resources can be quickly positioned when the system fails, the fault positioning time is greatly saved, and the retrieval and analysis efficiency is improved. Through the operation of an actual cloud platform, the log query efficiency is greatly improved, and meanwhile, the log center also collects the log information of each component in the distributed system to the central service, so that convenience is brought to log investigation.
In addition, when the system fails, development and operation and maintenance personnel need to quickly locate the reason of the failure, so that quick repair is facilitated, and risks and losses caused by system downtime are reduced. In order to solve the problem of quick positioning errors, the log and each resource processing flow of the cloud platform are managed and grouped and summarized according to various resources, so that operation and maintenance management personnel of the cloud platform can position corresponding problem points through specific resources and corresponding services.
Drawings
Fig. 1 is a block diagram of a resource processing flow log collection system based on an event mechanism according to an embodiment of the present invention.
Detailed Description
The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.
Examples
The present embodiment provides an event mechanism-based resource processing flow log collection system, a block diagram of which is shown in fig. 1, wherein the system includes: the system comprises a resource flow processing module, a resource storage module, a log collection module and a log server; wherein: the resource flow processing module is used for grouping the log information of the resources according to the service components or program flows generated by the logs, and is arranged on the cloud platform to facilitate user-defined grouping; the resource storage module is used for storing resource data, generating a resource state change event when the resource state changes, and notifying the change event to the log collection module in the form of log information; the log collection module is used for sending the log information to a log server when the log information is received; the log server is used for summarizing and analyzing the received log information according to the grouping condition of the log information of the resources, and when the log server inquires the log information, the processing flow condition of the state change condition of the resources according to the time sequence can be inquired, so that the resource processing condition can be quickly positioned.
In the above system, in order to effectively monitor the status of the resource, it may be set that each resource must have a status field, and is in different status at different processing stages, and when the program flow goes to a certain stage, the system will send the log information of the resource at the stage.
It should be noted that all resources and corresponding processing flows send log information through the system at a certain stage, and after receiving the log, the log collection module sends the log information to the log server through remote invocation. After receiving the log information and performing summary analysis on the log information, the log server can store the log information in the key value database.
In a specific application process, a user queries log information through a query interface of a log server, and when the log information is queried, the log information generated by a corresponding service component or a service process can be queried according to a system component or a specific resource in a cloud platform.
In the embodiment, through the log server and the log storage process, the constructed log system achieves the effect of grouping the logs according to the generated service components or program processes, so that when the logs are positioned according to problems or some problems of a certain service are checked, the efficiency of log query is greatly improved, a large amount of labor cost is saved, and great economic benefits are generated for enterprises using the cloud platform; and grouping is combined with traditional program log records, so that the effect of log grouping is achieved, logs of all service components in the whole distributed system are gathered through a log center, log query efficiency is higher than that of a traditional log system by combining the capacity of grouping and aggregating the logs, and the log system constructed by the method even exceeds the query efficiency of the traditional log center by combining the power of grouping the logs.
In the embodiment, in the specific implementation process:
firstly, the specific program flows are grouped in the cloud platform, so that the logs generated in each flow have the characteristic of grouping, the logs can be inquired according to a certain specific program flow served by people when being inquired, and the traditional log inquiry means, such as inquiring all log contents of a certain log file in a certain time period, and then analyzing the log by the self is not needed. The efficiency of log query is greatly improved by directly positioning the packets.
Secondly, the log center server applies a self-research distributed storage technology, achieves the capacity of high availability at the limit, has much higher reliability than the traditional log center, and provides query according to conditions such as log grouping or service components, and the like, so that log data can be conveniently obtained according to the requirements during query.
Claims (5)
1. The resource processing flow log collection system based on the event mechanism is characterized by comprising the following steps: the system comprises a resource flow processing module, a resource storage module, a log collection module and a log server;
the resource flow processing module is used for grouping the log information of the resources according to the service components or program flows generated by the logs;
the resource storage module is used for storing resource data, generating a resource state change event when the resource state changes, and notifying the change event to the log collection module in the form of log information;
the log collection module is used for sending the log information to a log server when the log information is received;
the log server is used for collecting and analyzing the received log information according to the grouping condition of the log information of the resources, and when the log server inquires the log information, the processing flow condition of the state change condition of the resources according to the time sequence can be inquired.
2. The event mechanism-based resource processing flow log collection system of claim 1, wherein each resource is configured to have a status field, and to be in different status at different processing stages, and the program flow proceeds to a stage, and the system sends log information of the resource at the stage.
3. The event mechanism-based resource processing flow log collection system according to claim 1, wherein all resources and corresponding processing flows send log information through the system when proceeding to a certain stage, and the log collection module sends the log information to the log server through remote invocation after receiving the log.
4. The event mechanism-based resource processing flow log collection system of claim 1, wherein the log server stores the log information in the key-value database after receiving the log information and performing summary analysis on the log information.
5. The event mechanism-based resource processing flow log collection system according to any one of claims 1 to 4, wherein log information is queried through a query interface of a log server, and when log information is queried, log information generated by a corresponding service component or a business flow is queried according to a system component or a specific resource in a cloud platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011082574.XA CN112214459A (en) | 2020-10-12 | 2020-10-12 | Resource processing flow log collection system based on event mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011082574.XA CN112214459A (en) | 2020-10-12 | 2020-10-12 | Resource processing flow log collection system based on event mechanism |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112214459A true CN112214459A (en) | 2021-01-12 |
Family
ID=74054363
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011082574.XA Pending CN112214459A (en) | 2020-10-12 | 2020-10-12 | Resource processing flow log collection system based on event mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112214459A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704014A (en) * | 2021-08-24 | 2021-11-26 | 荣耀终端有限公司 | Log acquisition system, method, electronic device and storage medium |
| CN114726714A (en) * | 2022-02-18 | 2022-07-08 | 珠海紫讯信息科技有限公司 | WebRPA operation and maintenance method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103632331A (en) * | 2013-11-26 | 2014-03-12 | 福建四创软件有限公司 | Configurable plug-in type data service method |
| CN107343021A (en) * | 2017-05-22 | 2017-11-10 | 国网安徽省电力公司信息通信分公司 | A kind of Log Administration System based on big data applied in state's net cloud |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
| CN110263009A (en) * | 2019-06-21 | 2019-09-20 | 深圳前海微众银行股份有限公司 | Generation method, device, equipment and the readable storage medium storing program for executing of log classifying rules |
| CN110493348A (en) * | 2019-08-26 | 2019-11-22 | 山东融为信息科技有限公司 | A kind of intelligent monitoring and alarming system based on Internet of Things |
-
2020
- 2020-10-12 CN CN202011082574.XA patent/CN112214459A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103632331A (en) * | 2013-11-26 | 2014-03-12 | 福建四创软件有限公司 | Configurable plug-in type data service method |
| CN107343021A (en) * | 2017-05-22 | 2017-11-10 | 国网安徽省电力公司信息通信分公司 | A kind of Log Administration System based on big data applied in state's net cloud |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
| CN110263009A (en) * | 2019-06-21 | 2019-09-20 | 深圳前海微众银行股份有限公司 | Generation method, device, equipment and the readable storage medium storing program for executing of log classifying rules |
| CN110493348A (en) * | 2019-08-26 | 2019-11-22 | 山东融为信息科技有限公司 | A kind of intelligent monitoring and alarming system based on Internet of Things |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113704014A (en) * | 2021-08-24 | 2021-11-26 | 荣耀终端有限公司 | Log acquisition system, method, electronic device and storage medium |
| CN114726714A (en) * | 2022-02-18 | 2022-07-08 | 珠海紫讯信息科技有限公司 | WebRPA operation and maintenance method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112612675B (en) | Distributed big data log link tracking method and system under micro-service architecture | |
| CN109714192B (en) | Monitoring method and system for monitoring cloud platform | |
| CN110457190B (en) | Full link monitoring method, device and system based on block chain | |
| US9680716B2 (en) | System and method for monitoring and managing data center resources in real time incorporating manageability subsystem | |
| CN111176879A (en) | Fault repairing method and device for equipment | |
| CN110535713B (en) | Monitoring management system and monitoring management method | |
| US11322013B2 (en) | Monitoring method of MES, monitoring device, and readable storage medium | |
| CN111245672A (en) | Monitoring method and system for general extensible tracking service full link | |
| CN111259073A (en) | Intelligent business system running state studying and judging system based on logs, flow and business access | |
| US9311175B2 (en) | Method and system for processing log information | |
| CN112214459A (en) | Resource processing flow log collection system based on event mechanism | |
| CN112416902A (en) | One-key inspection method for host and database | |
| CN115269515A (en) | Processing method for searching specified target document data | |
| CN106951360B (en) | Data statistical integrity calculation method and system | |
| CN108809729A (en) | The fault handling method and device that CTDB is serviced in a kind of distributed system | |
| CN111143304B (en) | Micro-service system exception log analysis method based on request link | |
| CN113282464A (en) | Log monitoring method and system | |
| CN106789335B (en) | Method and system for processing information | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN113472881B (en) | Statistical method and device for online terminal equipment | |
| CN112306862B (en) | Front-end automatic test system and method, storage medium and computing device | |
| CN115840656A (en) | Automatic operation and maintenance method and system for application program based on fault self-healing | |
| CN112416638A (en) | Message middleware fault self-healing system | |
| CN112506886B (en) | Multi-source service operation log acquisition method and system | |
| CN115391286A (en) | Link tracking data management method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210112 |
|
| RJ01 | Rejection of invention patent application after publication |