CN112003831A - Log collection system and log collection method in distributed environment - Google Patents
Log collection system and log collection method in distributed environment Download PDFInfo
- Publication number
- CN112003831A CN112003831A CN202010742719.8A CN202010742719A CN112003831A CN 112003831 A CN112003831 A CN 112003831A CN 202010742719 A CN202010742719 A CN 202010742719A CN 112003831 A CN112003831 A CN 112003831A
- Authority
- CN
- China
- Prior art keywords
- log
- service
- traceid
- distributed environment
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0209—Architectural arrangements, e.g. perimeter networks or demilitarized zones
- H04L63/0218—Distributed architectures, e.g. distributed firewalls
Abstract
The invention discloses a log collection system and a log collection method in a distributed environment, wherein the system comprises: the user request receiving module is used for receiving a service request from a user, generating a corresponding traceID, storing the traceID into a system thread and transmitting the received service request backwards; the service execution module is used for receiving the service request, executing specific service operation and storing and recording a service log in the service operation process; and the log recording module is used for recording logs, issuing log messages, expanding a frame of the log messages, and then extracting and transmitting the traceID from the system thread. The invention aims at the actual characteristics of the distributed environment, realizes the collection, arrangement and sequencing of the logs in each application node, provides a traceID scheme for a request link, enables the whole request to be connected in series, and effectively solves the problems of log collection and link performance analysis in the distributed environment.
Description
Technical Field
The invention relates to a log collection system and a log collection method using the same, in particular to a log collection system and a log collection method applied to a distributed environment and suitable for java language development, and belongs to the technical field of the Internet.
Background
With the rapid development of internet technology in recent years, the number of users has increased dramatically year by year. In the process of network service, various high concurrency scenes are more and more common, which directly leads to the fact that the number of application nodes in a network is multiplied, and in addition, due to the isolation between a production network and a development network in the prior art, the log information in the application is difficult to directly view under the production environment, and the problem is more prominent especially for a distributed environment.
In a distributed environment, log information is scattered in each server, so that the log information is difficult to distinguish accurately and cannot be sorted when being checked, and a request link cannot be checked, which causes great obstruction to quick positioning of problems when production conditions occur.
In the prior art, the related operations of log collection are mainly completed by ELK (elastissearch, Logstash, Kibana), but when log collection is performed by using these open source software, the following problems often occur:
first, the ELK can only perform log collection operations and cannot provide link logs, which is fatal in a distributed environment, meaning that users cannot monitor the performance of the link logs in real time.
In addition, taking the Logstash as an example, the Logstash cannot realize dynamic modification during configuration acquisition, and the Logstash needs to be configured separately, cannot monitor the acquisition process, and has a slightly low acquisition performance.
In summary, based on the above technical status, how to design a brand-new log collection system and log collection method applicable to the distributed environment and suitable for java language development is provided to overcome the shortcomings in the prior art, which is a problem of common attention of the technicians in the field.
Disclosure of Invention
In view of the foregoing defects in the prior art, an object of the present invention is to provide a log collection system and a log collection method applied in a distributed environment, which are suitable for java language development, and are specifically as follows.
A log collection system in a distributed environment, comprising:
the user request receiving module is used for receiving a service request from a user, generating a corresponding traceID, storing the traceID into a system thread and transmitting the received service request backwards;
the service execution module is used for receiving the service request from the user request receiving module, executing specific service operation and storing and recording a service log in the service operation process;
and the log recording module is used for recording logs and issuing log messages before the service request enters the service execution module and after the service execution module finishes service operation, expanding a frame of the log messages, and then extracting and transmitting the traceID from the system thread.
Preferably, the user request receiving module comprises:
the tracking ID generation submodule is used for receiving a service request from a user, generating a corresponding traceID according to the received service request of the user and storing the traceID into a shared variable of a system thread;
and the user request intercepting submodule is used for intercepting other service requests when the traceID is generated aiming at the service requests of the users according to a preset intercepting mode.
Preferably, the logging module comprises:
the log format definition submodule is used for defining and appointing the format of the log message and the frame of the log message;
and the log message processing submodule is used for acquiring the log message, performing corresponding judgment processing operation, and displaying the result after aggregation analysis.
A log collection method in a distributed environment is based on the log collection system in the distributed environment, and comprises the following steps:
s1, the user request receiving module receives the service request from the user, generates the corresponding traceID and stores the traceID into the shared variable of the system thread, and then transmits the received service request backwards;
s2, the service execution module receives the service request from the user request receiving module, executes the specific service operation, and stores and records the service log in the service operation process;
between S1 and S2 and after S2, a logging step is further included, which is as follows,
s0, the log recording module records the log and issues the log information, expands the frame of the log information, extracts the traceID from the shared variable of the system thread and transmits the traceID, and acquires the log information and carries out corresponding judgment processing operation.
Preferably, the traceID described in S1 is generated in a UUID or a random number.
Preferably, the logging step specifically includes:
s01, the log format definition sub-module defines and appoints the format of the log message and the frame of the log message, and then the log message processing sub-module records the log and issues the log message;
s02, acquiring the log message, expanding the frame of the log message, and then extracting the traceID from the shared variable of the system thread for transmission;
s03, temporarily storing the logs by using redis or kafka, if the temporary storage area is in an unavailable state, writing all the logs into a disk and sending an alarm signal, and after the temporary storage area is recovered, manually triggering the logs in the disk and writing the logs again;
s04, using redis or kafka to consume the log in real time, judging whether the condition of batch consumption is met and whether consumption is abnormal, if consumption is abnormal, trying after waiting, and if the number of attempts is exceeded, manually writing the log;
and S05, performing aggregation analysis on the elastic search by using the interface, displaying the operation result in the form of a webpage and a chart, and providing a query.
Preferably, the tool of the framework of the expanded log message in S02 is logback or log4j2 or log4j, and the manner of the expanded log message is a custom appendix.
Preferably, the logging output is performed in S02 in an asynchronous manner all the time.
Compared with the prior art, the invention has the advantages that:
the log collection system provided by the invention aims at the actual characteristics of a distributed environment, realizes log collection, arrangement and sequencing in each application node, provides a traceID scheme for a request link, enables the whole request to be connected in series, and effectively solves the problems of log collection and link performance analysis in the distributed environment.
By combining the system scheme, the log collection method provided by the invention has the advantages of clear flow, high usability, no code intrusion in the whole execution process, strong overall adaptability and wide application range, and can provide log collection service for some produced projects only by introducing corresponding dependence and a small amount of configuration. Particularly for some common data frames such as dubbo, seluth and the like, the method is more convenient to apply and is convenient for large-scale popularization and application of enterprises.
In addition, the invention also provides a brand-new thought for the relevant research and application of log collection and link performance analysis in internet application, provides reference for other relevant problems in the same field, can be used for extended extension and in-depth research on the basis of the brand-new thought, and has wide industrial application prospect.
The following detailed description of the embodiments of the present invention is provided in connection with the accompanying drawings for the purpose of facilitating understanding and understanding of the technical solutions of the present invention.
Drawings
FIG. 1 is a schematic overall flow chart of the method of the present invention.
Detailed Description
The invention discloses a switch power amplifier circuit with fault-tolerant function and six-bridge-arm five-phase output for a five-degree-of-freedom magnetic suspension bearing, which adopts the following specific scheme.
As shown in fig. 1, a log collection system in a distributed environment includes:
the user request receiving module is used for receiving a service request from a user, generating a corresponding traceID, storing the traceID into a system thread and transmitting the received service request backwards;
the service execution module is used for receiving the service request from the user request receiving module, executing specific service operation and storing and recording a service log in the service operation process;
and the log recording module is used for recording logs and issuing log messages before the service request enters the service execution module and after the service execution module finishes service operation, expanding a frame of the log messages, and then extracting and transmitting the traceID from the system thread.
The user request receiving module comprises:
the tracking ID generation submodule is used for receiving a service request from a user, generating a corresponding traceID according to the received service request of the user and storing the traceID into a shared variable of a system thread;
and the user request intercepting submodule is used for intercepting other service requests when the traceID is generated aiming at the service requests of the users according to a preset intercepting mode.
The logging module comprises:
the log format definition submodule is used for defining and appointing the format of the log message and the frame of the log message;
and the log message processing submodule is used for acquiring the log message, performing corresponding judgment processing operation, and displaying the result after aggregation analysis.
The log collection system provided by the invention aims at the actual characteristics of a distributed environment, realizes log collection, arrangement and sequencing in each application node, provides a traceID scheme for a request link, enables the whole request to be connected in series, and effectively solves the problems of log collection and link performance analysis in the distributed environment.
A log collection method in a distributed environment is based on the log collection system in the distributed environment, and comprises the following steps:
s1, the user request receiving module receives the service request from the user, generates the corresponding traceID and stores it in the shared variable of the system thread, and then transmits the received service request backward.
The traceID is generated by UUID, random number, or the like, and the main purpose is to ensure uniqueness.
S2, the service executing module receives the service request from the user request receiving module, executes a specific service operation, and stores and records a service log in the service operation process.
According to the principle of the surround design, between S1 and S2 and after S2, a logging step is further included, which is as follows:
s0, the log recording module records the log and issues the log information, expands the frame of the log information, extracts the traceID from the shared variable of the system thread and transmits the traceID, and acquires the log information and carries out corresponding judgment processing operation.
Further, the logging step specifically includes:
s01, the log format definition sub-module defines and appoints the format of the log message and the frame of the log message, and then the log message processing sub-module records the log and issues the log message;
s02, acquiring the log message, expanding the frame of the log message, and then extracting the traceID from the shared variable of the system thread for transmission;
the tool of the framework of the extended log message is logback or log4j2 or log4j, and the extended log message is in a custom appendix (a plug-in outputting log format). According to the format of the appointed output log, the link log can be combined by matching with the traceId. In addition, in the operation process, the performance is improved by adopting an asynchronous mode in the whole process when the log is output, and the operation of the main service of the system is not influenced.
S03, temporary storage of the logs is carried out by using a redis (cache frame) or a kafka (message frame), if the temporary storage area is in an unavailable state, all the logs are written into the disk and an alarm signal is sent out, and after the temporary storage area is recovered, the logs in the disk are manually triggered and written again;
s04, using redis (cache frame) or kafka (message frame) to consume the log in real time, judging whether the condition of batch consumption (such as consumption once every 100 milliseconds or how many times) is reached and whether consumption is abnormal, if consumption is abnormal, waiting for the test, and if the number of attempts is exceeded, manually writing the log;
and S05, performing aggregation analysis on the elastic search by using the interface, displaying the operation result in the form of a webpage and a chart, and providing a query.
By combining the system scheme, the log collection method provided by the invention has the advantages of clear flow, high usability, no code intrusion in the whole execution process, strong overall adaptability and wide application range, and can provide log collection service for some produced projects only by introducing corresponding dependence and a small amount of configuration. Particularly for some common data frames such as dubbo, seluth and the like, the method is more convenient to apply and is convenient for large-scale popularization and application of enterprises.
In addition, the invention also provides a brand-new thought for the relevant research and application of log collection and link performance analysis in internet application, provides reference for other relevant problems in the same field, can be used for extended extension and in-depth research on the basis of the brand-new thought, and has wide industrial application prospect.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Finally, it should be understood that although the present description refers to embodiments, not every embodiment contains only a single technical solution, and such description is for clarity only, and those skilled in the art should integrate the description, and the technical solutions in the embodiments can be appropriately combined to form other embodiments understood by those skilled in the art.
Claims (8)
1. A log collection system in a distributed environment, comprising:
the user request receiving module is used for receiving a service request from a user, generating a corresponding traceID, storing the traceID into a system thread and transmitting the received service request backwards;
the service execution module is used for receiving the service request from the user request receiving module, executing specific service operation and storing and recording a service log in the service operation process;
and the log recording module is used for recording logs and issuing log messages before the service request enters the service execution module and after the service execution module finishes service operation, expanding a frame of the log messages, and then extracting and transmitting the traceID from the system thread.
2. The log collection system in a distributed environment according to claim 1, wherein said user request receiving module comprises:
the tracking ID generation submodule is used for receiving a service request from a user, generating a corresponding traceID according to the received service request of the user and storing the traceID into a shared variable of a system thread;
and the user request intercepting submodule is used for intercepting other service requests when the traceID is generated aiming at the service requests of the users according to a preset intercepting mode.
3. The system for collecting logs in a distributed environment according to claim 1, wherein said logging module comprises:
the log format definition submodule is used for defining and appointing the format of the log message and the frame of the log message;
and the log message processing submodule is used for acquiring the log message, performing corresponding judgment processing operation, and displaying the result after aggregation analysis.
4. A log collection method in a distributed environment based on the log collection system in the distributed environment according to any one of claims 1 to 3, comprising the steps of:
s1, the user request receiving module receives the service request from the user, generates the corresponding traceID and stores the traceID into the shared variable of the system thread, and then transmits the received service request backwards;
s2, the service execution module receives the service request from the user request receiving module, executes the specific service operation, and stores and records the service log in the service operation process;
between S1 and S2 and after S2, a logging step is further included, which is as follows,
s0, the log recording module records the log and issues the log information, expands the frame of the log information, extracts the traceID from the shared variable of the system thread and transmits the traceID, and acquires the log information and carries out corresponding judgment processing operation.
5. The log collection method in a distributed environment according to claim 4, wherein: the traceID described in S1 is generated in the form of a UUID or a random number.
6. The method for collecting logs in a distributed environment according to claim 4, wherein the logging step specifically comprises:
s01, the log format definition sub-module defines and appoints the format of the log message and the frame of the log message, and then the log message processing sub-module records the log and issues the log message;
s02, acquiring the log message, expanding the frame of the log message, and then extracting the traceID from the shared variable of the system thread for transmission;
s03, temporarily storing the logs by using redis or kafka, if the temporary storage area is in an unavailable state, writing all the logs into a disk and sending an alarm signal, and after the temporary storage area is recovered, manually triggering the logs in the disk and writing the logs again;
s04, using redis or kafka to consume the log in real time, judging whether the condition of batch consumption is met and whether consumption is abnormal, if consumption is abnormal, trying after waiting, and if the number of attempts is exceeded, manually writing the log;
and S05, performing aggregation analysis on the elastic search by using the interface, displaying the operation result in the form of a webpage and a chart, and providing a query.
7. The log collection method in a distributed environment according to claim 6, wherein: the tool of the frame of the expanded log message in the S02 is logback or log4j2 or log4j, and the mode of the expanded log message is a custom appendix.
8. The log collection method in a distributed environment according to claim 6, wherein: in S02, the log output is performed all the way asynchronously.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010742719.8A CN112003831A (en) | 2020-07-29 | 2020-07-29 | Log collection system and log collection method in distributed environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010742719.8A CN112003831A (en) | 2020-07-29 | 2020-07-29 | Log collection system and log collection method in distributed environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112003831A true CN112003831A (en) | 2020-11-27 |
Family
ID=73462351
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010742719.8A Pending CN112003831A (en) | 2020-07-29 | 2020-07-29 | Log collection system and log collection method in distributed environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112003831A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114629929A (en) * | 2022-03-16 | 2022-06-14 | 北京奇艺世纪科技有限公司 | Log recording method, device and system |
| CN114697204A (en) * | 2022-03-21 | 2022-07-01 | 昭通亮风台信息科技有限公司 | Log collection method and system for distributed system |
| CN115883666A (en) * | 2022-11-14 | 2023-03-31 | 天翼云科技有限公司 | One-stop request link monitoring system based on CDN node |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106354765A (en) * | 2016-08-19 | 2017-01-25 | 广东亿迅科技有限公司 | Log analysis system and method based on distributed collection |
| CN108038145A (en) * | 2017-11-23 | 2018-05-15 | 携程旅游网络技术（上海）有限公司 | Distributed Services tracking, system, storage medium and electronic equipment |
| CN109600375A (en) * | 2018-12-13 | 2019-04-09 | 锐捷网络股份有限公司 | Message tracing method, device, electronic equipment and storage medium |
| CN109656700A (en) * | 2018-12-17 | 2019-04-19 | 广州市玄武无线科技股份有限公司 | Distributed link tracking, system, equipment and storage medium under multi-tenant |
| CN110069354A (en) * | 2019-04-15 | 2019-07-30 | 必成汇(成都)科技有限公司 | The full link trace method of micro services and micro services framework |
| CN111459766A (en) * | 2019-11-14 | 2020-07-28 | 国网浙江省电力有限公司信息通信分公司 | Calling chain tracking and analyzing method for micro-service system |
-
2020
- 2020-07-29 CN CN202010742719.8A patent/CN112003831A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106354765A (en) * | 2016-08-19 | 2017-01-25 | 广东亿迅科技有限公司 | Log analysis system and method based on distributed collection |
| CN108038145A (en) * | 2017-11-23 | 2018-05-15 | 携程旅游网络技术（上海）有限公司 | Distributed Services tracking, system, storage medium and electronic equipment |
| CN109600375A (en) * | 2018-12-13 | 2019-04-09 | 锐捷网络股份有限公司 | Message tracing method, device, electronic equipment and storage medium |
| CN109656700A (en) * | 2018-12-17 | 2019-04-19 | 广州市玄武无线科技股份有限公司 | Distributed link tracking, system, equipment and storage medium under multi-tenant |
| CN110069354A (en) * | 2019-04-15 | 2019-07-30 | 必成汇(成都)科技有限公司 | The full link trace method of micro services and micro services framework |
| CN111459766A (en) * | 2019-11-14 | 2020-07-28 | 国网浙江省电力有限公司信息通信分公司 | Calling chain tracking and analyzing method for micro-service system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114629929A (en) * | 2022-03-16 | 2022-06-14 | 北京奇艺世纪科技有限公司 | Log recording method, device and system |
| CN114629929B (en) * | 2022-03-16 | 2024-03-08 | 北京奇艺世纪科技有限公司 | Log recording method, device and system |
| CN114697204A (en) * | 2022-03-21 | 2022-07-01 | 昭通亮风台信息科技有限公司 | Log collection method and system for distributed system |
| CN115883666A (en) * | 2022-11-14 | 2023-03-31 | 天翼云科技有限公司 | One-stop request link monitoring system based on CDN node |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112003831A (en) | Log collection system and log collection method in distributed environment | |
| CN110362544B (en) | Log processing system, log processing method, terminal and storage medium | |
| CN101997925A (en) | Server monitoring method with early warning function and system thereof | |
| WO2021169268A1 (en) | Data processing method, apparatus and device, and storage medium | |
| US20130081001A1 (en) | Immediate delay tracker tool | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN112347071A (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN112000350A (en) | Dynamic rule updating method, device and storage medium | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| CN112235262A (en) | Message analysis method and device, electronic equipment and computer readable storage medium | |
| CN111488257A (en) | Cloud service link tracking monitoring method, device, equipment and storage medium | |
| CN113360413A (en) | Code testing method and device | |
| CN114036167A (en) | Method for recording data change operation log in micro service | |
| CN112182086A (en) | Data transmission interaction system and method for rear-end interface of meteorological data warehouse | |
| CN110928885B (en) | Method and device for updating data of Mysql database to Es database | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| CN112445861A (en) | Information processing method, device, system and storage medium | |
| CN113742408B (en) | Protobuf protocol dynamic analysis-based data interaction method | |
| CN114844771A (en) | Monitoring method, device, storage medium and program product for micro-service system | |
| CN114510398A (en) | Anomaly monitoring method, apparatus, device, system and medium | |
| CN114625763A (en) | Information analysis method and device for database, electronic equipment and readable medium | |
| CN113886472A (en) | Data access system, access method, computer equipment and storage medium | |
| CN113779117A (en) | Data monitoring method and device, storage medium and electronic equipment | |
| CN116074388B (en) | Flow forwarding method and system based on log queue | |
| CN111654410B (en) | Gateway request monitoring method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201127 |
|
| RJ01 | Rejection of invention patent application after publication |