CN116662218A - Method and device for collecting and processing logs in real time - Google Patents

Method and device for collecting and processing logs in real time Download PDF

Info

Publication number
CN116662218A
CN116662218A CN202310952297.0A CN202310952297A CN116662218A CN 116662218 A CN116662218 A CN 116662218A CN 202310952297 A CN202310952297 A CN 202310952297A CN 116662218 A CN116662218 A CN 116662218A
Authority
CN
China
Prior art keywords
log
data
log data
real time
search engine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310952297.0A
Other languages
Chinese (zh)
Inventor
刁志伟
冯晋阳
韩锡锐
蒋林雪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Deta Jingyao Information Technology Co ltd
Original Assignee
Beijing Deta Jingyao Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Deta Jingyao Information Technology Co ltd filed Critical Beijing Deta Jingyao Information Technology Co ltd
Priority to CN202310952297.0A priority Critical patent/CN116662218A/en
Publication of CN116662218A publication Critical patent/CN116662218A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/08Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0844Multiple simultaneous or quasi-simultaneous cache accessing
    • G06F12/0853Cache with multiport tag or data arrays
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The application relates to the technical field of log acquisition, in particular to a log real-time collecting and processing method and device, wherein the method comprises the following steps: an abstract layer realized through the log is accessed into a log framework; the output source of the inheritance log component acquires appointed log data in real time; sorting log data, and pushing the sorted log data to a cache queue; when the set condition is reached, the log data in the buffer queue is sent to a search engine for storage, and a downloading interface is provided. According to the technical scheme, no additional service is required to be deployed independently, the occupation of server resources is reduced, and the maintenance is convenient. According to the technical scheme, the output source of the inheritance log component acquires the appointed log data, and the output source is consistent with the main stream log record mode in use, so that additional learning cost is not required. The log data is obtained in real time and downloaded in real time, and no delay exists.

Description

Method and device for collecting and processing logs in real time
Technical Field
The application relates to the technical field of log acquisition, in particular to a method and equipment for collecting and processing logs in real time.
Background
The log is used as an important carrier for solving and recording the problems in the production operation and maintenance process, and is simple to record, convenient to inquire and convenient to export.
The service of containerization, privatization and clustered deployment needs to meet the requirements of convenient use, targeted query and convenient export when checking logs. According to the existing log data collection technical scheme, corresponding services are required to be deployed independently, server resources are occupied, and maintenance is inconvenient; and only logs of a certain class need to be configured independently, so that the use cost is increased.
Disclosure of Invention
In order to overcome the technical scheme of log data collection in the related technology at least to a certain extent, corresponding services are required to be deployed independently, server resources are occupied, and maintenance is inconvenient; and only records the problem that a certain type of log needs to be configured independently, and the use cost is increased.
The scheme of the application is as follows:
according to a first aspect of an embodiment of the present application, there is provided a log real-time collecting and processing method, including:
an abstract layer realized through the log is accessed into a log framework;
the output source of the inheritance log component acquires appointed log data in real time;
sorting the log data, and pushing the sorted log data to a cache queue;
when the set condition is reached, the log data in the buffer queue is sent to a search engine for storage, and a downloading interface is provided.
Preferably, the method further comprises:
logging parameters are enhanced by MDC.
Preferably, the method further comprises:
and acquiring a configuration file from the starting class, loading the configuration file, and creating a log information structured template.
Preferably, the sorting the log data includes:
filtering the log data, and deleting useless data and irrelevant error reporting data in the log data;
and packaging the filtered log data into structured data based on the log information structured template.
Preferably, the setting conditions include:
reaching a preset execution thread timing and/or the buffer queue reaching a storage capacity threshold.
Preferably, the search engine provides a variety of query functions.
Preferably, the search engine stores log data in a classified manner.
Preferably, the search engine provides multiple download pathways and supports log data downloads in multiple data formats.
According to a second aspect of the embodiment of the present application, there is provided a log real-time collection processing apparatus, including:
a processor and a memory;
the processor is connected with the memory through a communication bus:
the processor is used for calling and executing the program stored in the memory;
the memory is used for storing a program, and the program is at least used for executing a log real-time collection processing method according to any one of the above.
The technical scheme provided by the application can comprise the following beneficial effects: the method for collecting and processing the logs in real time comprises the following steps: an abstract layer realized through the log is accessed into a log framework; the output source of the inheritance log component acquires appointed log data in real time; sorting log data, and pushing the sorted log data to a cache queue; when the set condition is reached, the log data in the buffer queue is sent to a search engine for storage, and a downloading interface is provided. According to the technical scheme, no additional service is required to be deployed independently, the occupation of server resources is reduced, and the maintenance is convenient. According to the technical scheme, the output source of the inheritance log component acquires the appointed log data, and the output source is consistent with the main stream log record mode in use, so that additional learning cost is not required. The log data is obtained in real time and downloaded in real time, and no delay exists.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
FIG. 1 is a flow chart of a log real-time collection processing method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a log real-time collecting and processing device according to an embodiment of the present application.
Reference numerals: a processor-21; and a memory 22.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
Example 1
Fig. 1 is a flow chart of a log real-time collecting and processing method according to an embodiment of the present application, referring to fig. 1, the log real-time collecting and processing method includes:
s11: an abstract layer realized through the log is accessed into a log framework;
s12: the output source of the inheritance log component acquires appointed log data in real time;
s13: sorting log data, and pushing the sorted log data to a cache queue;
s14: when the set condition is reached, the log data in the buffer queue is sent to a search engine for storage, and a downloading interface is provided.
It should be noted that, the technical solution in this embodiment relates to the technical field of log acquisition, and may be specifically applied to the requirement of service for containerization, privatization, clustered deployment on log data.
It should be noted that the log framework is divided into a log portal (abstraction layer) and a log implementation.
Log door facing: is the abstract layer of log implementation. The log implementation: implementation of specific log functions.
Because the log is implemented, there may be some code optimization and modification to avoid affecting the user's use in the project, using the unified interfaces of the log portal, the interfaces called by the user using the log in the project, etc. will not be affected, assuming the code is modified at the implementation level.
In specific practice, the abstract layer implemented by the log in this embodiment may be SLF4J. SLF4J, simple Logging Facade for Java, is not a true log implementation, but rather an abstraction layer, allowing any log implementation to be used in the background. This means that the log framework can be switched from one to another at run-time, only the jar packets need to be changed. SLF4J is mainly intended to provide a standard, canonical API framework for Java log access, and its main meaning is to provide an interface, and specific implementations may be passed through other log frameworks, such as log4J and log back, etc. For a general Java project, the log framework selects SLF4J as a portal, and a specific implementation framework (log 4J, log back, etc.) is matched, and bridging is completed by using a bridge in the middle.
Further, in this embodiment, the log record parameter is enhanced by MDC. The MDC, mapped Diagnostic Context, is a type of data element used to distinguish log output. It is a way to insert specific data into each log request so that each log request can have a specific context. For example, MDC may be used to insert a user ID so that each user's activity is seen in the log.
It should be noted that, the addition of log information is divided into two ways, one is the capture of automatic exception information, and the other is the custom addition in the program. The logs recorded in the two modes are recorded through a log component, and the technology acquires appointed log data in real time through inheriting the output source of the log component.
It can be appreciated that the present solution is simple to use, and one line of code can complete the recording as it is consistent with the manner of the main stream log recording.
The data acquired from the log component is non-visual data, so it needs to be consolidated into visual structured data. Based on this, the method further comprises:
and acquiring a configuration file from the starting class, loading the configuration file, and creating a log information structured template.
Further, the sorting of the log data includes:
filtering the log data, and deleting useless data and irrelevant error reporting data in the log data;
and packaging the filtered log data into structured data based on the log information structured template.
In this embodiment, according to the configuration, useless data and irrelevant error reporting data in the log data are deleted, and the filtered log data are packaged into structured data based on the log information structured template.
Specifically, the filtered log data is added with relevant information such as time, IP address, node name, problem position and the like, and is packaged into structured data.
It should be noted that, the log information structured template in this embodiment is flexible to configure, all parameters in the log information structured template can be modified in the configuration file, and the recorded structured log can be dynamically set.
It should be noted that, log data encapsulated into structured data is pushed into the cache queue.
The data in the cache queue can be acquired by setting the timing of the execution thread and/or the memory threshold of the cache queue when the timing of the execution thread and/or the memory threshold of the cache queue are reached, thereby preventing recording abnormality caused by data blocking,
the execution thread may send the data in batches to the search engine for storage.
It should be noted that the search engine provides various query functions.
The search engine stores the log data in a classified manner.
The search engine provides a variety of download approaches and supports log data downloads in a variety of data formats.
In particular practice, the search engine may be selected from the group consisting of an elastomer search.
In the elastic search, structured data can be queried. This is because the elastic search is not just a full text search engine, but it also provides many complex query functions such as aggregating queries and the like. This allows complex queries and analyses to be performed in the log to better understand the behavior of the application.
The search engine stores the log data in a classified mode, provides various downloading ways and supports log data downloading in various data formats. The user can obtain the corresponding log data according to the classified ID and the log type, and can select to download the log data of the common text type and the RTF format log data with the format.
According to the technical scheme, the access can be completed rapidly only by introducing the packed jar integration. This avoids the complexity of using ELK or the like. ELK is a combination of Elasticsearch, logstash and Kibana, which are responsible for searching and analysis, log collection and log visualization, respectively. Although ELK is powerful, its installation and configuration is relatively complex, requiring high learning costs. The technical scheme can realize a powerful log acquisition function only by simple integration.
It can be understood that the log real-time collecting and processing method in this embodiment includes: an abstract layer realized through the log is accessed into a log framework; the output source of the inheritance log component acquires appointed log data in real time; sorting log data, and pushing the sorted log data to a cache queue; when the set condition is reached, the log data in the buffer queue is sent to a search engine for storage, and a downloading interface is provided. According to the technical scheme, no additional service is required to be deployed independently, the occupation of server resources is reduced, and the maintenance is convenient. According to the technical scheme, the output source of the inheritance log component acquires the appointed log data, and the output source is consistent with the main stream log record mode in use, so that additional learning cost is not required. The log data is obtained in real time and downloaded in real time, and no delay exists.
Example two
Fig. 2 is a schematic structural diagram of a log real-time collecting and processing device according to an embodiment of the present application, and referring to fig. 2, the log real-time collecting and processing device includes:
a processor 21 and a memory 22;
the processor 21 is connected to the memory 22 via a communication bus:
wherein the processor 21 is used for calling and executing the program stored in the memory 22;
a memory 22 for storing a program for executing at least one log real-time collection processing method as in the above embodiment.
It is to be understood that the same or similar parts in the above embodiments may be referred to each other, and that in some embodiments, the same or similar parts in other embodiments may be referred to.
It should be noted that in the description of the present application, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "plurality" means at least two.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.

Claims (9)

1. The method for collecting and processing the logs in real time is characterized by comprising the following steps of:
an abstract layer realized through the log is accessed into a log framework;
the output source of the inheritance log component acquires appointed log data in real time;
sorting the log data, and pushing the sorted log data to a cache queue;
when the set condition is reached, the log data in the buffer queue is sent to a search engine for storage, and a downloading interface is provided.
2. The method according to claim 1, wherein the method further comprises:
logging parameters are enhanced by MDC.
3. The method according to claim 1, wherein the method further comprises:
and acquiring a configuration file from the starting class, loading the configuration file, and creating a log information structured template.
4. A method according to claim 3, wherein sorting the log data comprises:
filtering the log data, and deleting useless data and irrelevant error reporting data in the log data;
and packaging the filtered log data into structured data based on the log information structured template.
5. The method of claim 1, wherein the setting conditions comprise:
reaching a preset execution thread timing and/or the buffer queue reaching a storage capacity threshold.
6. The method of claim 1, wherein the search engine provides a plurality of query functions.
7. The method of claim 1, wherein the search engine stores log data in categories.
8. The method of claim 1, wherein the search engine provides multiple download pathways and supports log data downloads in multiple data formats.
9. A log real-time collection processing apparatus, characterized by comprising:
a processor and a memory;
the processor is connected with the memory through a communication bus:
the processor is used for calling and executing the program stored in the memory;
the memory is used for storing a program, and the program is at least used for executing a log real-time collection processing method according to any one of claims 1-8.
CN202310952297.0A 2023-08-01 2023-08-01 Method and device for collecting and processing logs in real time Pending CN116662218A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310952297.0A CN116662218A (en) 2023-08-01 2023-08-01 Method and device for collecting and processing logs in real time

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310952297.0A CN116662218A (en) 2023-08-01 2023-08-01 Method and device for collecting and processing logs in real time

Publications (1)

Publication Number Publication Date
CN116662218A true CN116662218A (en) 2023-08-29

Family

ID=87710105

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310952297.0A Pending CN116662218A (en) 2023-08-01 2023-08-01 Method and device for collecting and processing logs in real time

Country Status (1)

Country Link
CN (1) CN116662218A (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130031613A1 (en) * 2011-07-26 2013-01-31 Salesforce.Com, Inc. Secure access to customer log data in a multi-tenant environment
CN109033280A (en) * 2018-07-11 2018-12-18 平安科技(深圳)有限公司 Blog search method, system, computer equipment and storage medium
CN109542733A (en) * 2018-12-05 2019-03-29 焦点科技股份有限公司 A kind of highly reliable real-time logs collection and visual m odeling technique method
CN110765483A (en) * 2019-09-29 2020-02-07 北京淇瑀信息科技有限公司 Configured log desensitization method and device and electronic equipment
CN111104302A (en) * 2019-12-06 2020-05-05 紫光云(南京)数字技术有限公司 Improved ELK log analysis method for cluster system
CN112506743A (en) * 2020-12-09 2021-03-16 天津狮拓信息技术有限公司 Log monitoring method and device and server
CN115470090A (en) * 2022-09-27 2022-12-13 中邮消费金融有限公司 Log data acquisition method
CN115904876A (en) * 2022-12-15 2023-04-04 重庆紫光华山智安科技有限公司 Method, system, terminal and medium for solving conflict of log data in different formats
CN116340269A (en) * 2023-03-29 2023-06-27 中电云数智科技有限公司 Method for acquiring and searching Flink task logs in real time based on elastic search

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130031613A1 (en) * 2011-07-26 2013-01-31 Salesforce.Com, Inc. Secure access to customer log data in a multi-tenant environment
CN109033280A (en) * 2018-07-11 2018-12-18 平安科技(深圳)有限公司 Blog search method, system, computer equipment and storage medium
CN109542733A (en) * 2018-12-05 2019-03-29 焦点科技股份有限公司 A kind of highly reliable real-time logs collection and visual m odeling technique method
CN110765483A (en) * 2019-09-29 2020-02-07 北京淇瑀信息科技有限公司 Configured log desensitization method and device and electronic equipment
CN111104302A (en) * 2019-12-06 2020-05-05 紫光云(南京)数字技术有限公司 Improved ELK log analysis method for cluster system
CN112506743A (en) * 2020-12-09 2021-03-16 天津狮拓信息技术有限公司 Log monitoring method and device and server
CN115470090A (en) * 2022-09-27 2022-12-13 中邮消费金融有限公司 Log data acquisition method
CN115904876A (en) * 2022-12-15 2023-04-04 重庆紫光华山智安科技有限公司 Method, system, terminal and medium for solving conflict of log data in different formats
CN116340269A (en) * 2023-03-29 2023-06-27 中电云数智科技有限公司 Method for acquiring and searching Flink task logs in real time based on elastic search

Similar Documents

Publication Publication Date Title
US10503623B2 (en) Monitoring containerized applications
CN107145489B (en) Information statistics method and device for client application based on cloud platform
US10552852B1 (en) Service monitor for monitoring and tracking the performance of applications running on different mobile devices
CN103401698B (en) For the monitoring system that server health is reported to the police in server set group operatione
US6167358A (en) System and method for remotely monitoring a plurality of computer-based systems
CN110162345B (en) Application program access method and device and storage medium
US7721266B2 (en) Unified logging service with a logging formatter
CN110320891B (en) Railway vehicle braking system CAN bus message maintenance monitoring system and monitoring method
CN103188112B (en) Network flow detection method and device
US20070006154A1 (en) Controlling collection of debugging data
CN1248116C (en) General purpose testing arrangement for embedded module and subsystem based on host machine platform
US11675682B2 (en) Agent profiler to monitor activities and performance of software agents
WO2022016847A1 (en) Automatic test method and device applied to cloud platform
CN107404417A (en) A kind of processing method of monitoring data, processing unit and processing system
CN109582517A (en) System application automated testing method, device, system and tool storage end
CN111737207B (en) Method and device for showing and collecting logs of service nodes in distributed system
US20040233237A1 (en) Development environment for DSP
CN113391973B (en) Internet of things cloud container log collection method and device
CN111367873A (en) Log data storage method and device, terminal and computer storage medium
CN102752770B (en) Method and device for polling service system
CN113111107A (en) Data comprehensive access system and method
CN105187393B (en) A kind of mobile terminal from malicious software network behavior reconstructing method and its system
EP2577458B1 (en) Distillation and reconstruction of provisioning components
CN113746883B (en) Link tracking method and system
CN116662218A (en) Method and device for collecting and processing logs in real time

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination