CN115408344A

CN115408344A - Log formatting method and device, electronic equipment and storage medium

Info

Publication number: CN115408344A
Application number: CN202211196520.5A
Authority: CN
Inventors: 徐洪宽
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2022-09-29
Filing date: 2022-09-29
Publication date: 2022-11-29
Anticipated expiration: 2042-09-29
Also published as: CN115408344B

Abstract

The disclosure provides a log formatting method, a log formatting device, log formatting equipment and a log formatting storage medium, which can be applied to the technical field of computers. The method comprises the following steps: acquiring a log file related to the log address information according to the log address information configured in the configuration file, wherein the configuration file is also configured with an output format of the log file; determining a target formatting service with a format conversion result configured for the log address information as an output format according to the log address information and the output format; scheduling the log file to a log processor corresponding to the target formatting service by using the first scheduling service; and utilizing the log processor to format the log file.

Description

Log formatting method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a log formatting method and apparatus, an electronic device, and a storage medium.

Background

Network devices, systems, service programs, etc. typically generate a log of events, called logs, during operation. Various log files are designed in the operating system, such as application program logs, security logs, system logs, service logs, protocol logs, server logs and the like, and the various logs are different according to different services started by the system.

The development of multiple project groups and the use of multiple development languages in multiple projects under self-research results in the diversity of log formats, which is inconvenient for data collection, integration and mining analysis, and the log content before the project log is normalized is lost if the corresponding formatting process is not performed.

Disclosure of Invention

In view of the above, the present disclosure provides a log formatting method, apparatus, electronic device, and storage medium.

According to a first aspect of the present disclosure, there is provided a log formatting method, including: according to the log address information configured in the configuration file, acquiring a log file related to the log address information, wherein the configuration file is also configured with an output format of the log file; according to the log address information and the output format, determining that a format conversion result configured for the log address information is a target formatting service of the output format; scheduling the log file to a log processor corresponding to the target formatting service by using a first scheduling service; and utilizing the log processor to format the log file.

A second aspect of the present disclosure provides a log formatting device, including: the first acquisition module is used for acquiring a log file related to the log address information according to the log address information configured in a configuration file, wherein the configuration file is also configured with an output format of the log file; the determining module is used for determining a format conversion result configured aiming at the log address information as a target formatting service of the output format according to the log address information and the output format; the scheduling module is used for scheduling the log file to a log processor corresponding to the target formatting service by utilizing a first scheduling service; and the processing module is used for formatting the log file by using the log processor.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the log formatting method of the present disclosure.

The fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the log formatting method described in the present disclosure.

The fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the log formatting method of the present disclosure.

According to the log formatting method, the log formatting device, the electronic equipment and the storage medium, the target formatting service which takes the format conversion result configured aiming at the log address information as the output format is determined according to the log address information configured in the configuration file and the output format of the log file related to the log address information; scheduling the log file to a log processor corresponding to the target formatting service by using the first scheduling service; and utilizing the log processor to format the log file. Because the formatting treatment of the log file can be realized by combining the formatting service and the first scheduling service in a mode of configuring log address information, outputting format and formatting service, the log content in a fixed format can be formatted by the situation of the configuration file, the problems of data acquisition, integration and mining analysis which are inconvenient due to the diversity of the log formats are at least partially solved, the loss and abandonment of useful data in the log caused by missing problems are avoided by realizing the consistency of the log formats, and convenience is also brought to the later mining analysis.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be apparent from the following description of embodiments of the disclosure, which proceeds with reference to the accompanying drawings, in which:

fig. 1 schematically illustrates an application scenario diagram of a log formatting method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a log formatting method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a code design architecture diagram for implementing a log formatting method according to an embodiment of the present disclosure;

fig. 4 schematically shows a block diagram of a structure of a log formatting apparatus according to an embodiment of the present disclosure; and

fig. 5 schematically shows a block diagram of an electronic device adapted to implement a method of log formatting according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction should be interpreted in the sense one having ordinary skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B, a and C, B and C, and/or A, B, C, etc.).

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure, application and other processing of the related data (such as but not limited to personal information of the user) are all in accordance with the regulations of related laws and regulations, and necessary security measures are taken without violating the customs of the public order.

All kinds of cloud application have corresponding log service functions and can provide corresponding log collection clients. However, log collection clients provided by various cloud applications all have requirements on log formats, and if the logs are required to have a certain preset format, a separate client is not provided for log format arrangement. In addition, in the solution proposed in the market for log format arrangement, relatively more complicated components are required to be relied on when operating on a plurality of self-research projects, and inconvenience is brought to log collection, arrangement, mining and analysis in the later period.

The embodiment of the disclosure provides a log formatting method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a log file related to the log address information according to the log address information configured in the configuration file, wherein the configuration file is also configured with an output format of the log file; determining a target formatting service for the format conversion result configured aiming at the log address information into an output format according to the log address information and the output format; scheduling the log file to a log processor corresponding to the target formatting service by using the first scheduling service; and utilizing the log processor to format the log file.

Fig. 1 schematically shows an application scenario diagram of a log formatting method according to an embodiment of the present disclosure.

As shown in fig. 1, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium of communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may interact with the server 105 via the network 104 using at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages or the like. Various communication client applications, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, etc. (for example only), may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop portable computer, a desktop computer, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the log formatting method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the log formatting device provided by the embodiment of the present disclosure may be generally disposed in the server 105. The log formatting method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and/or the server 105. Accordingly, the log formatting device provided in the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for an implementation.

The log formatting method of the disclosed embodiment will be described in detail below with fig. 2 to 3 based on the scenario described in fig. 1.

Fig. 2 schematically shows a flow chart of a log formatting method according to an embodiment of the present disclosure.

As shown in fig. 2, the log formatting method of this embodiment includes operations S210 to S240.

In operation S210, a log file associated with the log address information is obtained according to the log address information configured in the configuration file, and an output format of the log file is also configured in the configuration file.

According to an embodiment of the present disclosure, the configuration file may include an xml (Extensible Markup Language) file. The configuration file may be configured with log address information of the log file to be processed and configuration information such as an output format of the log file. The configuration information in the configuration file can be configured by a user according to actual use requirements in a customized mode. Or under the condition of configuring the initial configuration information, the user performs operations such as adding, deleting, modifying, updating and the like on the initial configuration information by combining with the actual use requirement to obtain the configuration file with the target configuration information. The configuration information in the configuration file can provide reference for the subsequent processes of obtaining the log file and formatting the log file.

According to an embodiment of the present disclosure, the log address information may include at least one of: the first path information representing a storage location of the offline log, the second path information representing a generation location of the real-time log, the port information representing a network transmission channel of the real-time log, and the like, and may not be limited thereto. The log address information may be used to provide a reference path or a reference port for the process of retrieving the log file. For example, by accessing or monitoring the path or port, an offline or real-time log file may be obtained.

According to an embodiment of the present disclosure, the output format may include at least one of: JSON (JS object Notation, a lightweight data exchange format), regularizable format, and the like, and may be not limited thereto. The output format can be used for providing reference for processing results of formatting the processing log file. For example, in a case where the output format is configured as JSON, it may be determined that the result of the formatting process on the log file is a JSON-formatted file, so that an appropriate formatting service can be selected accordingly.

According to an embodiment of the present disclosure, log address information and output formats may be modularly configurable. For example, JSON may be configured for the first log address information, JSON and a regularizable format may be configured for the second log address information, and so on. The output format may also exist as global configuration information in a configuration file. For example, the output format may be configured as JSON independently of all log address information, serving as a normalized format of the formatting processing result of the log file corresponding to all log address information.

It should be noted that a variety of output formats can be designed for a plurality of items. Accordingly, a plurality of output formats and formatting services may be modularly configured for each project in the configuration file, or may be globally configured for a plurality of projects in the configuration file.

In operation S220, a target formatting service in which a format conversion result configured for the log address information is an output format is determined according to the log address information and the output format.

According to an embodiment of the present disclosure, a formatting service may also be configured in a configuration file, according to the configured output format in the configuration file. The formatting service may be characterized as one or more pieces of program code or execution paths thereof, each piece of program code being operable to implement the conversion of the original format of the log file into the configured output format.

According to embodiments of the present disclosure, the log address information and formatting service may be configured modularly. For example, a formatting service for converting a log file into a JSON file may be configured for the first log address information, a formatting service for converting a log file into a regularizable format file may be configured for the second log address information, and the like. The formatting service may also exist in the configuration file as global configuration information. For example, the formatting service may be configured independently of all log address information, and may include configuring, on a global scale, a formatting service that converts the log file into at least one of a JSON format file and a regularizable format file, and the like, and using the at least one of the JSON format file and the regularizable format file as the formatting service corresponding to all log address information.

According to an embodiment of the present disclosure, in a case where a corresponding log file is obtained according to log address information configured in a configuration file, in a case where the log address information, an output format, and formatting services configured corresponding to a category of the output format are modularly configured, an output format required for the log file may be first determined according to the modularly configured output format, and then a target formatting service may be determined from the modularly configured formatting services according to the output format required for the log file. In the case that the log address information and the output format are in the modular configuration and the formatting service is in the global configuration, the output format required by the log file may be determined according to the output format of the modular configuration, and then the target formatting service may be determined from the formatting service of the global configuration in combination with the required output format.

It should be noted that the manner of determining the target formatting service may not be limited to the above, and may have a corresponding determination manner according to the difference between the log address information, the output format, and the configuration manner of the formatting service, such as the difference between the modular configuration and the global configuration, as long as an appropriate formatting service can be selected.

In operation S230, the log file is scheduled to a log processor corresponding to the target formatting service by using the first scheduling service.

According to an embodiment of the present disclosure, the first scheduling service may be characterized as a code program or a script tool that acquires a log file and transmits the log file to a processing party, and may also be characterized as a code program or a script tool that transmits the acquired log file to a processing party. The log processor may be a software class processor, which may include the target formatting service itself, for example. The log processor may also be a hardware tool, such as a hardware circuit board that may include an FPGA (field programmable gate array) embedded with target formatting service functionality and other writeable programs, and so on.

In operation S240, the log file is formatted using the log processor.

According to the embodiment of the disclosure, after receiving the log file, the log processor can execute the operation process, so as to realize the formatting process of the log file.

Through the embodiment of the disclosure, the formatting treatment of the log file can be realized by combining the formatting service and the first scheduling service in a mode of configuring log address information, outputting a format and the formatting service, the log content in a fixed format can be formatted according to the situation of the configuration file, the problems of inconvenience for data acquisition, integration and mining analysis caused by the diversity of the log formats are at least partially solved, the loss and abandonment of useful data in the log due to omission are avoided by realizing the consistency of the log formats, and convenience is brought to the later mining analysis.

The method shown in fig. 2 is further described below with reference to specific embodiments.

According to an embodiment of the present disclosure, in the case that the log address information includes first path information representing a storage location of the offline log, the operation S210 may include: and running the binary execution file to obtain an offline log file related to the log address information. The binary execution file comprises log address information.

According to an embodiment of the present disclosure, the entire process of the log formatting process may be implemented in the form of a binary execution file. The binary execution file may be characterized in the form of a gadget that can be used to process offline log files. The binary execution file may store a storage path of the offline log file, that is, the first path information. By operating the binary execution file, the first path information can be accessed, and the log file stored in the path can be obtained, namely the off-line log file can be obtained.

According to an embodiment of the present disclosure, in the case that the log address information includes second path information representing a generation location of the real-time log, the operation S210 may further include: and responding to the fact that the content of the log file corresponding to the second path information is detected to be changed based on the first daemon process, and obtaining the changed log file to obtain a first real-time log file.

According to the embodiment of the disclosure, the whole process of the log formatting process can also be realized in the form of a daemon process. The first daemon process may be characterized as a resident running service deployed in the computer system and operable to process log files generated in real time. The first daemon process can detect information change under a path corresponding to the second path information in real time, and can acquire changed information in real time under the condition that the information under the path is changed, so that a first real-time log file can be obtained.

According to an embodiment of the present disclosure, in the case that the log address information includes port information of a network transmission channel representing a real-time log, the operation S210 may further include: and responding to the fact that the port corresponding to the port information receives new log information based on the second daemon process, obtaining the new log information, and obtaining a second real-time log file.

According to an embodiment of the present disclosure, the second daemon process and the first daemon process may be the same daemon process. For example, the daemon process may detect, in real time, information change in a path corresponding to the second path information and information change transmitted by a port corresponding to the port information at the same time. The second daemon process and the first daemon process can also be two independent daemon processes. For example, the first daemon process may detect the information change under the path corresponding to the second path information in real time, and the second daemon process may detect the information change transmitted by the port corresponding to the port information in real time.

According to an embodiment of the present disclosure, the second daemon process may be used to process log files received over the network. The second daemon process can acquire the changed information in real time under the condition that the information transmitted by the port corresponding to the port information is changed, and then a second real-time log file can be obtained.

According to the embodiment of the disclosure, the configuration file may further configure an execution mode, and the configuration content for the execution mode may include at least one execution mode of a binary execution file, a daemon process, and the like. For example, the execution mode of the binary execution file may be configured modularly for the log address information representing the offline log file, and the execution mode of the daemon process may be configured modularly for the log address information representing the real-time log file. The execution mode can be configured to determine the execution carrier of each log formatting process.

It should be noted that the binary execution file and the daemon perform two execution modes as log formatting, and the two execution modes can exist and run in parallel independently and without mutual interference. When the logs corresponding to the log address information configured in the configuration file are all offline log files, the execution mode of the binary execution file can also be configured in the configuration file in a global configuration mode. When the logs corresponding to the log address information configured in the configuration file are all real-time log files, the execution mode of the daemon process can be configured in the configuration file in a global configuration mode.

Through the embodiment of the disclosure, the contents of the offline log file and the real-time log file transmitted online can be run and processed in a configurable mode in combination with the forms of service deployment and gadgets, formatting processing is realized, loss and abandonment of data are avoided, and convenience is brought to later mining analysis.

According to an embodiment of the present disclosure, the operation S220 may include: acquiring a plurality of formatting services configured in a configuration file and at least one output format configured for log address information, wherein the plurality of formatting services are configured in a serialized manner, and each formatting service is used for converting an initial format into one of the at least one output format; and determining a formatting service for converting the target initial format into any one of the at least one output format from the plurality of formatting services as the target formatting service according to the target initial format and the at least one output format of the log file.

According to an embodiment of the present disclosure, at least one formatting service may be configured corresponding to each output format. The initial format may include at least one of a format of the log file and another format different from the output format, and the like. The plurality of formatting services configured in a serialization manner can realize the conversion of files in various initial formats into files corresponding to output formats, and the target formatting services can be conveniently selected by the formatting processing process.

According to the embodiment of the disclosure, in the case of obtaining a corresponding log file according to the log address information configured in the configuration file, the output format that the log file needs to satisfy may be determined in combination with the output format corresponding to the modular configuration or global configuration of the log address information in the configuration file. Further, in combination with the target initial format of the log file, a target formatting service capable of converting the target initial format into an output format that the log file needs to satisfy can be determined.

According to the embodiment of the disclosure, the configuration file can also be configured with log level information of the log file to be filtered.

According to an embodiment of the present disclosure, the log level information of the log file that needs to be filtered may include at least one of the following: debug (debug information), info (output information), war (warning information), error (error information), and the like, and may not be limited thereto.

According to the embodiment of the disclosure, the log address information and the log level information of the log file to be filtered can be configured in a modularization mode. For example, debug, info may be configured for the first log address information, and war, error, etc. may be configured for the second log address information. The log level information of the log file that needs to be filtered out may also be present in the configuration file as global configuration information. For example, the log level information of the log file to be filtered may be configured as debug, info, war and error independently of all the log address information, and used as the log level information of the log file to be filtered corresponding to all the log address information.

Through the embodiment of the disclosure, the journal is selected and processed in a configurable mode, so that more useful journal files can be extracted, the processing process is simplified, and more useful journal information is obtained.

According to an embodiment of the present disclosure, output stream information may also be configured in the configuration file, and the log formatting method may further include: acquiring a formatted file obtained after formatting the log file; acquiring at least one output stream information configured aiming at the log address information in a configuration file; and distributing the formatted file to at least one output stream corresponding to the at least one output stream information by utilizing the second scheduling service.

According to an embodiment of the present disclosure, the output stream information may be used to define the output manner of the formatted file. The configuration content of the output flow information may include at least one of, for example, and may not be limited to, a floor file, a docking kafka (a distributed message queue based on a publish/subscribe scheme), and other manners that can be developed secondarily. For example, where the output stream information is configured as a floor file, the formatted file may be output in the manner of a floor file. In the case where the output stream information is configured to interface with kafka, the formatted file may be output to kafka and subsequent processing procedures may be performed based on kafka.

According to embodiments of the present disclosure, log address information and output stream information may be configured modularly. For example, a floor file may be configured for a first log address information as an output stream, a docking kafka may be configured for a second log address information as an output stream, and so on. The output stream information may also exist in the configuration file as global configuration information. For example, the output stream information may be configured to interface kafka independently of all log address information, and used as an output mode of a formatted file obtained by log file formatting processing corresponding to all log address information.

According to an embodiment of the present disclosure, the second scheduling service may be characterized as a code program or a scripting tool that acquires the formatted file and sends the formatted file to the output stream, and may also be characterized as a code program or a scripting tool that sends the acquired formatted file to the output stream.

According to the embodiment of the disclosure, after the log file is formatted based on the target formatting service to obtain the formatted file, the output stream of the formatted file to be transmitted to may be determined according to the output stream information configured in the configuration file for the log address information corresponding to the log file. The formatted file may then be distributed to the output stream using a second scheduling service.

According to an embodiment of the present disclosure, after the formatted file is distributed to the output stream to be scheduled, the log formatting method may further include: and outputting the formatted file in an output mode corresponding to at least one output stream by utilizing the output service.

According to an embodiment of the present disclosure, the output service may be characterized as a code program or a script tool that acquires the formatted file and outputs the formatting in the form of an output stream, and may also be characterized as a code program or a script tool that outputs the acquired formatted file in the form of an output stream.

According to an embodiment of the present disclosure, after the formatted file is distributed to the corresponding output stream using the second scheduling service, the formatted file may be output to the output stream using an output service, such as the formatted file may be output as a floor file or input into kafka.

Through the embodiment of the disclosure, the second scheduling service and the output service are combined, the formatted file can be automatically output to a plurality of output streams such as file streams, network streams and the like based on the output stream information configured in the configuration file in advance, the formatted file can be saved in real time, the subsequent data mining and analysis work can be conveniently carried out, and the workload of research and development personnel can be reduced.

According to an embodiment of the present disclosure, before invoking the scheduling service for scheduling the log file and for distributing the formatted file, the formatting service for formatting the log file, and the output service for outputting the formatted file, an initialization process may be further included. The initialization process may include: and initializing the scheduling service according to the log address information and the output stream information configured in the configuration file. And initializing the formatted service according to the output format configured in the configuration file. And initializing the output service according to the output stream information configured in the configuration file.

According to the embodiment of the disclosure, according to the actual project requirement, information such as log address information, output stream information, output format and the like which are adapted to the project can be firstly configured in the configuration file. Then, the configuration file can be analyzed, and the configuration information in the configuration file can be read. After that, the above initialization process may be performed on the relevant parameters of the scheduling service, the formatting service, and the output service according to the configuration information in the configuration file, so that the log formatting method may be run according to the information configured in the configuration file to complete the log formatting process for the corresponding project when the log formatting process needs to be performed on the project log.

According to an embodiment of the present disclosure, the execution modes configured in the configuration file, such as the execution modes of the binary execution file, the resident process, and the like, may have corresponding command line configurations. In the initialization process, the method may further include: and analyzing the command line parameters, and judging whether the log formatting program to be run is a resident process type or is executed in a small tool (namely a binary execution file) form, so that the log formatting process is executed in a corresponding execution mode under the condition of starting the formatting program.

According to an embodiment of the present disclosure, in the initialization process, the method may further include: the setup log formats the identification output records. Through the steps, failure information can be output under the condition that log formatting fails, and research and development personnel can conveniently carry out operation and maintenance processing and the like.

It should be noted that, when new configuration information is reloaded in the configuration file, the log formatting program may be reloaded in an initialization manner, so as to implement a configurable and diversified log formatting processing method.

Through the embodiment of the disclosure, the corresponding log formatting related information can be configured for various projects in a configurable mode based on the configuration file, and the initialization process is combined, so that the method for supporting formatting processing of the log files of various projects can be realized, the workload of research and development personnel on log processing can be reduced, the loss of useful information in the log can be reduced in the formatting process, and the subsequent mining analysis work can be facilitated.

FIG. 3 schematically shows a code design architecture diagram for implementing a log formatting method according to an embodiment of the present disclosure.

As shown in fig. 3, the whole log formatting method can be divided into three functions, including: initialization, loader processing, reloading and the like.

According to an embodiment of the present disclosure, the initialization process may include: by parsing the command line parameters, it is determined whether the log formatter is resident in the process type or is executing in the form of a gadget. By parsing the configuration file, schedule service (Dispatcher), format service (formatters) and output service (output) are initialized, and in the format service (formatters), the formatters can be initialized according to a plurality of format tags. Setting a log formatting failure output record.

According to an embodiment of the present disclosure, the logger kernel function may be used to implement a logger processing procedure. logger processing is a core function implementation of the log formatter and may generally include initialized Dispatcher, formatters, and Outputer. The Dispatcher can be used for dispatching the log file to different log processor formatters according to the log source, and distributing the formatted file obtained by processing to different log output processors according to the configuration of the output stream information of the log. The formatters can master the formatting requirements of the format label in the serialization configuration file, and can set a plurality of Formaters according to the configuration information of the format label. The formatter may process the received log file and perform formatting operations thereon. The outputter may output the formatted file to a plurality of output streams, for example, may output the formatted file to a file stream, for example, output the formatted file as a floor file, and output the formatted file to a network stream, for example, output the formatted file to access kafka, and may also output the formatted file in other secondary development manners.

It should be noted that the initialization process and the processing processes of various services such as dispatch, formats, output, etc. have been described in the foregoing embodiments, and are not described herein again.

According to an embodiment of the present disclosure, the reloading process may include: reloading new configuration information in the configuration file, reloading the log formatter by way of initialization, and the like, but may not be limited thereto.

By the embodiment of the disclosure, a complete set of log data formatting processing scheme can be constructed, and the log files are run and processed through the configurable configuration files and in combination with the form of service deployment or small tools, so that the loss and abandonment of useful data in logs are reduced, the workload of research and development personnel on log processing is reduced, and convenience is brought to later-stage mining and analysis.

Based on the log formatting method, the disclosure also provides a log formatting device. The apparatus will be described in detail below with reference to fig. 4.

Fig. 4 schematically shows a block diagram of a structure of a log formatting apparatus according to an embodiment of the present disclosure.

As shown in fig. 4, the log formatting apparatus 400 of this embodiment includes a first obtaining module 410, a determining module 420, a scheduling module 430, and a processing module 440.

A first obtaining module 410, configured to obtain, according to log address information configured in a configuration file, a log file related to the log address information, where an output format of the log file is also configured in the configuration file. In an embodiment, the first obtaining module 410 may be configured to perform the operation S210 described above, which is not described herein again.

A determining module 420, configured to determine, according to the log address information and the output format, that a format conversion result configured for the log address information is a target formatting service in the output format. In an embodiment, the determining module 420 may be configured to perform the operation S220 described above, which is not described herein again.

The scheduling module 430 is configured to schedule the log file to a log processor corresponding to the target formatting service by using a first scheduling service. In an embodiment, the scheduling module 430 may be configured to perform the operation S230 described above, which is not described herein again.

The processing module 440 is configured to perform formatting processing on the log file by using the log processor. In an embodiment, the processing module 440 may be configured to perform the operation S240 described above, which is not described herein again.

According to an embodiment of the present disclosure, the log address information includes first path information characterizing a storage location of the offline log. The first acquisition module comprises a first acquisition unit.

The first obtaining unit is used for operating a binary execution file and obtaining an offline log file related to the log address information, wherein the binary execution file comprises the log address information.

According to the embodiment of the disclosure, the log address information includes second path information representing a generation position of the real-time log, and the first obtaining module includes a second obtaining unit.

And the second acquisition unit is used for acquiring the changed log file to obtain the first real-time log file in response to the fact that the content of the log file corresponding to the second path information is detected to be changed based on the first daemon.

According to an embodiment of the present disclosure, the log address information includes port information of a network transmission channel representing a real-time log, and the first obtaining module includes a third obtaining unit.

And the third acquisition unit is used for responding to the situation that a port corresponding to the port information receives new log information detected based on the second daemon process, acquiring the new log information and obtaining a second real-time log file.

According to an embodiment of the present disclosure, the determining module includes a fourth obtaining unit and a determining unit.

And a fourth obtaining unit, configured to obtain a plurality of formatting services configured in the configuration file and at least one output format configured for the log address information, where the plurality of formatting services are configured in a serialized manner, and each formatting service is configured to implement conversion from an initial format to one of the at least one output format.

A determining unit, configured to determine, according to a target initial format of the log file and the at least one output format, a formatting service for converting the target initial format into any one of the at least one output format from the plurality of formatting services, as the target formatting service.

According to the embodiment of the disclosure, the log formatting device further comprises a second obtaining module, a third obtaining module and a distributing module.

And the second acquisition module is used for acquiring a formatted file obtained after the log file is formatted.

And the third obtaining module is used for obtaining at least one output stream information configured in the configuration file aiming at the log address information.

And the distribution module is used for distributing the formatted file to at least one output stream corresponding to the at least one output stream information by using a second scheduling service.

According to an embodiment of the present disclosure, the log formatting apparatus further includes an output module.

And the output module is used for outputting the formatted file in an output mode corresponding to the at least one output stream by using output service.

According to an embodiment of the present disclosure, the log formatting device further includes a first initialization module, a second initialization module, and a third initialization module.

And the first initialization module is used for initializing the formatting service for formatting the log file according to the output format configured in the configuration file.

And the second initialization module is used for initializing scheduling services for scheduling the log file and distributing formatted files according to the log address information and the output stream information configured in the configuration file, wherein the formatted files comprise files obtained by formatting the log file.

And the third initialization module is used for initializing the output service for outputting the formatted file according to the output stream information configured in the configuration file.

According to the embodiment of the disclosure, the configuration file is also configured with log level information of the log file to be filtered.

According to an embodiment of the present disclosure, any plurality of the first obtaining module 410, the determining module 420, the scheduling module 430, and the processing module 440 may be combined into one module to be implemented, or any one of them may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first obtaining module 410, the determining module 420, the scheduling module 430 and the processing module 440 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of three implementations of software, hardware and firmware, or in any suitable combination of any of them. Alternatively, at least one of the first obtaining module 410, the determining module 420, the scheduling module 430 and the processing module 440 may be at least partially implemented as a computer program module, which when executed may perform the respective functions.

Fig. 5 schematically illustrates a block diagram of an electronic device adapted to implement a method of log formatting according to an embodiment of the present disclosure.

As shown in fig. 5, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. The processor 501 may include, for example, a general purpose microprocessor (e.g., CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., application Specific Integrated Circuit (ASIC)), among others. The processor 501 may also include onboard memory for caching purposes. Processor 501 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the disclosure.

In the RAM 503, various programs and data necessary for the operation of the electronic apparatus 500 are stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other by a bus 504. The processor 501 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 502 and/or the RAM 503. Note that the programs may also be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, electronic device 500 may also include an input/output (I/O) interface 505, input/output (I/O) interface 505 also being connected to bus 504. The electronic device 500 may also include one or more of the following components connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The above-mentioned computer-readable storage medium carries one or more programs which, when executed, implement a log formatting method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include ROM 502 and/or RAM 503 and/or one or more memories other than ROM 502 and RAM 503 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the item recommendation method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 501. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, downloaded and installed through the communication section 509, and/or installed from the removable medium 511. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program, when executed by the processor 501, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of log formatting, comprising:

acquiring a log file related to the log address information according to the log address information configured in a configuration file, wherein the configuration file is also configured with an output format of the log file;

according to the log address information and the output format, determining that a format conversion result configured aiming at the log address information is a target formatting service of the output format;

scheduling the log file to a log processor corresponding to the target formatting service by using a first scheduling service; and

and utilizing the log processor to format the log file.

2. The method of claim 1, wherein the log address information includes first path information characterizing a storage location of an offline log, and the obtaining a log file associated with the log address information according to the log address information configured in a configuration file includes:

and running a binary execution file to obtain an offline log file related to the log address information, wherein the binary execution file comprises the log address information.

3. The method of claim 1, wherein the log address information includes second path information characterizing a generation location of the real-time log, and the obtaining a log file related to the log address information according to the log address information configured in the configuration file includes:

and responding to the fact that the content of the log file corresponding to the second path information is detected to be changed based on the first daemon process, and obtaining the changed log file to obtain a first real-time log file.

4. The method according to any one of claims 1 to 3, wherein the log address information includes port information of a network transmission channel characterizing a real-time log, and the obtaining a log file related to the log address information according to the log address information configured in a configuration file includes:

and responding to the fact that a port corresponding to the port information receives new log information based on the second daemon process, acquiring the new log information, and obtaining a second real-time log file.

5. The method of claim 1, wherein determining, according to the log address information and the output format, that the format conversion result configured for the log address information is a target formatting service of the output format comprises:

acquiring a plurality of formatting services configured in the configuration file and at least one output format configured for the log address information, wherein the plurality of formatting services are configured in a serialized manner, and each formatting service is used for converting an initial format into one of the at least one output format; and

and according to the target initial format and the at least one output format of the log file, determining a formatting service for converting the target initial format into any one of the at least one output format from the plurality of formatting services as the target formatting service.

6. The method of claim 1, further comprising:

obtaining a formatted file obtained after formatting the log file;

acquiring at least one output stream information configured for the log address information in the configuration file; and

and distributing the formatted file to at least one output stream corresponding to the at least one output stream information by utilizing a second scheduling service.

7. The method of claim 6, further comprising:

and outputting the formatted file in an output mode corresponding to the at least one output stream by using an output service.

8. The method of any of claims 1-3, further comprising:

initializing formatting service for formatting the log file according to the output format configured in the configuration file;

initializing scheduling services for scheduling the log files and distributing formatted files according to the log address information and the output stream information configured in the configuration files, wherein the formatted files comprise files obtained after the log files are formatted; and

and initializing output service for outputting the formatted file according to the output stream information configured in the configuration file.

9. The method according to any one of claims 1-3, wherein the configuration file is further configured with log level information of log files that need to be filtered out.

10. A log formatting apparatus comprising:

the first acquisition module is used for acquiring a log file related to the log address information according to the log address information configured in a configuration file, wherein the configuration file is also configured with an output format of the log file;

the determining module is used for determining a format conversion result configured aiming at the log address information as a target formatting service of the output format according to the log address information and the output format;

the scheduling module is used for scheduling the log file to a log processor corresponding to the target formatting service by utilizing a first scheduling service; and

and the processing module is used for carrying out formatting processing on the log file by utilizing the log processor.

11. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-9.

12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.