CN113434473A - Distributed log processing method, device, medium and electronic equipment - Google Patents
Distributed log processing method, device, medium and electronic equipment Download PDFInfo
- Publication number
- CN113434473A CN113434473A CN202110722036.0A CN202110722036A CN113434473A CN 113434473 A CN113434473 A CN 113434473A CN 202110722036 A CN202110722036 A CN 202110722036A CN 113434473 A CN113434473 A CN 113434473A
- Authority
- CN
- China
- Prior art keywords
- log file
- log
- level
- output
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 94
- 238000012545 processing Methods 0.000 claims abstract description 53
- 238000012216 screening Methods 0.000 claims abstract description 16
- 238000007639 printing Methods 0.000 claims abstract description 10
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 7
- 239000003795 chemical substances by application Substances 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 4
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 9
- 238000007726 management method Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 235000019580 granularity Nutrition 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application discloses a distributed log processing method, a distributed log processing device, a distributed log processing medium and electronic equipment. Wherein, the method comprises the following steps: carrying out distributed log information acquisition on a target method through a preset cut-in point, and writing the acquired log information into a log file; determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level; determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the printing screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises a print log file and a non-print log file. According to the technical scheme, the problem of high resource consumption in the log processing process can be solved, efficient log management is achieved, and the performance of a distributed system is improved.
Description
Technical Field
The embodiment of the application relates to the technical field of internet, in particular to a distributed log processing method, a distributed log processing device, a distributed log processing medium and electronic equipment.
Background
With the rapid development of internet technology and the continuous expansion of the number of users, event records, i.e., logs, are generated at all times during the operation of network devices, systems, service programs, and the like. Each row of the log records the description of the date, time, user and action.
At present, log collection and storage are managed in a centralized mode, but with the continuous increase of business volume, the scale of log data increases greatly, and a task of how to store massive log data and optimize the query efficiency of logs is urgent.
The log data volume is large, the distribution is dispersed, the management difficulty is large, and the existing log management method can not realize the efficient management of the log. In the operation processes of acquisition, storage, query and the like, a large amount of calculation power is usually consumed, and a large amount of storage space is occupied.
Disclosure of Invention
The embodiment of the application provides a processing method, a processing device, a processing medium and electronic equipment of a distributed log, which can perform distributed log information acquisition on a target method through a preset cut-in point to form a log file, and further management of the log file is realized according to the type of the log file. Therefore, efficient log management is realized, and the operation cost is saved while the performance of the distributed system is improved.
In a first aspect, an embodiment of the present application provides a method for processing a distributed log, where the method includes:
carrying out distributed log information acquisition on a target method through a preset cut-in point, and writing the acquired log information into a log file;
determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the printing screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises a print log file and a non-print log file.
In a second aspect, an embodiment of the present application provides an apparatus for processing a distributed log, where the apparatus includes:
the log information writing module is used for carrying out distributed log information acquisition on a target method through a preset cut-in point and writing the acquired log information into a log file;
the log file belonging level determining module is used for determining the synchronization rule of the log file according to the belonging type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
the log file processing result determining module is used for determining the processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a processing method of a distributed log according to an embodiment of the present application.
In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable by the processor, where the processor implements a processing method of a distributed log according to an embodiment of the present application when executing the computer program.
According to the technical scheme provided by the embodiment of the application, distributed log information acquisition is carried out on a target method through a preset cut-in point to form a log file; then, determining a synchronization rule of the log file according to the type of the log file, and simultaneously determining the belonging level of the log file according to the configuration information; and finally, determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the printing screening condition of the preset log file and the belonged level of the log file. The technical scheme provided by the application can realize efficient log management, and can save calculation power and storage space while improving the performance of the distributed system.
Drawings
Fig. 1A is a flowchart of a processing method of a distributed log according to an embodiment of the present application;
fig. 1B is a schematic diagram of an overall structure of collecting and outputting a distributed log file according to an embodiment of the present application;
FIG. 1C is a schematic diagram illustrating a section-oriented programming method according to an embodiment of the present disclosure;
fig. 2 is a flowchart of a processing method of a distributed log according to a second embodiment of the present invention;
fig. 3 is a block diagram of a distributed log processing apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1A is a flowchart of a processing method for a distributed log according to an embodiment of the present application, where the present embodiment is applicable to any distributed log management scenario, and the method may be executed by a processing apparatus for a distributed log according to an embodiment of the present application, where the apparatus may be implemented by software and/or hardware, and may be integrated in an electronic device.
As shown in fig. 1A, the processing method of the distributed log includes:
and S110, carrying out distributed log information acquisition on the target method through a preset cut-in point, and writing the acquired log information into a log file.
The scheme is performed by a platform that can generate logs, which can include one or more network devices, systems, and service programs, among others. The distributed storage mode may be a distributed storage mode, and the distributed storage mode is a data storage technology, and uses the disk space on each machine in the main body through a network, and forms the distributed storage resources into a virtual storage device, and the data is dispersedly stored in each corner of the main body. The distributed log information collection may be collection of log information stored in a distributed manner. Fig. 1B is a schematic diagram of an overall structure of collecting and outputting a distributed log file according to an embodiment of the present application. As shown in fig. 1B, the functions are mainly divided into collection and transmission of logs, log preprocessing, storage and indexing, log level, log function and log configuration, and the function of each functional module is as shown in fig. 1B.
The target method may be a program for executing a certain task, such as a deposit information recording program, or may be a function, such as a character string matching function. The target method can be run on the host machine, can also be run in a Docker container, and if the target method is run in the Docker container, the target method can be mapped onto the host machine through some technical means. Before distributed log information acquisition is carried out, the platform needs to preset access points, the access points can be the characteristics of different dimensionalities of the log information, and the system can set one or more access points according to the actual scene needs. For example, based on a profile-oriented programming approach, as shown in FIG. 1C, two entry points are defined, which may be an abnormal entry and an in-out entry for the target method. And configuring a target method needing to be tracked in the entry point, and configuring special information on the target method through annotation, wherein the special information comprises additional special log information and information for marking whether the parameter content is sensitive or not. Through the arrangement, transparent and non-sensory performance output of each method of the application system can be provided for users, and the users can be platform maintenance personnel, such as personnel for managing and operating and maintaining the platform based on log information.
After performing distributed log information collection, the platform may write collected log information into a log file, where the log information may include log ID, time, operation form, operation device information, and the like. The log file may be written directly into all log information, or may be written after the log information is screened under a certain condition.
In this scheme, optionally, if the target method runs in a Docker container, mapping the log file collected in the Docker container to a host;
and if the log file meets the storage condition of the database, forwarding the log file on the host machine to a machine data engine server through a machine data engine agent so as to store the log file in the database.
It is understood that the log distributed storage may include file storage and database storage. If the target method runs in the Docker container, the platform can map the log collected in the Docker container to the host. If the log file meets the storage condition of the database, the platform can forward the log file on the host machine to a machine data engine server through a machine data engine agent. The machine data engine proxy may be equivalent to an Agent in the log transmission structure shown in fig. 1B, and the machine data engine server may be equivalent to a SplunkServer in the log analysis infrastructure shown in fig. 1B, that is, the host may forward the log file to the SplunkServer through the Agent. In addition, the log file can be stored in a database for subsequent analysis according to the functional requirements of the platform.
S120, determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level.
The types of the log files can be divided according to actual scene requirements, for example, the types of the log files can be divided into normal logs, error logs, performance logs and the like, and in addition, different naming specifications can be formulated for different types of log files; according to the type of the log file, the platform can determine the synchronization rule of the log file, wherein the synchronization rule refers to a log output mode.
The configuration information may be information storing a configuration mode of the log file, for example, the log configuration file shown in fig. 1B, and the output of the log file may be controlled by a configuration mode such as an xml file or an yml file. In order to reduce the reading and writing pressure of the system and control the output magnitude of log information, the affiliated level of the log file can be optimally set, and the affiliated level comprises a program running level and a method performance level.
For example, the affiliated level of the log file can be divided into six levels of debug, info, war, error, must and perf, wherein the program running level can be a must level, the custom must level is used for forcibly outputting necessary information by a platform when a program is started and exited and a key module enters and exits, the necessary information can be used for determining the log file range of a transaction, a custom log file filtering strategy can also be realized, and an error source can be quickly positioned to help debugging. If best gradeThe log information is normal, e.g. the method operating logic is in accordance with the business logicIt indicates that the platform stability is good, and if not,for example, the exception stack information is printed out by log information, such as information accompanied by an exception flag,this indicates that there is a problem with platform stability.
The method performance level can be a perf level, the self-defined perf level is used for outputting a performance log file and recording performance exception information of the transaction method, for example, the method performance level can be the execution duration of the target method, the execution duration of the general method is in a millisecond level, and if the execution duration of the target method is found to reach a second level, the performance of the target method is indicated to be abnormal and needs to be debugged. In addition, the debug level is the lowest, and can be freely used at any level which is considered to be beneficial to more detailed understanding of the running state of the platform during debugging; info is used to feed back the current state of the platform to the level of the end user; the level to which the war platform can continue to run; error cannot determine the level at which the platform will function properly. The platform can respectively configure the belonged level of the log file according to the application scenes of different transactions, and filter the log file lower than the belonged level of the log file, so that the effect of outputting the log files of different belonged levels for different transactions is achieved, and the performance of the distributed system is improved.
S130, determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
The synchronization rule refers to the way log output, and may be, for example, synchronous, fully asynchronous, or mixed asynchronous. The mixed asynchronous mode can be that the log files of which the types are relatively important in the log files are set as synchronous output, and the log files of other types are set as asynchronous output. The scheme preferably adopts a mixed asynchronous mode, and the mixed asynchronous mode can not only accelerate the efficiency of log processing, but also ensure the storage of important log files as far as possible. Meanwhile, the platform can further optimize the performance of log processing according to the corresponding relation between the output screening conditions of the preset log files and the belonged levels of the log files, for example, four types of log files can be defined, which are respectively used for recording system key information, transaction method performance information, program error information and all log information in the transaction process. And corresponding the four types of log files to the grades of the log files, and determining the processing result of the log files according to the obtained corresponding relation and the synchronization rule of the log files. Wherein, the processing result can comprise outputting the log file and not outputting the log file.
In this scheme, optionally, in response to a system operation analysis request, determining system operation stability according to the program operation level, and determining a method to be optimized according to the method performance level.
It can be understood that, if the platform receives a system operation analysis request, whether the distributed system operates stably can be judged according to the program operation level in the belonging level of the log file, the performance of the target method can also be judged according to the method performance level in the belonging level of the log file, and if the performance of the target method does not reach the expected superiority, the target method can be determined as the method to be optimized. By means of the arrangement, the system can be maintained in time based on the log data representing the operation stability of the system and the performance level of the method, and the usability of the log and the convenience degree of the using process are improved.
According to the technical scheme provided by the embodiment of the application, distributed log information acquisition is carried out on a target method through a preset cut-in point to form a log file; then, determining a synchronization rule of the log file according to the type of the log file, and simultaneously determining the belonging level of the log file according to the configuration information; and finally, determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the printing screening condition of the preset log file and the belonged level of the log file. The technical scheme provided by the application can realize efficient log management, and can save calculation power and storage space while improving the performance of the distributed system.
Example two
The present embodiment is a preferred embodiment provided on the basis of the above-described embodiments. Fig. 2 is a flowchart of processing a distributed log according to a second embodiment of the present invention, and the second embodiment of the present invention is optimized based on the foregoing embodiments.
As shown in fig. 2, the method of this embodiment specifically includes the following steps:
s210, distributed log information collection is carried out on the target method through a preset cut-in point, and the collected log information is written into a log file.
S220, determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level.
S230, determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
S240, if the processing result of the log file is the output log file, outputting the log file through a log printing interface according to a preset log file output rule; the preset log file output rule comprises at least one of an output format, an output directory and a log file name.
It can be understood that, if the processing result of the log file is to output the log file, the platform may output the log file at the log printing interface according to the preset log file output rule. The preset log file output rule may include an output format, an output directory, a log file name, and the like.
In this embodiment, specifically, the output format may include a transaction code, a thread name, and a container IP;
the outputting the catalog may include generating a log catalog according to a preset date rule or a preset time rule;
the log file name includes a transaction code, a container IP, and a log type.
In the transaction log file, the transaction codes can be used as important marking information, and one transaction code can represent one transaction, so that convenience is provided for subsequent inquiry and screening. When the target method runs in the container, the container IP can mark the container to realize the positioning of the log file. In addition, through screening, the thread names can gather the log files of the unified threads, and the analysis of the running conditions of the threads is realized.
The generating of the log catalog according to the preset date rule or the preset time rule may be automatically generating the log catalog according to a fixed date or a fixed time interval, for example, generating a catalog at 8:00 every day.
The log type refers to a type to which the log file belongs, and may be, for example, a normal log, an abnormal log, an error log, or the like.
According to the scheme, each piece of log information can be associated based on information such as transaction codes, the content contained in the whole log file can be embodied, and the indexing and calling efficiency of the subsequent log file use process can be improved.
In this embodiment, optionally, the output format may further include transaction flow information, and the transaction code corresponds to at least one piece of transaction flow information.
The transaction chronological information may be all transaction information of the transaction activities recorded in chronological order. The transaction flow information corresponding to a certain transaction code may be all information of the transaction activity, for example, a certain transaction may include multiple stages of transaction activation, order generation, transfer payment, payment success, order submission, transaction completion, and the like, and the transaction flow information corresponding to a certain transaction code may be all information generated by the multiple stages. According to the scheme, the transaction flow information is set, the log data can be further divided from the dimension of the transaction flow, so that the available range of the log data can be expanded, and information with more granularities can be embodied.
In this scheme, optionally, in response to a classification request of the log file, performing log classification according to the transaction code or the transaction flow information.
It is understood that if the platform receives a classification request of the log file, the log classification can be performed according to the transaction code or the transaction stream information. The log file classification request may be various, for example, the log file may be classified according to a transaction code to check a certain transaction activity, or the log file may be classified according to a transaction time in the transaction flow information to check a certain time period. According to the scheme, the log files are classified according to the transaction codes or the transaction flow information, and the using efficiency of the log files of the distributed system can be improved.
According to the technical scheme provided by the embodiment of the application, distributed log information acquisition is carried out on a target method through a preset cut-in point to form a log file; then, determining a synchronization rule of the log file according to the type of the log file, and simultaneously determining the belonging level of the log file according to the configuration information; finally, determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the printing screening condition of the preset log file and the belonged level of the log file; and when the processing result of the log file is the output log file, outputting the log file through a log printing interface according to a preset log file output rule. The technical scheme provided by the application can realize efficient log management, and can save calculation power and storage space while improving the performance of the distributed system.
EXAMPLE III
Fig. 3 is a block diagram of a distributed log processing apparatus according to a third embodiment of the present invention, where the apparatus is capable of executing a distributed log processing method according to any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method. As shown in fig. 3, the apparatus may include:
the log information writing module 310 is configured to perform distributed log information acquisition on a target method through a preset cut-in point, and write the acquired log information into a log file;
the log file belonging level determining module 320 is used for determining a synchronization rule of the log file according to the belonging type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
the log file processing result determining module 330 is configured to determine a processing result of the log file according to the synchronization rule of the log file and according to a corresponding relationship between an output screening condition of a preset log file and a belonging level of the log file; wherein the processing result comprises an output log file and a non-output log file.
In this embodiment, optionally, the apparatus further includes:
the output module is used for outputting through the log printing interface according to a preset log file output rule; the preset log file output rule comprises at least one of an output format, an output directory and a log file name.
Optionally, the output format includes a transaction code, a thread name, and a container IP;
the output catalog comprises a log catalog generated according to a preset date rule or a preset time rule;
the log file name includes a transaction code, a container IP, and a log type.
In this embodiment, optionally, the output format further includes transaction flow information, and the transaction code corresponds to at least one piece of transaction flow information.
In this embodiment, optionally, the apparatus further includes:
and the log classifying module is used for responding to a classifying request of the log file and classifying the log according to the transaction code or the transaction running water information.
In this embodiment, optionally, the apparatus further includes:
the log file mapping module is used for mapping the log file collected in the Docker container to a host machine if the target method runs in the Docker container;
and if the log file meets the storage condition of the database, forwarding the log file on the host machine to a machine data engine server through a machine data engine agent so as to store the log file in the database.
In this embodiment, optionally, the apparatus further includes:
and the to-be-optimized method determining module is used for responding to a system operation analysis request, determining the system operation stability according to the program operation level, and determining the to-be-optimized method according to the method performance level.
The product can execute the processing method of the distributed log provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method.
Example four
A fourth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a method for processing a distributed log according to the embodiments of the present invention:
carrying out distributed log information acquisition on a target method through a preset cut-in point, and writing the acquired log information into a log file;
determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
EXAMPLE five
The fifth embodiment of the application provides electronic equipment. Fig. 4 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application. As shown in fig. 4, the present embodiment provides an electronic device 400, which includes: one or more processors 420; the storage device 410 is configured to store one or more programs, and when the one or more programs are executed by the one or more processors 420, the one or more processors 420 implement the method for processing the distributed log provided in the embodiment of the present application, the method includes:
carrying out distributed log information acquisition on a target method through a preset cut-in point, and writing the acquired log information into a log file;
determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
Of course, those skilled in the art can understand that the processor 420 also implements the technical solution of the processing method of the distributed log provided in any embodiment of the present application.
The electronic device 400 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 4, the electronic device 400 includes a processor 420, a storage device 410, an input device 430, and an output device 440; the number of the processors 420 in the electronic device may be one or more, and one processor 420 is taken as an example in fig. 4; the processor 420, the storage device 410, the input device 430, and the output device 440 in the electronic apparatus may be connected by a bus or other means, and are exemplified by a bus 450 in fig. 4.
The storage device 410 is a computer-readable storage medium, and can be used to store software programs, computer-executable programs, and module units, such as program instructions corresponding to the processing method of the distributed log in the embodiment of the present application.
The storage device 410 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 410 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 410 may further include memory located remotely from processor 420, which may be connected via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 430 may be used to receive input numbers, character information, or voice information, and to generate key signal inputs related to user settings and function control of the electronic device. The output device 440 may include a display screen, speakers, or other electronic equipment.
According to the electronic device provided by the embodiment of the application, the log file is formed by carrying out distributed log information acquisition on the target method through the preset cut-in point, and the further management of the log file is realized according to the type of the log file. Therefore, efficient log management is realized, and the operation cost is saved while the performance of the distributed system is improved.
The processing device, medium, and electronic device for the distributed log provided in the foregoing embodiments may execute the processing method for the distributed log provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method. Technical details that are not described in detail in the above embodiments may be referred to a processing method of a distributed log provided in any embodiment of the present application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for processing distributed logs, the method comprising:
carrying out distributed log information acquisition on a target method through a preset cut-in point, and writing the acquired log information into a log file;
determining a synchronization rule of the log file according to the type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
determining a processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
2. The method of claim 1, further comprising:
outputting through a log printing interface according to a preset log file output rule; the preset log file output rule comprises at least one of an output format, an output directory and a log file name.
3. The method of claim 2, wherein the output format comprises a transaction code, a thread name, and a container IP;
the output catalog comprises a log catalog generated according to a preset date rule or a preset time rule;
the log file name includes a transaction code, a container IP, and a log type.
4. The method of claim 3, wherein the output format further includes transaction flow information, and wherein the transaction code corresponds to at least one transaction flow information.
5. The method of claim 4, further comprising:
and responding to a classification request of the log file, and performing log classification according to the transaction code or the transaction running water information.
6. The method of claim 1, wherein if the target method runs in a Docker container, mapping a log file collected in the Docker container onto a host machine;
and if the log file meets the storage condition of the database, forwarding the log file on the host machine to a machine data engine server through a machine data engine agent so as to store the log file in the database.
7. The method of claim 1, further comprising:
and responding to a system operation analysis request, determining the system operation stability according to the program operation level, and determining the method to be optimized according to the method performance level.
8. An apparatus for processing a distributed log, the apparatus comprising:
the log information writing module is used for carrying out distributed log information acquisition on a target method through a preset cut-in point and writing the acquired log information into a log file;
the log file belonging level determining module is used for determining the synchronization rule of the log file according to the belonging type of the log file; determining the level of the log file according to the configuration information; wherein the affiliated level comprises a program running level and a method performance level;
the log file processing result determining module is used for determining the processing result of the log file according to the synchronization rule of the log file and the corresponding relation between the output screening condition of the preset log file and the belonged level of the log file; wherein the processing result comprises an output log file and a non-output log file.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method of processing a distributed log according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of processing a distributed log according to any of claims 1-7 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110722036.0A CN113434473A (en) | 2021-06-28 | 2021-06-28 | Distributed log processing method, device, medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110722036.0A CN113434473A (en) | 2021-06-28 | 2021-06-28 | Distributed log processing method, device, medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113434473A true CN113434473A (en) | 2021-09-24 |
Family
ID=77757372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110722036.0A Pending CN113434473A (en) | 2021-06-28 | 2021-06-28 | Distributed log processing method, device, medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113434473A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115114250A (en) * | 2022-06-29 | 2022-09-27 | 重庆长安汽车股份有限公司 | Log storage method and system, electronic equipment and storage medium |
| CN115129682A (en) * | 2022-08-30 | 2022-09-30 | 北京智芯微电子科技有限公司 | Log management method and log management system |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183333A (en) * | 2007-12-12 | 2008-05-21 | 腾讯科技（深圳）有限公司 | Method for outputting diary of debugging program, device and system |
| US20110246826A1 (en) * | 2010-03-31 | 2011-10-06 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
| CN103488558A (en) * | 2013-09-17 | 2014-01-01 | 北京思特奇信息技术股份有限公司 | Device and method of automatically acquiring application anomalies based on LOG4J logging framework |
| CN106156355A (en) * | 2016-07-27 | 2016-11-23 | 腾讯科技（深圳）有限公司 | Log processing method and device |
| CN109460343A (en) * | 2018-09-10 | 2019-03-12 | 平安科技（深圳）有限公司 | System exception monitoring method, device, equipment and storage medium based on log |
| CN109614379A (en) * | 2018-10-22 | 2019-04-12 | 中国平安人寿保险股份有限公司 | Log-output method, device, computer storage medium and computer equipment |
| CN110046155A (en) * | 2018-12-07 | 2019-07-23 | 阿里巴巴集团控股有限公司 | The update of property data base, data characteristics determination method, device and equipment |
| CN110083582A (en) * | 2019-03-22 | 2019-08-02 | 平安普惠企业管理有限公司 | Method, apparatus, computer equipment and the storage medium of output journal |
| CN110377244A (en) * | 2019-07-12 | 2019-10-25 | 苏州浪潮智能科技有限公司 | A kind of log Method of printing, device, equipment and storage medium |
| CN112346721A (en) * | 2020-10-23 | 2021-02-09 | 杭州安恒信息技术股份有限公司 | Log acquisition method and device based on section-oriented programming and electronic equipment |
| CN112799903A (en) * | 2019-11-14 | 2021-05-14 | 北京沃东天骏信息技术有限公司 | Method and device for evaluating health state of business system |
-
2021
- 2021-06-28 CN CN202110722036.0A patent/CN113434473A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183333A (en) * | 2007-12-12 | 2008-05-21 | 腾讯科技（深圳）有限公司 | Method for outputting diary of debugging program, device and system |
| US20110246826A1 (en) * | 2010-03-31 | 2011-10-06 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
| CN103488558A (en) * | 2013-09-17 | 2014-01-01 | 北京思特奇信息技术股份有限公司 | Device and method of automatically acquiring application anomalies based on LOG4J logging framework |
| CN106156355A (en) * | 2016-07-27 | 2016-11-23 | 腾讯科技（深圳）有限公司 | Log processing method and device |
| CN109460343A (en) * | 2018-09-10 | 2019-03-12 | 平安科技（深圳）有限公司 | System exception monitoring method, device, equipment and storage medium based on log |
| CN109614379A (en) * | 2018-10-22 | 2019-04-12 | 中国平安人寿保险股份有限公司 | Log-output method, device, computer storage medium and computer equipment |
| CN110046155A (en) * | 2018-12-07 | 2019-07-23 | 阿里巴巴集团控股有限公司 | The update of property data base, data characteristics determination method, device and equipment |
| CN110083582A (en) * | 2019-03-22 | 2019-08-02 | 平安普惠企业管理有限公司 | Method, apparatus, computer equipment and the storage medium of output journal |
| CN110377244A (en) * | 2019-07-12 | 2019-10-25 | 苏州浪潮智能科技有限公司 | A kind of log Method of printing, device, equipment and storage medium |
| CN112799903A (en) * | 2019-11-14 | 2021-05-14 | 北京沃东天骏信息技术有限公司 | Method and device for evaluating health state of business system |
| CN112346721A (en) * | 2020-10-23 | 2021-02-09 | 杭州安恒信息技术股份有限公司 | Log acquisition method and device based on section-oriented programming and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| SHEEN口开河: "Log4j2简介和异步日志梳理", 《简书》 * |
| 鸭血粉丝: "全网最全Log配置教程及框架性能比较，看这篇就够了！", 《微信公众平台》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115114250A (en) * | 2022-06-29 | 2022-09-27 | 重庆长安汽车股份有限公司 | Log storage method and system, electronic equipment and storage medium |
| CN115129682A (en) * | 2022-08-30 | 2022-09-30 | 北京智芯微电子科技有限公司 | Log management method and log management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108595316B (en) | Lifecycle management method, manager, device, and medium for distributed application | |
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| CN110245078A (en) | A kind of method for testing pressure of software, device, storage medium and server | |
| CN111752799A (en) | Service link tracking method, device, equipment and storage medium | |
| CN112306719B (en) | Task scheduling method and device | |
| CN111399909B (en) | Service system data distribution processing method, device and storage medium | |
| CN113434473A (en) | Distributed log processing method, device, medium and electronic equipment | |
| CN112612523B (en) | Embedded equipment driving system and method | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| US20220179711A1 (en) | Method For Platform-Based Scheduling Of Job Flow | |
| CN111143167A (en) | Alarm merging method, device, equipment and storage medium for multiple platforms | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN112187509A (en) | Multi-architecture cloud platform execution log management method, system, terminal and storage medium | |
| CN111552521A (en) | Application data reporting method, device, server and storage medium | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN117560277A (en) | Log processing method and system, log management platform and electronic equipment | |
| US8566279B1 (en) | Enhanced data collection techniques | |
| CN115757045A (en) | Transaction log analysis method, system and device | |
| CN112134927A (en) | Power grid data exchange platform and data exchange method based on plug-in mode | |
| CN116483543A (en) | Task processing method, device, equipment and storage medium | |
| CN114090382B (en) | Health inspection method and device for super-converged cluster | |
| CN111698109A (en) | Method and device for monitoring log | |
| CN114756301A (en) | Log processing method, device and system | |
| CN114020565A (en) | Intelligent log collection processing method and device, electronic equipment and storage medium | |
| CN115204251A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210924 |