CN110362547B - Method and device for encoding, analyzing and storing log file - Google Patents
Method and device for encoding, analyzing and storing log file Download PDFInfo
- Publication number
- CN110362547B CN110362547B CN201810283468.4A CN201810283468A CN110362547B CN 110362547 B CN110362547 B CN 110362547B CN 201810283468 A CN201810283468 A CN 201810283468A CN 110362547 B CN110362547 B CN 110362547B
- Authority
- CN
- China
- Prior art keywords
- event
- data
- log file
- data body
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a method and a device for encoding, analyzing and storing a log file, wherein the encoding method comprises the following steps: acquiring event data of a client, wherein the event data comprises client description information, event description information and event record data; generating a text description section according to the client description information and the event description information; generating an event data body according to the event record data; the text description segment and the event data volume are encoded into a log file. According to the embodiment, the coding protocol of the log file is provided, so that various data of the client can be uniformly coded according to the coding protocol, the expandability and flexibility of the data are guaranteed, the coded log file is conveniently used in a plurality of different platforms, and the cross-platform compatibility of the data is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method for encoding a log file, a device for encoding a log file, a method for storing a log file, a device for storing a log file, a method for parsing a log file, and a device for parsing a log file.
Background
A Client (Client), also referred to as a Client, refers to an application corresponding to a server that provides local services to the Client. More common clients include web browsers for use on the world wide web, email clients for receiving and sending email, chat software for instant messaging, and the like.
In the process of using a client application program, a user often causes business anomalies or system anomalies such as application program crashes and the like due to various reasons. In order to improve the system performance of client applications, developers often need to troubleshoot and duplicate the anomalies that occur. In the case of troubleshooting and recovering from an abnormality, the client application is required to provide as full and complete as possible information of the abnormality.
However, the occurrence frequency of the abnormal event is high, the data volume is large, and the data of the abnormal event can not be completely recorded due to the reasons of memory data loss, file writing failure, network request failure and the like. When recording data, if the performance of data processing is not high enough, the usability of the application program is also greatly affected. In addition, with the rapid expansion of the data volume to be recorded, a large amount of storage space and network traffic of the user terminal are consumed according to the conventional data storage method.
Disclosure of Invention
In view of the foregoing, embodiments of the present application are directed to providing a method for encoding a log file, an apparatus for encoding a log file, a method for storing a log file, an apparatus for storing a log file, a method for parsing a log file, and a corresponding apparatus for parsing a log file, which overcome or at least partially solve the foregoing problems.
In order to solve the above problems, the present application discloses a method for encoding a log file, comprising:
acquiring event data of a client, wherein the event data comprises client description information, event description information and event record data;
generating a text description section according to the client description information and the event description information;
generating an event data body according to the event record data;
the text description segment and the event data volume are encoded into a log file.
In order to solve the above problems, the present application discloses a method for storing log files, comprising:
acquiring event data of a client;
encoding the event data into a log file;
determining a memory address space;
and storing the log file into the memory address space.
In order to solve the above problems, the present application discloses a method for parsing log files, comprising:
acquiring a log file, wherein the log file comprises a text description section and an event data body;
reading event description information in the text description section;
and analyzing the event data body according to the event description information.
In order to solve the above problems, the present application discloses a log file encoding device, comprising:
the system comprises an event data acquisition module, a data processing module and a data processing module, wherein the event data acquisition module is used for acquiring event data of a client, and the event data comprises client description information, event description information and event record data;
the text description section generation module is used for generating a text description section according to the client description information and the event description information;
the event data body generation module is used for generating an event data body according to the event record data;
and the log file encoding module is used for encoding the text description section and the event data body into a log file.
In order to solve the above problems, the present application discloses a log file storage device, comprising:
the event data acquisition module is used for acquiring event data of the client;
the log file coding module is used for coding the event data into a log file;
The memory address space determining module is used for determining a memory address space;
and the log file storage module is used for storing the log file into the memory address space.
In order to solve the above problems, the present application discloses a log file parsing apparatus, comprising:
the system comprises a log file acquisition module, a log file generation module and a log file generation module, wherein the log file acquisition module is used for acquiring a log file, and the log file comprises a text description section and an event data body;
the event description information reading module is used for reading the event description information in the text description section;
and the event data body analysis module is used for analyzing the event data body according to the event description information.
Compared with the background art, the embodiment of the application has the following advantages:
according to the embodiment of the application, the event data of the client is obtained, the text description section is generated according to the client description information and the event description information in the event data, and the event data body is generated according to the event record data, so that the text description section and the event data body can be encoded into the log file. The embodiment provides the coding protocol of the log file, so that various data of the client can be uniformly coded according to the coding protocol, and the coding protocol of the embodiment is improved by more than 5 times compared with the json data protocol widely used at present in the compression ratio, thereby solving the influence of data expansion of the client on equipment or the client in the prior art. Meanwhile, the type, the data structure and the coding information in the event data are written into the data file through the description information of the streaming data body, data analysis can be carried out on the data of any platform through the meaning and the format represented by each record type in the streaming data body, and the interpretation information of the data body is carried out by the data file, so that other information besides the data is not needed in the process of carrying out data analysis, the expandability and the flexibility of the data are greatly ensured, the coded log file is conveniently used in a plurality of different platforms, the cross-platform compatibility of the data is improved, and the complexity brought to data analysis by frequent change and expansion of the data is greatly reduced.
Drawings
FIG. 1 is a flow chart of steps of an embodiment of a method for encoding a log file of the present application;
FIG. 2 is a schematic diagram of a binary data volume of the present application;
FIG. 3 is a coding structure diagram of a log file of the present application;
FIG. 4 is a flow chart of steps of an embodiment of a method for storing log files of the present application;
FIG. 5 is a block diagram of a method of storing log files according to the present application;
FIG. 6 is a flow chart of steps of an embodiment of a method for parsing a log file according to the present application;
FIG. 7 is a schematic diagram of a method for parsing a log file according to the present application;
FIG. 8 is a block diagram illustrating an embodiment of an encoding apparatus for log files according to the present application;
FIG. 9 is a block diagram illustrating the structure of an embodiment of a log file storage device of the present application;
fig. 10 is a block diagram illustrating an embodiment of a log file parsing apparatus according to the present application.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description.
Referring to fig. 1, a flowchart illustrating steps of an embodiment of a method for encoding a log file according to the present application may specifically include the following steps:
Step 101, acquiring event data of a client, wherein the event data comprises client description information, event description information and event record data;
it should be noted that, the client may refer to an application program installed on various terminal devices such as a mobile phone, a tablet computer, and the like. For example, a cell phone panning APP, a payment APP, etc. installed on a cell phone. The type of the terminal device and the client is not limited in this embodiment.
In the embodiment of the application, the event data can comprise client description information, event record data and the like.
The client description information may include client information (i.e., information of a client or an application itself), device information (i.e., information of a terminal device in which the client or the application is installed), and environment information (i.e., information of a network state of a user when using the client or the application).
In particular implementations, the client information may include at least one of the following:
protocol version number, application version number, device identification information.
The device information may include at least one of the following:
device hardware type, CPU core number, CPU main frequency, display parameters, operating system version, and sensor type supported by the device.
The environmental information may include at least one of the following:
device network status, operator information.
Of course, the above is merely an example, and the above information may also include other different information or data according to actual needs, which is not limited in this embodiment.
In the embodiment of the present application, the event description information may include event information and format information of event record data corresponding to each event.
The respective events may be events generated by the client itself or events generated by a user when using the client. Such as system events, user operation events, application execution events, network events, and the like.
The event information may be information or data related to the event recorded when the event of the above-mentioned kind occurs, and the format information indicates information of a recording format or parameters adopted when the event is recorded.
In the embodiment of the application, the event record data can be information or data which is completely recorded when various events occur.
Step 102, generating a text description section according to the client description information and the event description information;
in the embodiment of the application, the text description section of the log file can be generated according to the obtained client description information and the event description information.
Thus, the text description section of a log file may include client information (protocol version number, application version number, device identification information, etc.), device information (device hardware type, CPU core number, CPU main frequency, display parameters, operating system version, sensor type supported by the device, etc.), environment information (device network status, operator information, etc.), and event description information.
Step 103, generating an event data body according to the event record data;
in embodiments of the present application, the event data body may be a binary data body that may serially record the various events of the client or application.
In a specific implementation, the binary data body may include a plurality of records, each record may include a text header and a text body, and each record records an event occurring on the client or the application program.
When generating the binary data body, the event type, the occurrence time and the size of the event record data of each event can be determined first, and the event type, the occurrence time and the size of the event record data of each event are written into the text header of the binary data body respectively. Then, event record data of each event is written into a text body of the binary data body in the order of the text head, thereby obtaining the binary data body.
As shown in fig. 2, a schematic diagram of a binary data body of the present application. In fig. 2, an event type (record type) may be recorded using 2 bytes, an event occurrence time (time) may be recorded using 4 bytes, a data body size (body size) may be recorded using 4 bytes, and then a text body conforming to the corresponding data body size may be recorded.
In the embodiment of the application, the event data body can also comprise a binary data head, and the binary data head can have a fixed byte length and is used for recording various attribute information of the corresponding event data body.
In a specific implementation, when generating the binary data header, attribute information of the event data body may be first determined, where the attribute information may include information such as a Magic number, a version of the protocol, a data body generation time, and a time offset value. Then, the information of the above-mentioned Magic number, protocol version, data volume generation time, and/or time offset value may be written into the binary data header.
For example, a 4-byte recording protocol Magic number may be used, a length of a binary header may be recorded using 2 bytes, a start time of log recording may be recorded using 8 bytes, and then the remaining bytes may be recorded as 0.
Of course, the information written in the binary data header may also include other information, for example, whether the file overflows or not, etc., which is not limited in this embodiment.
And 104, encoding the text description section and the event data body into a log file.
In the embodiment of the application, after the event data of the client is acquired and the text description section and the event data body of the log file to be generated are obtained according to the event data, the file description section and the event data body can be encoded into the log file so as to store the log file.
As shown in fig. 3, a log file encoding structure diagram of the present application is shown. In fig. 3, a text description section header, a binary header, and a binary data body are included. The text header includes information such as data self-description information type-descriptors, and the binary header further includes information such as a Magic number of a protocol, a length header length of a binary header, and a start time record start time of log record. While for a binary data volume, its parts are similar to the binary data shown in fig. 2.
The data self-description information type-descriptions records which rule or format the server should interpret the data.
In general, during the interaction between the client and the server, the server parses the data reported by the client according to a predetermined format. However, the content and format of the data often change, for example, when the client performs version iteration, the server cannot well accommodate data with different versions or formats, so that the convention needs to be revised again, the communication cost between the client and the server is increased, and maintenance of the developer is troublesome. In order to reduce maintenance burden, the embodiment writes the self-description information of the data into the text description section in the encoding process of the log file, so that the data itself has the explanation information of the data body, and other information besides the data is not needed in the data analysis process, thereby greatly ensuring the expandability and flexibility of the data.
In the embodiment of the application, the text description section is generated according to the client description information and the event description information in the event data by acquiring the event data of the client, and the event data body is generated according to the event record data, so that the text description section and the event data body can be encoded into a log file. The embodiment provides the coding protocol of the log file, so that various data of the client can be uniformly coded according to the coding protocol, and the coding protocol of the embodiment is improved by more than 5 times compared with the json data protocol widely used at present in the compression ratio, thereby solving the influence of data expansion of the client on equipment or the client in the prior art. Meanwhile, the type, the data structure and the coding information in the event data are written into the data file through the description information of the streaming data body, data analysis can be carried out on the data of any platform through the meaning and the format represented by each record type in the streaming data body, and the interpretation information of the data body is carried out by the data file, so that other information besides the data is not needed in the process of carrying out data analysis, the expandability and the flexibility of the data are greatly ensured, the coded log file is conveniently used in a plurality of different platforms, the cross-platform compatibility of the data is improved, and the complexity brought to data analysis by frequent change and expansion of the data is greatly reduced.
Referring to fig. 4, a flowchart illustrating steps of an embodiment of a method for storing a log file according to the present application may specifically include the following steps:
step 401, obtaining event data of a client;
it should be noted that, the client may refer to an application program installed on various terminal devices such as a mobile phone, a tablet computer, and the like. For example, a cell phone panning APP, a payment APP, etc. installed on a cell phone. The type of the terminal device and the client is not limited in this embodiment.
In the embodiment of the application, the event data can comprise client description information, event record data and the like.
The client description information may include client information (i.e., information of a client or an application itself), device information (i.e., information of a terminal device in which the client or the application is installed), and environment information (i.e., information of a network state of a user when using the client or the application).
In particular implementations, the client information may include at least one of the following:
protocol version number, application version number, device identification information.
The device information may include at least one of the following:
Device hardware type, CPU core number, CPU main frequency, display parameters, operating system version, and sensor type supported by the device.
The environmental information may include at least one of the following:
device network status, operator information.
Of course, the above is merely an example, and the above information may also include other different information or data according to actual needs, which is not limited in this embodiment.
In the embodiment of the present application, the event description information may include event information and format information of event record data corresponding to each event.
The respective events may be events generated by the client itself or events generated by a user when using the client. Such as system events, user operation events, application execution events, network events, and the like.
The event information may be information or data related to the event recorded when the event of the above-mentioned kind occurs, and the format information indicates information of a recording format or parameters adopted when the event is recorded.
In the embodiment of the application, the event record data can be information or data which is completely recorded when various events occur.
Step 402, encoding the event data into a log file;
in the embodiment of the application, when the event data is encoded into the log file, a text description section can be generated according to the client description information and the event description information, an event data body can be generated according to the event record data, and then the text description section and the event data body are encoded into the log file.
Since step 402 in this embodiment is similar to steps 102 to 104 in the above embodiment, reference can be made to each other, and the description thereof will not be repeated.
Step 403, determining a memory address space;
in the embodiment of the present application, the memory address space may refer to an address space where the user stores the generated log file, and the memory address space may have a fixed size.
In a specific implementation, after determining the memory address space, a preset method may be further used to initialize the memory address space. For example, the mmap method may be employed.
mmap is a method for mapping files in a memory, namely mapping a file or other objects to an address space of a process, and realizing one-to-one correspondence between file disk addresses and a section of virtual address in a process virtual address space. After the mapping relation is realized, the process can read and write the memory section by adopting a pointer mode, and the system can automatically write back pages to the corresponding file disk, namely the operation on the file is completed without calling system calling functions such as read, write and the like. On the contrary, the modification of the memory space to the section of the region also directly reflects the user space, so that file sharing among different processes can be realized.
Step 404, storing the log file in the memory address space.
In the embodiment of the application, when the log file is stored in the memory address space, whether the memory address space overflows or not can be firstly determined, if so, the data in the memory address space can be obtained, and the data can be written into the disk; if not, a step of storing the log file into the memory address space may be performed.
In a specific implementation, the memory address space mapped by using mmap is limited in size, and after the used size of the corresponding space exceeds a threshold value, the sub-thread can write the data copy stored in the mapped memory address space into the disk file.
Fig. 5 is a block diagram showing a method for storing log files according to the present application. In fig. 5, for event data (such as a system event, a user operation event, an application program execution event, a network event, etc.) of a client to be stored, the client may be initialized first, then a mmap method is used to initialize a determined memory address space, and after a text description section and an event data body of a log file are generated, whether the memory address space overflows or not is checked, thereby implementing the storage of the log file. If the memory address space overflows, data in the memory address space can be written to disk by the sub-thread, and then the pointer is moved back to the initial value. If the memory address space does not overflow, indicating that the address may also store a log file, the log file may be written to the memory address space.
In the embodiment of the application, the event data of the client is obtained, the event data can be encoded into a log file, and then the log file can be stored into the memory address space after the memory address space is determined. According to the embodiment, a mmap method can be used for mapping the log file to a memory address space with a fixed size, and performance loss caused by directly operating the file can be avoided by operating a memory and synchronizing data to the log file by a system kernel; meanwhile, when extreme conditions such as client side or application program crash occur, even if a user process is killed, the subprocess can still write data in the mmap address space into the file, so that the data integrity of the log file is guaranteed to the greatest extent.
Referring to fig. 6, a step flow chart of an embodiment of a log file parsing method according to the present application may specifically include the following steps:
step 601, acquiring a log file, wherein the log file comprises a text description section and an event data body;
it should be noted that, the log file in this embodiment may refer to a log file obtained by encoding or storing according to the encoding method of the log file or the encoding protocol shown in the storing method of the log file, and each log file may include a text description section and an event data body.
Step 602, reading event description information in the text description section;
in the embodiment of the present application, the event description information may include event information and format information of event record data corresponding to each event.
The respective events may be events generated by the client itself or times generated by the user when using the client. Such as system events, user operation events, application execution events, network events, and the like.
The event information may be information or data related to the event recorded when the event of the above-mentioned kind occurs, and the format information indicates information of a recording format or parameters adopted when the event is recorded.
For example, for an open page (openPage) event, the corresponding event description information may be expressed as:
openpage:u4:u1*,freeMemory:f,residentMemory:f,virtualMemory:f,cpuUsa geOfApp:f,cpuUsageOfDevice:f
and step 603, analyzing the event data body according to the event description information.
In the embodiment of the application, after the event description information in the text description section is read, the event data body corresponding to the read event description information can be determined first, and then the event data body is analyzed.
In a specific implementation, since the event data volume includes a binary data header and a binary data volume. Therefore, during analysis, the binary data head can be read first to obtain attribute information such as the Magic number, the protocol version, the data volume generation time, and the time deviation value of the event data volume. Then, the binary data volume is parsed.
As shown in fig. 7, which is a schematic diagram of a log file parsing method according to the present application, 10 bytes can be read first, wherein 1-2 bytes are parsed into a short event type (openPage in fig. 7), and corresponding event description information is searched from a text description section according to the event type, so as to determine which data format is used to parse the current event; 3-6 bytes are parsed into integer form as the time of occurrence of the event; 7-10 bytes are parsed into integer form, recorded as event body length.
Then, the event body is parsed according to the event description information. Taking an openPage event as an example: reading 4 bytes, analyzing the length bytes into an unsigned integer, continuously reading the length bytes corresponding to the integer value, analyzing the length bytes into a character string, and recording the character string as a page; reading 4 bytes, analyzing the read data into a float type, and recording the read data as a freeMemory; reading 4 bytes resolves to float type, records as virtualmery, until all other fields shown in fig. 7 are resolved.
It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred embodiments, and that the acts are not necessarily required by the embodiments of the application.
Referring to fig. 8, a block diagram illustrating an embodiment of an encoding apparatus for log files according to the present application may specifically include the following modules:
an event data obtaining module 801, configured to obtain event data of a client, where the event data includes client description information, event description information, and event record data;
a text description section generating module 802, configured to generate a text description section according to the client description information and the event description information;
an event data body generating module 803, configured to generate an event data body according to the event record data;
a log file encoding module 804, configured to encode the text description segment and the event data body into a log file.
In the embodiment of the application, the client description information can comprise client information, equipment information and environment information;
the event description information may include event information and format information of event record data corresponding to each event.
In an embodiment of the present application, the client information may include at least one of the following:
protocol version number, application version number, device identification information;
the device information may include at least one of:
The device hardware type, the CPU core number, the CPU main frequency, the display parameter, the operating system version and the sensor type supported by the device;
the environmental information may include at least one of:
device network status, operator information.
In the embodiment of the present application, the event data body may be a binary data body, and the event data body generating module 803 may specifically include the following sub-modules:
the determining submodule is used for determining the event type and the occurrence time of each event and the size of event record data;
the text head writing sub-module is used for writing the event type and the occurrence time of each event and the size of the event record data into the text head of the binary data body;
and the text body writing sub-module is used for writing the event record data of each event into the text body of the binary data body according to the sequence of the text head.
In an embodiment of the present application, the event data body may further include a binary data header, and the event data body generating module 803 may further include the following sub-modules:
the attribute information determining sub-module is used for determining attribute information of the event data body, wherein the attribute information can comprise a protocol magic number, a protocol version, data body generation time and a time deviation value;
A binary data header writing sub-module, configured to write the protocol magic number, the protocol version, the data volume generation time, and/or the time offset value into a binary data header, where the binary data header has a fixed byte length.
Referring to fig. 9, a block diagram illustrating an embodiment of a log file storage device according to the present application may specifically include the following modules:
the event data acquisition module 901 is configured to acquire event data of a client;
a log file encoding module 902, configured to encode the event data into a log file;
a memory address space determining module 903, configured to determine a memory address space;
a log file storage module 904, configured to store the log file into the memory address space.
In the embodiment of the application, the event data can comprise client description information, event description information and event record data; the log file encoding module 902 may specifically include the following sub-modules:
the text description section generation sub-module is used for generating a text description section according to the client description information and the event description information;
the event data body generation sub-module is used for generating an event data body according to the event record data;
And the log file encoding sub-module is used for encoding the text description section and the event data body into a log file.
In an embodiment of the present application, the apparatus may further include the following modules:
the initialization module is used for initializing the memory address space by adopting a preset method, and the preset method can comprise a mmap method.
In the embodiment of the present application, the log file storage module 904 may specifically include the following sub-modules:
a determining submodule, configured to determine whether the memory address space overflows;
the writing sub-module is used for acquiring data in the memory address space and writing the data into a disk when the memory address space overflows;
and the storage sub-module is used for calling the log file storage module 904 when the memory address space is not overflowed.
Referring to fig. 10, a block diagram of an embodiment of a log file parsing apparatus according to the present application is shown, which may specifically include the following modules:
a log file obtaining module 1001, configured to obtain a log file, where the log file may include a text description section and an event data body;
an event description information reading module 1002, configured to read event description information in the text description section;
The event data body parsing module 1003 is configured to parse the event data body according to the event description information.
In the embodiment of the present application, the event data body parsing module 1003 may specifically include the following sub-modules:
the determining submodule is used for determining an event data body corresponding to the event description information, and the event data body can be a binary data body;
and the analysis sub-module is used for analyzing the binary event data body.
In the embodiment of the present application, the event data body may further include a binary data header, and the event data body parsing module 1003 may further include the following sub-modules:
the reading sub-module is used for reading the binary data head and acquiring attribute information of the event data body, wherein the attribute information can comprise a protocol magic cube, a protocol version, data body generation time and a time deviation value.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
In a typical configuration, the computer device includes one or more processors (CPUs), an input/output interface, a network interface, and memory. The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media. Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include non-transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the application.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The above description of the present application provides a method for encoding a log file, an apparatus for encoding a log file, a method for storing a log file, an apparatus for storing a log file, a method for parsing a log file, and an apparatus for parsing a log file, which are described in detail, and specific examples are applied herein to illustrate the principles and embodiments of the present application, and the description of the above examples is only for helping to understand the method and core ideas of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (12)
1. A method of encoding a log file, comprising:
acquiring event data of a client, wherein the event data comprises client description information, event description information and event record data;
generating a text description section according to the client description information and the event description information;
generating an event data body according to the event record data;
encoding the text description section and the event data body into a log file;
The event data body is a binary data body, and the generating the event data body according to the event record data includes:
determining the event type and the occurrence time of each event and the size of event record data;
writing the event type, the occurrence time of each event and the size of event record data into the text head of a binary data body;
and writing the event record data of each event into the text body of the binary data body according to the sequence of the text head.
2. The method of claim 1, wherein the client description information includes client information, device information, and environmental information;
the event description information comprises event information and format information of event record data corresponding to each event.
3. The method of claim 2, wherein the client information comprises at least one of:
protocol version number, application version number, device identification information;
the device information includes at least one of:
the device hardware type, the CPU core number, the CPU main frequency, the display parameter, the operating system version and the sensor type supported by the device;
The environmental information includes at least one of:
device network status, operator information.
4. The method of claim 1, wherein the event data body further comprises a binary data header, the method further comprising:
determining attribute information of the event data body, wherein the attribute information comprises a protocol magic number, a protocol version, data body generation time and a time deviation value;
the protocol magic number, protocol version, data volume generation time, and/or time offset value are written into a binary data header having a fixed byte length.
5. A method for storing a log file, comprising:
acquiring event data of a client;
encoding the event data into a log file;
determining a memory address space;
storing the log file to the memory address space;
the event data comprises client description information, event description information and event record data; the encoding the event data into a log file includes:
generating a text description section according to the client description information and the event description information;
generating an event data body according to the event record data;
Encoding the text description section and the event data body into a log file;
the event data body is a binary data body, and the generating the event data body according to the event record data includes:
determining the event type and the occurrence time of each event and the size of event record data;
writing the event type, the occurrence time of each event and the size of event record data into the text head of a binary data body;
and writing the event record data of each event into the text body of the binary data body according to the sequence of the text head.
6. The method of claim 5, further comprising, after said determining a memory address space:
initializing the memory address space by adopting a preset method, wherein the preset method comprises a mmap method.
7. The method of claim 5, wherein storing the log file to the memory address space comprises:
determining whether the memory address space overflows;
if yes, acquiring data in the memory address space, and writing the data into a disk;
and if not, executing the step of storing the log file into the memory address space.
8. The method for analyzing the log file is characterized by comprising the following steps:
acquiring a log file, wherein the log file comprises a text description section and an event data body; the log file is obtained by encoding event data of the client, wherein the event data comprises client description information, event description information and event record data; the log file is obtained by encoding event data of a client in the following way: generating a text description section according to the client description information and the event description information; generating an event data body according to the event record data; encoding the text description section and the event data body into a log file; the event data body is a binary data body, and the generating the event data body according to the event record data includes: determining the event type and the occurrence time of each event and the size of event record data; writing the event type, the occurrence time of each event and the size of event record data into the text head of a binary data body; writing the event record data of each event into a text body of a binary data body according to the sequence of the text head;
Reading event description information in the text description section;
analyzing the event data body according to the event description information;
the analyzing the event data body according to the event description information comprises the following steps:
determining an event data body corresponding to the event description information, wherein the event data body is a binary data body;
and analyzing the binary data body.
9. The method of claim 8, wherein the event data body further comprises a binary data header, the method further comprising:
and reading the binary data head, and acquiring attribute information of the event data body, wherein the attribute information comprises a protocol magic number, a protocol version, data body generation time and a time deviation value.
10. A log file encoding apparatus, comprising:
the system comprises an event data acquisition module, a data processing module and a data processing module, wherein the event data acquisition module is used for acquiring event data of a client, and the event data comprises client description information, event description information and event record data;
the text description section generation module is used for generating a text description section according to the client description information and the event description information;
the event data body generation module is used for generating an event data body according to the event record data;
The log file coding module is used for coding the text description section and the event data body into a log file;
the event data body is a binary data body, and the event data body generation module comprises:
the determining submodule is used for determining the event type and the occurrence time of each event and the size of event record data;
the text head writing sub-module is used for writing the event type and the occurrence time of each event and the size of the event record data into the text head of the binary data body;
and the text body writing sub-module is used for writing the event record data of each event into the text body of the binary data body according to the sequence of the text head.
11. A storage device for log files, comprising:
the event data acquisition module is used for acquiring event data of the client;
the log file coding module is used for coding the event data into a log file;
the memory address space determining module is used for determining a memory address space;
the log file storage module is used for storing the log file into the memory address space;
the event data comprises client description information, event description information and event record data; the log file encoding module comprises:
The text description section generation sub-module is used for generating a text description section according to the client description information and the event description information;
the event data body generation sub-module is used for generating an event data body according to the event record data; the event data body is a binary data body, and the event data body generation submodule is specifically used for determining the event type and the occurrence time of each event and the size of event record data; writing the event type, the occurrence time of each event and the size of event record data into the text head of a binary data body; writing the event record data of each event into a text body of a binary data body according to the sequence of the text head;
and the log file encoding sub-module is used for encoding the text description section and the event data body into a log file.
12. A log file parsing apparatus, comprising:
the system comprises a log file acquisition module, a log file generation module and a log file generation module, wherein the log file acquisition module is used for acquiring a log file, and the log file comprises a text description section and an event data body; the log file is obtained by encoding event data of the client, wherein the event data comprises client description information, event description information and event record data; the log file is obtained by encoding event data of a client in the following way: generating a text description section according to the client description information and the event description information; generating an event data body according to the event record data; encoding the text description section and the event data body into a log file; the event data body is a binary data body, and the generating the event data body according to the event record data includes: determining the event type and the occurrence time of each event and the size of event record data; writing the event type, the occurrence time of each event and the size of event record data into the text head of a binary data body; writing the event record data of each event into a text body of a binary data body according to the sequence of the text head;
The event description information reading module is used for reading the event description information in the text description section;
the event data body analysis module is used for analyzing the event data body according to the event description information;
the event data volume analysis module comprises:
the determining submodule is used for determining an event data body corresponding to the event description information, wherein the event data body is a binary data body;
and the analysis sub-module is used for analyzing the binary data body.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810283468.4A CN110362547B (en) | 2018-04-02 | 2018-04-02 | Method and device for encoding, analyzing and storing log file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810283468.4A CN110362547B (en) | 2018-04-02 | 2018-04-02 | Method and device for encoding, analyzing and storing log file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362547A CN110362547A (en) | 2019-10-22 |
| CN110362547B true CN110362547B (en) | 2023-10-03 |
Family
ID=68213442
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810283468.4A Active CN110362547B (en) | 2018-04-02 | 2018-04-02 | Method and device for encoding, analyzing and storing log file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362547B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143823B (en) * | 2019-12-30 | 2022-07-26 | 宁波三星智能电气有限公司 | Method and device for detecting safe access event of electric meter |
| CN111367755A (en) * | 2020-02-17 | 2020-07-03 | 上海基分文化传播有限公司 | User log writing method and system of mobile terminal |
| CN111931203A (en) * | 2020-07-15 | 2020-11-13 | 深信服科技股份有限公司 | Sensitive data analysis method, device, equipment and storage medium |
| CN113423079B (en) * | 2021-06-18 | 2022-11-01 | 中国农业银行股份有限公司 | Information encoding method, apparatus, device, storage medium, and program product |
| CN116028434B (en) * | 2023-03-23 | 2023-07-07 | 中科星图测控技术股份有限公司 | File coding method and system for describing space analysis scene |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007059057A2 (en) * | 2005-11-12 | 2007-05-24 | Logrhythm, Inc | Log collection, structuring and processing |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| CN104636365A (en) * | 2013-11-11 | 2015-05-20 | 数码辰星科技发展（北京）有限公司 | Log record method based on ticket data |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| CN105260399A (en) * | 2015-09-17 | 2016-01-20 | 西安未来国际信息股份有限公司 | Method for acquiring and retrieving distributed log |
| CN106055450A (en) * | 2016-05-20 | 2016-10-26 | 北京神州绿盟信息安全科技股份有限公司 | Binary log analysis method and apparatus |
| CN106371770A (en) * | 2016-09-05 | 2017-02-01 | 杭州华为数字技术有限公司 | Data write-in method and device |
| CN107391762A (en) * | 2017-08-28 | 2017-11-24 | 京信通信系统（中国）有限公司 | The processing method and processing device of daily record data |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8832125B2 (en) * | 2010-06-14 | 2014-09-09 | Microsoft Corporation | Extensible event-driven log analysis framework |
| US8468391B2 (en) * | 2010-08-04 | 2013-06-18 | International Business Machines Corporation | Utilizing log event ontology to deliver user role specific solutions for problem determination |
| US20120246303A1 (en) * | 2011-03-23 | 2012-09-27 | LogRhythm Inc. | Log collection, structuring and processing |
| CN102981933B (en) * | 2012-12-18 | 2015-05-13 | 武汉邮电科学研究院 | Method and system for data increment backup of sensing layer of Internet of Things |
| GB2514590B (en) * | 2013-05-30 | 2016-01-06 | Keysight Technologies Singapore Holdings Pte Ltd | Method and apparatus for logging data records |
-
2018
- 2018-04-02 CN CN201810283468.4A patent/CN110362547B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007059057A2 (en) * | 2005-11-12 | 2007-05-24 | Logrhythm, Inc | Log collection, structuring and processing |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| CN104636365A (en) * | 2013-11-11 | 2015-05-20 | 数码辰星科技发展（北京）有限公司 | Log record method based on ticket data |
| CN105260399A (en) * | 2015-09-17 | 2016-01-20 | 西安未来国际信息股份有限公司 | Method for acquiring and retrieving distributed log |
| CN106055450A (en) * | 2016-05-20 | 2016-10-26 | 北京神州绿盟信息安全科技股份有限公司 | Binary log analysis method and apparatus |
| CN106371770A (en) * | 2016-09-05 | 2017-02-01 | 杭州华为数字技术有限公司 | Data write-in method and device |
| CN107391762A (en) * | 2017-08-28 | 2017-11-24 | 京信通信系统（中国）有限公司 | The processing method and processing device of daily record data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362547A (en) | 2019-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110362547B (en) | Method and device for encoding, analyzing and storing log file | |
| CN106649788B (en) | Database data transmission method and device | |
| KR20220125198A (en) | Data additional writing method, apparatus, electronic device, storage medium and computer programs | |
| CN111859205A (en) | Analytic method and equipment for batch processing tasks | |
| US20190384825A1 (en) | Method and device for data protection and computer readable storage medium | |
| CN112667246A (en) | Application function extension method and device and electronic equipment | |
| CN111314535A (en) | Method and equipment for recording quality inspection process of mobile phone | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN112436973B (en) | Automatic service experience index testing method and equipment for cloud game | |
| CN117421160A (en) | Data backup method, device, computer equipment and storage medium | |
| CN115426375A (en) | Data processing method and data processing system | |
| CN113377376A (en) | Data packet generation method, data packet generation device, electronic device, and storage medium | |
| CN114816816A (en) | Collapse stack information processing method, device, equipment and storage medium | |
| CN113507529A (en) | Method for realizing file downloading based on Web application | |
| CN112765129A (en) | Method and equipment for realizing database data version management | |
| US20170286195A1 (en) | Information object system | |
| CN111858487A (en) | Data updating method and device | |
| CN111125565A (en) | Method and equipment for inputting information in application | |
| CN112131193B (en) | Application program compression method and device | |
| CN114065079A (en) | Method and equipment for generating PDF file by Web | |
| CN114489686B (en) | Middleware decoupling method, device and equipment under multi-cloud deployment | |
| CN116248502A (en) | Method, device and system for updating user-related data local to browser | |
| CN110083520B (en) | Data acquisition method and device | |
| CN117331738A (en) | Log processing method and device based on stability problem | |
| CN118069495A (en) | Operation restoration method, system, equipment and storage medium based on user interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230731 Address after: Room 211, Floor 2, Building 5, No. 699, Wangshang Road, Changhe Street, Binjiang District, Hangzhou City, Zhejiang Province Applicant after: Hangzhou Alibaba Zhirong Digital Technology Co.,Ltd. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Applicant before: ALIBABA GROUP HOLDING Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |