CN110362547A - Coding, parsing, storage method and the device of journal file - Google Patents
Coding, parsing, storage method and the device of journal file Download PDFInfo
- Publication number
- CN110362547A CN110362547A CN201810283468.4A CN201810283468A CN110362547A CN 110362547 A CN110362547 A CN 110362547A CN 201810283468 A CN201810283468 A CN 201810283468A CN 110362547 A CN110362547 A CN 110362547A
- Authority
- CN
- China
- Prior art keywords
- event
- data
- journal file
- description information
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present application provides coding, parsing, storage method and the device of a kind of journal file, the coding method includes: to obtain the event data of client, the event data includes client description information, event description information, and, event log data；Text, which is generated, according to the client description information and event description information describes section；Event data body is generated according to the event log data；The text is described into section and event data body is encoded to journal file.The present embodiment is by providing a kind of coding protocol of journal file, so as to uniformly be encoded to the Various types of data of client according to the coding protocol, it ensure that the scalability and flexibility of data, journal file after facilitating coding is carried out in multiple and different platforms using improving the cross-platform compatibility of data.
Description
Technical field
This application involves field of computer technology, coding method, a kind of log text more particularly to a kind of journal file
The code device of part, a kind of storage method of journal file, a kind of storage device of journal file, a kind of parsing of journal file
Method and a kind of resolver of journal file.
Background technique
Client (Client) is also referred to as user terminal, refers to corresponding with server, that local service is provided for client one
Kind application program.More common client include WWW use web browser, receive Email when posting Email
Client, and, the chat software etc. for instant messaging.
User is during using client application, often because a variety of causes causes to occur such as application program
Service exceptions or the system exceptions such as collapse.In order to improve the system performance of client application, developer is usually required pair
The exception of appearance is checked, is reappeared.In the exception that investigation, reproduction occur, client application is just needed to provide as far as possible
Abundant and complete abnormal events information.
But the frequency of anomalous event generation is high, data volume is big, often since internal storage data is lost, file write-in is lost
Lose, network request failure and other reasons cause the data of anomalous event not record completely.When recording data, if data
The performance of processing is not high enough, also can cause tremendous influence to the service performance of application program.In addition, the number recorded with needs
According to the rapid expansion of amount, according to existing data storage method, a large amount of memory space of user terminal and net can be also consumed
Network flow.
Summary of the invention
In view of the above problems, it proposes the embodiment of the present application and overcomes the above problem or at least partly in order to provide one kind
A kind of coding method of the journal file to solve the above problems, a kind of code device of journal file, a kind of journal file are deposited
Method for storing, a kind of storage device of journal file, a kind of analytic method of journal file and a kind of corresponding solution of journal file
Analysis apparatus.
To solve the above-mentioned problems, this application discloses a kind of coding methods of journal file, comprising:
The event data of client is obtained, the event data includes client description information, event description information, with
And event log data；
Text, which is generated, according to the client description information and event description information describes section；
Event data body is generated according to the event log data；
The text is described into section and event data body is encoded to journal file.
To solve the above-mentioned problems, this application discloses a kind of storage methods of journal file, comprising:
Obtain the event data of client；
The event data is encoded to journal file；
Determine memory address space；
The journal file is stored to the memory address space.
To solve the above-mentioned problems, this application discloses a kind of analytic methods of journal file, comprising:
Journal file is obtained, the journal file includes that text describes section and event data body；
It reads the text and describes event description information in section；
The event data body is parsed according to the event description information.
To solve the above-mentioned problems, this application discloses a kind of code devices of journal file, comprising:
Event data obtains module, and for obtaining the event data of client, the event data includes client description
Information, event description information, and, event log data；
Text describes section generation module, retouches for generating text according to the client description information and event description information
State section；
Event data body generation module, for generating event data body according to the event log data；
Journal file coding module, for the text being described section and event data body is encoded to journal file.
To solve the above-mentioned problems, this application discloses a kind of storage devices of journal file, comprising:
Event data obtains module, for obtaining the event data of client；
Journal file coding module, for the event data to be encoded to journal file；
Memory address space determining module, for determining memory address space；
Journal file memory module, for storing the journal file to the memory address space.
To solve the above-mentioned problems, this application discloses a kind of resolvers of journal file, comprising:
Journal file obtains module, and for obtaining journal file, the journal file includes that text describes section and event number
According to body；
Event description information read module describes event description information in section for reading the text；
Event data body parsing module, for being parsed according to the event description information to the event data body.
Compared with the background art, the embodiment of the present application includes the following advantages:
The embodiment of the present application, by obtaining the event data of client, and according to the client description in the event data
Information and event description information generate text and describe section, generate event data body according to event log data, so as to will be literary
This describes section and event data body is encoded to journal file.The present embodiment by providing a kind of coding protocol of journal file, from
And uniformly the Various types of data of client can be encoded according to the coding protocol, on compression ratio, the volume of the present embodiment
Code agreement improves 5 times or more than very widely used today json data protocol, solves the data of client in the prior art
Expansion is influenced caused by equipment or client.Meanwhile the present embodiment is by by type, data structure and coding in event data
Data file is written by the description information of stream data body in information, can pass through stream data for the data of any platform
The meaning and format that each record type represents in body carry out data parsing, since data file inherently has to data volume
It explains information, greatly ensure that during carrying out data parsing just without relying on the other information except data in this way
The scalability and flexibility of data, the journal file after facilitating coding carry out using raising in multiple and different platforms
The cross-platform compatibility of data dramatically reduces data and frequently changes, extends and give data parsing bring complexity.
Detailed description of the invention
Fig. 1 is a kind of step flow chart of the coding method embodiment of journal file of the application；
Fig. 2 is a kind of schematic diagram of binary data body of the application；
Fig. 3 is a kind of coding structure figure of journal file of the application；
Fig. 4 is a kind of step flow chart of the storage method embodiment of journal file of the application；
Fig. 5 is a kind of design frame chart of the storage method of journal file of the application；
Fig. 6 is a kind of step flow chart of the analytic method embodiment of journal file of the application；
Fig. 7 is a kind of schematic diagram of the analytic method of journal file of the application；
Fig. 8 is a kind of structural block diagram of the code device embodiment of journal file of the application；
Fig. 9 is a kind of structural block diagram of the memory device embodiment of journal file of the application；
Figure 10 is a kind of structural block diagram of the resolver embodiment of journal file of the application.
Specific embodiment
In order to make the above objects, features, and advantages of the present application more apparent, with reference to the accompanying drawing and it is specific real
Applying mode, the present application will be further described in detail.
Referring to Fig.1, a kind of step flow chart of the coding method embodiment of journal file of the application is shown, specifically may be used
To include the following steps:
Step 101, the event data of client is obtained, the event data includes client description information, event description
Information, and, event log data；
It should be noted that client can refer to the application being mounted on all kinds of terminal devices such as mobile phone, tablet computer
Program.For example, mobile phone Taobao APP, the Alipay APP etc. that are mounted on mobile phone.The present embodiment is to terminal device and client
Type be not construed as limiting.
In the embodiment of the present application, event data may include client description information, event description information, and, thing
Part records data etc..
Wherein, client description information may include client-side information (i.e. the information of client or application program itself),
Facility information (installs the information of the terminal device of the client or application program), and, (i.e. user is using environmental information
The information such as the network state when client or application program).
In the concrete realization, client-side information may include at least one of following various information:
Protocol version, application version number, equipment identification information.
Facility information may include at least one of following various information:
The sensor that device hardware type, CPU core number, CPU frequency, parameters of display, operating system version, equipment are supported
Type.
Environmental information may include at least one of following various information:
Device network state, operator's informaiton.
Certainly, the above is only a kind of examples, difference according to actual needs, and above-mentioned various information can also be including other not
Same information or data, the present embodiment are not construed as limiting this.
In the embodiment of the present application, event description information may include the event of the corresponding event log data of each event
Information and format information.
The thing that each event can be the event of client itself generation or user is generated when using the client
Part.For example, system event, user operation case, application program execute event, network event etc..
Event information can be the relevant information or data of the event recorded when above-mentioned all kinds of events occur, format
Information then illustrates the information of used record format or parameter when recording above-mentioned event.
In the embodiment of the present application, event log data can be completely record occur above-mentioned all kinds of events when letter
Breath or data.
Step 102, text is generated according to the client description information and event description information describe section；
In the embodiment of the present application, log text can be generated according to the client description information and event description information of acquisition
The text of part describes section.
Therefore, it may include client-side information (protocol version, application version that the text of a journal file, which describes section,
Number, equipment identification information etc.), facility information (device hardware type, CPU core number, CPU frequency, parameters of display, operating system
Sensor type etc. that version, equipment are supported), environmental information (device network state, operator's informaiton etc.), and, event is retouched
State information.
Step 103, event data body is generated according to the event log data；
In the embodiment of the present application, event data body can be binary data body, which can be serial
Record each event of client or application program.
In the concrete realization, binary data body may include a plurality of record, and every record may include having text head
Header and body of text body records the event that client or application program occur respectively.
When generating binary data body, event type, the time of origin of each event can be determined first, and, thing
Part records the size of data, and respectively by event type, the time of origin of above-mentioned each event, and, event log data
The text head of size write-in binary data body.Then, according to the sequence of text head, the event log data of each event is write
Enter the body of text of binary data body, to obtain binary data body.
As shown in Fig. 2, a kind of schematic diagram of binary data body of the application.In fig. 2 it is possible to use 2 byte records
Event type (record type) uses 4 byte records data volume sizes using 4 byte records Time To Events (time)
(body size), then, record meet the body of text body of corresponding data body size.
In the embodiment of the present application, event data body can also include binary data head, which can
To have fixed byte length, for recording all kinds of attribute informations of corresponding event data body.
In the concrete realization, when generating binary data head, the attribute information of event data body can be determined first, on
State attribute information may include agreement evil spirit number Magic number, protocol version, data volume generate the time, and, time deviation
The information such as value.It is then possible to above-mentioned agreement evil spirit number Magic number, protocol version, data volume are generated into the time, and/or, when
Between the information such as deviation binary data head is written.
It is, for example, possible to use 4 byte records agreement evil spirit number Magic number, use the length of 2 byte records binary system heads
Remainder bytes at the beginning of using 8 byte records log recordings, are then denoted as 0 by degree.
Certainly, the information that binary data head is written can also include other information, for example, whether file overflows etc.,
The present embodiment is not construed as limiting this.
Step 104, the text is described into section and event data body is encoded to journal file.
In the embodiment of the present application, the event data of client is being obtained, and is obtaining the journal file to be generated accordingly
After text describes section and event data body, this document can be described to section and event data body is encoded to journal file, so as to right
Above-mentioned journal file is stored.
As shown in figure 3, being a kind of coding structure figure of journal file of the application.In Fig. 3, including text describes section
Text header, binary data head binary header and binary data body.Wherein, text describes section text
Header includes the information such as data self-description information type-descriptors, and binary data head binary header is again
It further comprise the length header length of agreement evil spirit number Magic number, binary system head, at the beginning of log recording
The information such as record start time.And for binary data body, each section and binary data class shown in Fig. 2
Seemingly.
Data self-description information type-descriptors record has server-side that should interpret according to which kind of rule or format
The data.
In general, server-side reports client according to the format of agreement in the interactive process of client and server-side
Data are parsed.But the content of data, format often change, for example, when version iteration, clothes occur for client
Business end, to need again to revise above-mentioned agreement, cannot be raised compatible with the data of different editions or format
Communications cost between the two, developer's maintenance are got up also more troublesome.The present embodiment passes through to mitigate maintenance load
In the cataloged procedure of journal file, section is described into data self-description information write-in text, so that data inherently have logarithm
According to the explanation information of body, so just without relying on the other information except data during carrying out data parsing, greatly
It ensure that the scalability and flexibility of data.
In the embodiment of the present application, by obtaining the event data of client, and according to the client in the event data
Description information and event description information generate text and describe section, generate event data body according to event log data, so as to
Text is described into section and event data body is encoded to journal file.The present embodiment is assisted by providing a kind of coding of journal file
View, so as to uniformly be encoded to the Various types of data of client according to the coding protocol, on compression ratio, the present embodiment
Coding protocol improve 5 times or more than very widely used today json data protocol, solve client in the prior art
Data expansion is influenced caused by equipment or client.Meanwhile the present embodiment by by type in event data, data structure and
Data file is written by the description information of stream data body in encoded information, can pass through streaming for the data of any platform
The meaning and format that each record type represents in data volume carry out data parsing, since data file inherently has to data
The explanation information of body is greatly protected during carrying out data parsing just without relying on the other information except data in this way
The scalability and flexibility for having demonstrate,proved data, facilitate coding after journal file carried out in multiple and different platforms using,
The cross-platform compatibility for improving data dramatically reduces data and frequently changes, extends and give data parsing bring complexity.
Referring to Fig. 4, a kind of step flow chart of the storage method embodiment of journal file of the application is shown, specifically may be used
To include the following steps:
Step 401, the event data of client is obtained；
It should be noted that client can refer to the application being mounted on all kinds of terminal devices such as mobile phone, tablet computer
Program.For example, mobile phone Taobao APP, the Alipay APP etc. that are mounted on mobile phone.The present embodiment is to terminal device and client
Type be not construed as limiting.
In the embodiment of the present application, event data may include client description information, event description information, and, thing
Part records data etc..
Wherein, client description information may include client-side information (i.e. the information of client or application program itself),
Facility information (installs the information of the terminal device of the client or application program), and, (i.e. user is using environmental information
The information such as the network state when client or application program).
In the concrete realization, client-side information may include at least one of following various information:
Protocol version, application version number, equipment identification information.
Facility information may include at least one of following various information:
The sensor that device hardware type, CPU core number, CPU frequency, parameters of display, operating system version, equipment are supported
Type.
Environmental information may include at least one of following various information:
Device network state, operator's informaiton.
Certainly, the above is only a kind of examples, difference according to actual needs, and above-mentioned various information can also be including other not
Same information or data, the present embodiment are not construed as limiting this.
In the embodiment of the present application, event description information may include the event of the corresponding event log data of each event
Information and format information.
The thing that each event can be the event of client itself generation or user is generated when using the client
Part.For example, system event, user operation case, application program execute event, network event etc..
Event information can be the relevant information or data of the event recorded when above-mentioned all kinds of events occur, format
Information then illustrates the information of used record format or parameter when recording above-mentioned event.
In the embodiment of the present application, event log data can be completely record occur above-mentioned all kinds of events when letter
Breath or data.
Step 402, the event data is encoded to journal file；
In the embodiment of the present application, when event data is encoded to journal file, client description can be first depending on
Information and event description information generate text and describe section, event data body are generated according to event log data, then by above-mentioned text
This describes section and event data body is encoded to journal file.
It is similar to step 104 with step 102 in above-described embodiment by step 402 in this present embodiment, can mutually refering to,
The present embodiment repeats no more this.
Step 403, memory address space is determined；
In the embodiment of the present application, memory address space can refer to that user stores the address sky of the journal file generated
Between, size which can fix.
It in the concrete realization, can also be using presetting method to the memory address space after determining memory address space
It is initialized.For example, mmap method can be used.
Mmap is a kind of method of Memory Mapping File, i.e., a file or other objects are mapped to the address of process
The one-to-one relationship of one section of virtual address in file disk address and process virtual address space is realized in space.It realizes in this way
Mapping relations after, process can by the way of pointer this section of memory of read-write operation, and system can the automatic write-back page
Onto corresponding file disk, that is, the operation to file is completed without recalling the systems such as read, write and calls function.
On the contrary, memory headroom also directly reflects user's space to the modification in this section of region, so as to realize the file between different processes
It is shared.
Step 404, the journal file is stored to the memory address space.
In the embodiment of the present application, when storing journal file to memory address space, the memory can be determined first
Whether address space overflows, if so, the data in the available memory address space, and disk is written into above-mentioned data；
If it is not, the step of can then executing storage journal file to memory address space.
In the concrete realization, the memory address space size mapped using mmap is limited, when corresponding space is used big
It is small be more than threshold value after, sub thread can by the data copy stored in the memory address space of mapping be written disk file.
As shown in figure 5, being a kind of design frame chart of the storage method of journal file of the application.In Fig. 5, for needing
(such as system event, user operation case, application program execute event, network event to the event data of the client of storage
Deng), client can be initialized first, then the memory address space determining using the initialization of mmap method, and
After the text of generation journal file describes section and event data body, by checking whether memory address space overflows, to realize
Storage to journal file.If memory address space overflows, can be by sub thread, by the number in the memory address space
According to write-in disk, pointer is then moved back to initial value.If memory address space does not overflow, then it represents that the address can also store
The memory address space can be then written in journal file by journal file.
In the embodiment of the present application, by obtaining the event data of client, which can be encoded to log
File can store above-mentioned journal file to the memory address space then after determining memory address space.The present embodiment can
To use mmap method that journal file is mapped to the memory address space of fixed size, by operation memory and by system kernel
Data are synchronized to journal file, can be lost to avoid direct operation file bring performance；Client is occurring or is answering simultaneously
When with extreme feelings such as program crashings, even if consumer process is killed, still the data in mmap address space can be written for subprocess
File, to utmostly ensure that the data integrity of journal file.
Referring to Fig. 6, a kind of step flow chart of the analytic method embodiment of journal file of the application is shown, specifically may be used
To include the following steps:
Step 601, journal file is obtained, the journal file includes that text describes section and event data body；
It should be noted that the journal file in the present embodiment can refer to according to above-mentioned journal file coding method or
The journal file that coding protocol shown in the storage method of journal file is encoded or stored, every part of journal file can
To include that text describes section and event data body.
Step 602, it reads the text and describes event description information in section；
In the embodiment of the present application, event description information may include the event of the corresponding event log data of each event
Information and format information.
Each event can be client itself generation event or user generated when using the client when
Between.For example, system event, user operation case, application program execute event, network event etc..
Event information can be the relevant information or data of the event recorded when above-mentioned all kinds of events occur, format
Information then illustrates the information of used record format or parameter when recording above-mentioned event.
For example, for opening the page (openPage) event, corresponding event description information can be indicated are as follows:
openpage:u4:u1*,freeMemory:f,residentMemory:f,virtualMemory:f,cpuUsa
geOfApp:f,cpuUsageOfDevice:f
Step 603, the event data body is parsed according to the event description information.
In the embodiment of the present application, after reading the event description information that text describes in section, reading can be determined first
The corresponding event data body of the event description information taken, then again parses event data body.
In the concrete realization, since event data body includes binary data head and binary data body.Therefore, it is parsing
When, binary data head can be read first, obtain agreement evil spirit number Magic number of the event data body, protocol version,
Data volume generates the time, and, the attribute informations such as reporting.Then, then to binary data body parse.
As shown in fig. 7, being a kind of schematic diagram of the analytic method of journal file of the application, 10 words can be read first
Section, wherein 1-2 byte resolves to short event type (openPage in Fig. 7), and is described in section according to event type from text
Corresponding event description information is searched, to determine which kind of data format parsing current event with；3-6 byte resolves to integer, makees
The time occurred for event；7-10 byte resolves to integer, is recorded as event body length.
Then, event body is parsed according to event description information.By taking openPage event as an example: reading 4 bytes and resolve to
Unsigned int, and continue to read the corresponding length byte of integer value and resolve to character string, it is recorded as page；Read 4 bytes
Float type is resolved to, freeMemory is recorded as；It reads 4 bytes and resolves to float type, be recorded as virtualMemory, directly
To having parsed other whole fields shown in fig. 7.
It should be noted that for simple description, therefore, it is stated as a series of action groups for embodiment of the method
It closes, but those skilled in the art should understand that, the embodiment of the present application is not limited by the described action sequence, because according to
According to the embodiment of the present application, some steps may be performed in other sequences or simultaneously.Secondly, those skilled in the art also should
Know, the embodiments described in the specification are all preferred embodiments, and related movement not necessarily the application is implemented
Necessary to example.
Referring to Fig. 8, a kind of structural block diagram of the code device embodiment of journal file of the application is shown, it specifically can be with
Including following module:
Event data obtains module 801, and for obtaining the event data of client, the event data includes that client is retouched
Information, event description information are stated, and, event log data；
Text describes section generation module 802, for generating text according to the client description information and event description information
This describes section；
Event data body generation module 803, for generating event data body according to the event log data；
Journal file coding module 804, for the text being described section and event data body is encoded to journal file.
In the embodiment of the present application, the client description information may include client-side information, facility information, and,
Environmental information；
The event description information may include the event information and format letter of the corresponding event log data of each event
Breath.
In the embodiment of the present application, the client-side information may include following at least one:
Protocol version, application version number, equipment identification information；
The facility information may include following at least one:
The sensor that device hardware type, CPU core number, CPU frequency, parameters of display, operating system version, equipment are supported
Type；
The environmental information may include following at least one:
Device network state, operator's informaiton.
In the embodiment of the present application, the event data body can be binary data body, and the event data body generates
Module 803 can specifically include following submodule:
Determine submodule, for determining event type, the time of origin of each event, and, event log data it is big
It is small；
Submodule is written in text head, for respectively by event type, the time of origin of each event, and, event
Record the text head of the size write-in binary data body of data；
Submodule is written in body of text, for the sequence according to the text head, by the logout number of each event
According to the body of text of write-in binary data body.
In the embodiment of the present application, the event data body can also include binary data head, the event data body
Generation module 803 can also include following submodule:
Attribute information determines submodule, and for determining the attribute information of the event data body, the attribute information can be with
The time is generated including agreement evil spirit number, protocol version, data volume, and, reporting；
Submodule is written in binary data head, is used for the agreement evil spirit number, protocol version, data volume generation time, and/
Or, binary data head is written in reporting, the binary data head has fixed byte length.
Referring to Fig. 9, a kind of structural block diagram of the memory device embodiment of journal file of the application is shown, it specifically can be with
Including following module:
Event data obtains module 901, for obtaining the event data of client；
Journal file coding module 902, for the event data to be encoded to journal file；
Memory address space determining module 903, for determining memory address space；
Journal file memory module 904, for storing the journal file to the memory address space.
In the embodiment of the present application, the event data may include client description information, event description information, with
And event log data；The journal file coding module 902 can specifically include following submodule:
Text describes section and generates submodule, for generating text according to the client description information and event description information
Section is described；
Event data body generates submodule, for generating event data body according to the event log data；
Journal file encoding submodule, for the text being described section and event data body is encoded to journal file.
In the embodiment of the present application, described device can also include following module:
Initialization module, for being initialized using presetting method to the memory address space, the presetting method
It may include mmap method.
In the embodiment of the present application, the journal file memory module 904 can specifically include following submodule:
Submodule is determined, for determining whether the memory address space overflows；
Submodule is written, for obtaining the data in the memory address space when the memory address space overflows,
Disk is written into the data；
Sub-module stored, for calling the journal file memory module when the memory address space does not overflow
904。
Referring to Fig.1 0, a kind of structural block diagram of the resolver embodiment of journal file of the application is shown, specifically may be used
To include following module:
Journal file obtains module 1001, and for obtaining journal file, the journal file may include that text describes section
With event data body；
Event description information read module 1002 describes event description information in section for reading the text；
Event data body parsing module 1003, for being solved according to the event description information to the event data body
Analysis.
In the embodiment of the present application, the event data body parsing module 1003 can specifically include following submodule:
Determine submodule, for determining the corresponding event data body of the event description information, the event data body can
Think binary data body；
Analyzing sub-module, for being parsed to the binary event data volume.
In the embodiment of the present application, the event data body can also include binary data head, the event data body
Parsing module 1003 can also include following submodule:
Reading submodule obtains the attribute information of the event data body for reading the binary data head, described
Attribute information may include agreement evil spirit number, protocol version, data volume generate the time, and, reporting.
For device embodiment, since it is basically similar to the method embodiment, related so being described relatively simple
Place illustrates referring to the part of embodiment of the method.
All the embodiments in this specification are described in a progressive manner, the highlights of each of the examples are with
The difference of other embodiments, the same or similar parts between the embodiments can be referred to each other.
It should be understood by those skilled in the art that, the embodiments of the present application may be provided as method, apparatus or calculating
Machine program product.Therefore, the embodiment of the present application can be used complete hardware embodiment, complete software embodiment or combine software and
The form of the embodiment of hardware aspect.Moreover, the embodiment of the present application can be used one or more wherein include computer can
With in the computer-usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) of program code
The form of the computer program product of implementation.
In a typical configuration, the computer equipment includes one or more processors (CPU), input/output
Interface, network interface and memory.Memory may include the non-volatile memory in computer-readable medium, random access memory
The forms such as device (RAM) and/or Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is to calculate
The example of machine readable medium.Computer-readable medium includes that permanent and non-permanent, removable and non-removable media can be with
Realize that information is stored by any method or technique.Information can be computer readable instructions, data structure, the module of program or
Other data.The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory
(SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only memory
(ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory techniques, CD-ROM are read-only
Memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or
Other magnetic storage devices or any other non-transmission medium, can be used for storage can be accessed by a computing device information.According to
Herein defines, and computer-readable medium does not include non-persistent computer readable media (transitory media), such as
The data-signal and carrier wave of modulation.
The embodiment of the present application is referring to according to the method for the embodiment of the present application, terminal device (system) and computer program
The flowchart and/or the block diagram of product describes.It should be understood that flowchart and/or the block diagram can be realized by computer program instructions
In each flow and/or block and flowchart and/or the block diagram in process and/or box combination.It can provide these
Computer program instructions are set to general purpose computer, special purpose computer, Embedded Processor or other programmable data processing terminals
Standby processor is to generate a machine, so that being held by the processor of computer or other programmable data processing terminal devices
Capable instruction generates for realizing in one or more flows of the flowchart and/or one or more blocks of the block diagram
The device of specified function.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing terminal devices
In computer-readable memory operate in a specific manner, so that instruction stored in the computer readable memory generates packet
The manufacture of command device is included, which realizes in one side of one or more flows of the flowchart and/or block diagram
The function of being specified in frame or multiple boxes.
These computer program instructions can also be loaded into computer or other programmable data processing terminal devices, so that
Series of operation steps are executed on computer or other programmable terminal equipments to generate computer implemented processing, thus
The instruction executed on computer or other programmable terminal equipments is provided for realizing in one or more flows of the flowchart
And/or in one or more blocks of the block diagram specify function the step of.
Although preferred embodiments of the embodiments of the present application have been described, once a person skilled in the art knows bases
This creative concept, then additional changes and modifications can be made to these embodiments.So the following claims are intended to be interpreted as
Including preferred embodiment and all change and modification within the scope of the embodiments of the present application.
Finally, it is to be noted that, herein, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that process, method, article or terminal device including a series of elements not only wrap
Those elements are included, but also including other elements that are not explicitly listed, or further includes for this process, method, article
Or the element that terminal device is intrinsic.In the absence of more restrictions, being wanted by what sentence "including a ..." limited
Element, it is not excluded that there is also other identical elements in process, method, article or the terminal device for including the element.
Above to a kind of coding method of journal file provided herein, a kind of code device of journal file, one
Storage method, a kind of storage device of journal file, a kind of analytic method of journal file and a kind of log of kind journal file
The resolver of file, is described in detail, specific case used herein to the principle and embodiment of the application into
Elaboration is gone, the description of the example is only used to help understand the method for the present application and its core ideas；Meanwhile for this
The those skilled in the art in field, according to the thought of the application, there will be changes in the specific implementation manner and application range,
In conclusion the contents of this specification should not be construed as limiting the present application.
Claims (15)
1. a kind of coding method of journal file characterized by comprising
The event data of client is obtained, the event data includes client description information, event description information, and, thing
Part records data；
Text, which is generated, according to the client description information and event description information describes section；
Event data body is generated according to the event log data；
The text is described into section and event data body is encoded to journal file.
2. the method according to claim 1, wherein the client description information includes client-side information, sets
Standby information, and, environmental information；
The event description information includes the event information and format information of the corresponding event log data of each event.
3. according to the method described in claim 2, it is characterized in that, the client-side information includes following at least one:
Protocol version, application version number, equipment identification information；
The facility information includes following at least one:
The sensor class that device hardware type, CPU core number, CPU frequency, parameters of display, operating system version, equipment are supported
Type；
The environmental information includes following at least one:
Device network state, operator's informaiton.
4. the method according to claim 1, wherein the event data body be binary data body, it is described according to
Event data body is generated according to the event log data, comprising:
Determine event type, the time of origin of each event, and, the size of event log data；
Respectively by event type, the time of origin of each event, and, binary number is written in the size of event log data
According to the text head of body；
According to the sequence of the text head, by the text of the event log data write-in binary data body of each event
Body.
5. according to the method described in claim 4, it is characterized in that, the event data body further includes binary data head, institute
State method further include:
Determine that the attribute information of the event data body, the attribute information include agreement evil spirit number, protocol version, data volume generation
Time, and, reporting；
The agreement evil spirit number, protocol version, data volume are generated into the time, and/or, binary data head is written in reporting,
The binary data head has fixed byte length.
6. a kind of storage method of journal file characterized by comprising
Obtain the event data of client；
The event data is encoded to journal file；
Determine memory address space；
The journal file is stored to the memory address space.
7. according to the method described in claim 6, it is characterized in that, the event data includes client description information, event
Description information, and, event log data；It is described that the event data is encoded to journal file, comprising:
Text, which is generated, according to the client description information and event description information describes section；
Event data body is generated according to the event log data；
The text is described into section and event data body is encoded to journal file.
8. according to the method described in claim 6, it is characterized in that, after the determining memory address space, further includes:
The memory address space is initialized using presetting method, the presetting method includes mmap method.
9. according to the method described in claim 6, it is characterized in that, the storage journal file is empty to the memory address
Between, comprising:
Determine whether the memory address space overflows；
If so, obtaining the data in the memory address space, disk is written into the data；
If it is not, the step of then executing the storage journal file to the memory address space.
10. a kind of analytic method of journal file characterized by comprising
Journal file is obtained, the journal file includes that text describes section and event data body；
It reads the text and describes event description information in section；
The event data body is parsed according to the event description information.
11. according to the method described in claim 10, it is characterized in that, it is described according to the event description information to the event
Data volume is parsed, comprising:
Determine that the corresponding event data body of the event description information, the event data body are binary data body；
The binary event data volume is parsed.
12. according to the method for claim 11, which is characterized in that the event data body further includes binary data head,
The method also includes:
The binary data head is read, the attribute information of the event data body is obtained, the attribute information includes agreement evil spirit
Number, protocol version, data volume generate the time, and, reporting.
13. a kind of code device of journal file characterized by comprising
Event data obtains module, for obtaining the event data of client, the event data include client description information,
Event description information, and, event log data；
Text describes section generation module, for generating text description according to the client description information and event description information
Section；
Event data body generation module, for generating event data body according to the event log data；
Journal file coding module, for the text being described section and event data body is encoded to journal file.
14. a kind of storage device of journal file characterized by comprising
Event data obtains module, for obtaining the event data of client；
Journal file coding module, for the event data to be encoded to journal file；
Memory address space determining module, for determining memory address space；
Journal file memory module, for storing the journal file to the memory address space.
15. a kind of resolver of journal file characterized by comprising
Journal file obtains module, and for obtaining journal file, the journal file includes that text describes section and event data body；
Event description information read module describes event description information in section for reading the text；
Event data body parsing module, for being parsed according to the event description information to the event data body.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810283468.4A CN110362547B (en) | 2018-04-02 | 2018-04-02 | Method and device for encoding, analyzing and storing log file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810283468.4A CN110362547B (en) | 2018-04-02 | 2018-04-02 | Method and device for encoding, analyzing and storing log file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362547A true CN110362547A (en) | 2019-10-22 |
| CN110362547B CN110362547B (en) | 2023-10-03 |
Family
ID=68213442
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810283468.4A Active CN110362547B (en) | 2018-04-02 | 2018-04-02 | Method and device for encoding, analyzing and storing log file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362547B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143823A (en) * | 2019-12-30 | 2020-05-12 | 宁波三星智能电气有限公司 | Method and device for detecting safe access event of electric meter |
| CN111367755A (en) * | 2020-02-17 | 2020-07-03 | 上海基分文化传播有限公司 | User log writing method and system of mobile terminal |
| CN111931203A (en) * | 2020-07-15 | 2020-11-13 | 深信服科技股份有限公司 | Sensitive data analysis method, device, equipment and storage medium |
| CN113423079A (en) * | 2021-06-18 | 2021-09-21 | 中国农业银行股份有限公司 | Information encoding method, apparatus, device, storage medium, and program product |
| CN116028434A (en) * | 2023-03-23 | 2023-04-28 | 中科星图测控技术股份有限公司 | File coding method and system for describing space analysis scene |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007059057A2 (en) * | 2005-11-12 | 2007-05-24 | Logrhythm, Inc | Log collection, structuring and processing |
| US20110307502A1 (en) * | 2010-06-14 | 2011-12-15 | Microsoft Corporation | Extensible event-driven log analysis framework |
| US20120036397A1 (en) * | 2010-08-04 | 2012-02-09 | International Business Machines Corporation | Utilizing log event ontology to deliver user role specific solutions for problem determination |
| US20120246303A1 (en) * | 2011-03-23 | 2012-09-27 | LogRhythm Inc. | Log collection, structuring and processing |
| CN102981933A (en) * | 2012-12-18 | 2013-03-20 | 武汉邮电科学研究院 | Method and system for data increment backup of sensing layer of Internet of Things |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| CN104636365A (en) * | 2013-11-11 | 2015-05-20 | 数码辰星科技发展（北京）有限公司 | Log record method based on ticket data |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| CN105260399A (en) * | 2015-09-17 | 2016-01-20 | 西安未来国际信息股份有限公司 | Method for acquiring and retrieving distributed log |
| US20160110359A1 (en) * | 2013-05-30 | 2016-04-21 | Keysight Technologies Singapore (Holdings) Pte. Ltd. | Method and Apparatus for Logging Data Records |
| CN106055450A (en) * | 2016-05-20 | 2016-10-26 | 北京神州绿盟信息安全科技股份有限公司 | Binary log analysis method and apparatus |
| CN106371770A (en) * | 2016-09-05 | 2017-02-01 | 杭州华为数字技术有限公司 | Data write-in method and device |
| CN107391762A (en) * | 2017-08-28 | 2017-11-24 | 京信通信系统（中国）有限公司 | The processing method and processing device of daily record data |
-
2018
- 2018-04-02 CN CN201810283468.4A patent/CN110362547B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007059057A2 (en) * | 2005-11-12 | 2007-05-24 | Logrhythm, Inc | Log collection, structuring and processing |
| US20110307502A1 (en) * | 2010-06-14 | 2011-12-15 | Microsoft Corporation | Extensible event-driven log analysis framework |
| US20120036397A1 (en) * | 2010-08-04 | 2012-02-09 | International Business Machines Corporation | Utilizing log event ontology to deliver user role specific solutions for problem determination |
| US20120246303A1 (en) * | 2011-03-23 | 2012-09-27 | LogRhythm Inc. | Log collection, structuring and processing |
| CN103309767A (en) * | 2012-03-08 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Method and device for processing client log |
| CN102981933A (en) * | 2012-12-18 | 2013-03-20 | 武汉邮电科学研究院 | Method and system for data increment backup of sensing layer of Internet of Things |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| US20160110359A1 (en) * | 2013-05-30 | 2016-04-21 | Keysight Technologies Singapore (Holdings) Pte. Ltd. | Method and Apparatus for Logging Data Records |
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| CN104636365A (en) * | 2013-11-11 | 2015-05-20 | 数码辰星科技发展（北京）有限公司 | Log record method based on ticket data |
| CN105260399A (en) * | 2015-09-17 | 2016-01-20 | 西安未来国际信息股份有限公司 | Method for acquiring and retrieving distributed log |
| CN106055450A (en) * | 2016-05-20 | 2016-10-26 | 北京神州绿盟信息安全科技股份有限公司 | Binary log analysis method and apparatus |
| CN106371770A (en) * | 2016-09-05 | 2017-02-01 | 杭州华为数字技术有限公司 | Data write-in method and device |
| CN107391762A (en) * | 2017-08-28 | 2017-11-24 | 京信通信系统（中国）有限公司 | The processing method and processing device of daily record data |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143823A (en) * | 2019-12-30 | 2020-05-12 | 宁波三星智能电气有限公司 | Method and device for detecting safe access event of electric meter |
| CN111143823B (en) * | 2019-12-30 | 2022-07-26 | 宁波三星智能电气有限公司 | Method and device for detecting safe access event of electric meter |
| CN111367755A (en) * | 2020-02-17 | 2020-07-03 | 上海基分文化传播有限公司 | User log writing method and system of mobile terminal |
| CN111931203A (en) * | 2020-07-15 | 2020-11-13 | 深信服科技股份有限公司 | Sensitive data analysis method, device, equipment and storage medium |
| CN113423079A (en) * | 2021-06-18 | 2021-09-21 | 中国农业银行股份有限公司 | Information encoding method, apparatus, device, storage medium, and program product |
| CN116028434A (en) * | 2023-03-23 | 2023-04-28 | 中科星图测控技术股份有限公司 | File coding method and system for describing space analysis scene |
| CN116028434B (en) * | 2023-03-23 | 2023-07-07 | 中科星图测控技术股份有限公司 | File coding method and system for describing space analysis scene |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362547B (en) | 2023-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110362547A (en) | Coding, parsing, storage method and the device of journal file | |
| CN107122296B (en) | Method and apparatus for data assertion for test interface | |
| US9183072B1 (en) | Error troubleshooting using a correlated knowledge base | |
| CN107092535B (en) | Method and apparatus for data storage of test interface | |
| CN107479868B (en) | Interface loading method, device and equipment | |
| EP3812917A1 (en) | Data structure reading method and apparatus, data structure updating method and apparatus, and electronic device | |
| CN105791178A (en) | Message assembly method and device | |
| CN106411970A (en) | Fault handling method, device and system based on service call | |
| CN109885577A (en) | Data processing method, device, terminal and storage medium | |
| CN107454138A (en) | The implementation method and device that business degrades | |
| CN112579118A (en) | Method, device, system, medium and equipment for updating configuration information of microservice | |
| CN113253999B (en) | Plug-in data access method, data source management system and interface access method | |
| CN114860473A (en) | Message processing method, processor and storage medium | |
| CN110806913A (en) | Webpage screenshot method, device and equipment | |
| CN112052157B (en) | Method, device and system for constructing test message | |
| CN111124883A (en) | Test case library introduction method, system and equipment based on tree form | |
| CN116628773A (en) | Data processing method, device, electronic equipment and storage medium | |
| US9230022B1 (en) | Customizable result sets for application program interfaces | |
| CN108563514A (en) | Access method, application and the electronic equipment of application configuration service | |
| CN113377376A (en) | Data packet generation method, data packet generation device, electronic device, and storage medium | |
| CN108734149B (en) | Text data scanning method and device | |
| CN107608947B (en) | HTML file processing method and device and electronic equipment | |
| US20170286195A1 (en) | Information object system | |
| CN108089927B (en) | Method and device for realizing message communication based on Web Worker | |
| KR20170102902A (en) | Information presentation method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230731 Address after: Room 211, Floor 2, Building 5, No. 699, Wangshang Road, Changhe Street, Binjiang District, Hangzhou City, Zhejiang Province Applicant after: Hangzhou Alibaba Zhirong Digital Technology Co.,Ltd. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Applicant before: ALIBABA GROUP HOLDING Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |