CN112286896A - Log file processing method and device, computer equipment and storage medium - Google Patents

Log file processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN112286896A
CN112286896A CN202011303567.8A CN202011303567A CN112286896A CN 112286896 A CN112286896 A CN 112286896A CN 202011303567 A CN202011303567 A CN 202011303567A CN 112286896 A CN112286896 A CN 112286896A
Authority
CN
China
Prior art keywords
log
statement
identifier
log file
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011303567.8A
Other languages
Chinese (zh)
Other versions
CN112286896B (en
Inventor
吴子敬
陈雪龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baijia Technology Group Co ltd
Original Assignee
Beijing Baijia Technology Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baijia Technology Group Co ltd filed Critical Beijing Baijia Technology Group Co ltd
Priority to CN202011303567.8A priority Critical patent/CN112286896B/en
Publication of CN112286896A publication Critical patent/CN112286896A/en
Application granted granted Critical
Publication of CN112286896B publication Critical patent/CN112286896B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/602Providing cryptographic facilities or services
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/62Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2107File encryption
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The disclosure provides a log file processing method, a log file processing device, a computer device and a storage medium, wherein the method comprises the following steps: acquiring log statements in a log file, wherein the log statements comprise fixed data parts and variable data parts; replacing the log statement in the log file with the variable data portion and the identifier of the log statement; storing the fixed data part and the identifier of the log statement in a log statement mapping table; the log statement mapping table is used for restoring the variable data part in the replaced log file and the identifier of the log statement into the log statement. The method for replacing the log statement in the log file by the variable data part and the identifier realizes compression and encryption of the log file, reduces the size of the log file, and ensures the safety of the log file.

Description

Log file processing method and device, computer equipment and storage medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a log file, a computer device, and a storage medium.
Background
The log file is used as an important basis for storing the user information, the log file which is recorded by acquiring is equal to the log file which is stored by acquiring, and if the stored user information has key data, the log file leakage can cause unnecessary user loss, so that the generated log file needs to be encrypted.
At present, in order to ensure the security of a log file generated in the program running process, the log file in a text format is generally stored by using an encryption algorithm, but the log file is not compressed, when the log file needs to be obtained, a complete log file can be obtained by using a matched decryption tool or a text which can be consulted, otherwise, messy codes of the log file can be caused.
However, in the above method, since the log file is not compressed, when the amount of generated log file data is too large, a large amount of storage space is wasted in storing the log, and a certain amount of calculation power is required to be consumed to process the log file in the process of encrypting the log file, which causes waste of resources and reduces the processing efficiency of the log file.
Disclosure of Invention
The embodiment of the disclosure at least provides a log file processing method and device, computer equipment and a storage medium.
In a first aspect, an embodiment of the present disclosure provides a log file processing method, including:
acquiring a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part;
replacing a log statement in the log file with the variable data portion and an identifier of the log statement;
and storing the fixed data part and the identifier of the log statement in a log statement mapping table, wherein the log statement mapping table is used for restoring the variable data part and the identifier of the log statement in the replaced log file into the log statement.
In one possible embodiment, the method further comprises:
in response to an execution instruction for the replaced log file, obtaining an identifier of the log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
restoring the variable data portion in the replaced log file and the identifier of the log statement to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
In a possible implementation manner, the obtaining the log statement in the log file includes:
and acquiring the log statements in the log file based on the preset storage format information of the log statements.
In one possible embodiment, the storing the fixed data part and the identifier of the log statement in a log statement mapping table includes:
determining the position of a variable data part in a log statement based on the data name of the variable data part in the log statement;
setting a uniform data identifier at a position corresponding to the variable data part;
and taking the log statement with the set unified data identifier as a fixed data part, and storing the fixed data part and the identifier of the log statement in a log statement mapping table.
In one possible embodiment, the replacing the log statement in the log file with the variable data portion and the identifier of the log statement includes:
determining a location of the log statement in the log file based on the identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file by the target log statement based on the position of the log statement in the log file.
In one possible embodiment, the obtaining the identifier of the log statement from the replaced log file includes:
acquiring a target log statement in the replaced log file;
and extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, the restoring the variable data part in the replaced log file and the identifier of the log statement to the log statement specifically includes:
based on the identifier of the log statement, acquiring a variable data part in the target log statement from the replaced log file, and acquiring a fixed data part in the corresponding log statement from the log statement mapping table;
replacing the uniform data identification in the fixed data part with the variable data part to obtain the log statement;
and replacing the target log statement in the replaced log file by using the log statement to restore the log statement in the log file.
In a second aspect, an embodiment of the present disclosure further provides a log file processing apparatus, including:
the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring log statements in a log file, and the log statements comprise fixed data parts and variable data parts;
a replacement module for replacing a log statement in the log file with the variable data portion and an identifier of the log statement;
and the storage module is used for storing the fixed data part and the identifier of the log statement in a log statement mapping table, and the log statement mapping table is used for restoring the variable data part and the identifier of the log statement in the replaced log file into the log statement.
In a possible embodiment, the apparatus further comprises:
the restoring module is used for responding to an operating instruction of the replaced log file and acquiring the identifier of the log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
restoring the variable data portion in the replaced log file and the identifier of the log statement to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
In a possible implementation manner, the obtaining module is specifically configured to obtain the log statements in the log file based on preset storage format information of the log statements.
In a possible implementation manner, the storage module is specifically configured to determine, based on a data name of a variable data portion in a log statement, a position of the variable data portion in the log statement;
setting a uniform data identifier at a position corresponding to the variable data part;
and taking the log statement with the set unified data identifier as a fixed data part, and storing the fixed data part and the identifier of the log statement in a log statement mapping table.
In a possible implementation, the replacement module is configured to determine a location of the log statement in the log file based on an identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file by the target log statement based on the position of the log statement in the log file.
In a possible implementation manner, the replacement module is specifically configured to obtain a target log statement in the replaced log file;
and extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, the restoring module is specifically configured to obtain, based on the identifier of the log statement, a variable data portion in the target log statement from the replaced log file, and obtain, from the log statement mapping table, a fixed data portion in the corresponding log statement;
replacing the uniform data identification in the fixed data part with the variable data part to obtain the log statement;
and replacing the target log statement in the replaced log file by using the log statement to restore the log statement in the log file.
In a third aspect, this disclosure also provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions are executed by the processor to perform the steps in the first aspect or any one of the possible implementations of the first aspect.
In a fourth aspect, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed to perform the steps in the first aspect or any one of the possible implementation manners of the first aspect.
For the description of the effects of the log file processing apparatus, the computer device, and the computer-readable storage medium, reference is made to the description of the log file processing method, which is not repeated herein.
The log file processing method, device, computer equipment and storage medium provided by the embodiments of the present disclosure determine, based on the log statements obtained in the log file, the contained fixed data portion and variable data portion and their corresponding identifiers, replace the log statements obtained in the log file with the variable data portion and identifiers, and store the fixed data portion and identifiers in the log statement mapping table for restoring the obtained log statements, thereby implementing compression and encryption processing in the log file, reducing the size of the log file and the complexity of restoring the log file, improving the security of the log file, and compared with the prior art, in order to ensure the security of the log file, encrypting the log file using an encryption algorithm, for a micro operating system, using an algorithm for encryption increases the consumption of computation power, and reduces the efficiency of log file processing, compared with the method that the encrypted log file is not compressed and the storage space is wasted, the method that the log statement in the log file is replaced by the variable data part and the identifier corresponding to the obtained log statement realizes the compression of the log file, simultaneously realizes the encryption of the log file, reduces the size of the log file, ensures the safety of the log file, does not cause the waste of resources, and improves the processing efficiency of the log file.
Furthermore, the log file processing method provided by the embodiment of the present disclosure may further search, in response to an operation instruction for the replaced log file, a fixed data portion of an original log statement in the log statement mapping table according to an identifier of the replaced log statement, then implement restoration of the log statement based on the variable data portion, and complete decryption of the log file by referring to the established log statement mapping table, thereby improving applicability and security of the log file, and reducing complexity of log file restoration.
In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.
Fig. 1 is a schematic view illustrating an application scenario of a log file processing method provided by an embodiment of the present disclosure;
FIG. 2 is a flow chart illustrating a log file processing method provided by an embodiment of the present disclosure;
FIG. 3 illustrates a schematic diagram of a project file provided by an embodiment of the present disclosure;
FIG. 4 illustrates a flowchart of restoring replaced log statements provided by an embodiment of the present disclosure;
fig. 5 is a schematic diagram illustrating a log file processing apparatus provided by an embodiment of the present disclosure;
fig. 6 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of embodiments of the present disclosure, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.
Furthermore, the terms "first," "second," and the like in the description and in the claims, and in the drawings described above, in the embodiments of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein.
Reference herein to "a plurality or a number" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
Research shows that in the prior art, in order to ensure the security of a log file generated in the program running process, an encryption algorithm is used for encrypting the log file in the log file generating process, and under the condition that the log file is leaked, if a matched encryption tool or a file which can be consulted is not available, the obtained log file is messy code, so that the security of the log file is improved, but the encryption operation on the log file can be completed only by consuming a certain computing power, the consumption of computing power resources is increased, and the processing efficiency of the log file is reduced.
Based on the above research, the present disclosure provides a log file processing method, apparatus, computer device, and storage medium, in which a variable data portion and an identifier corresponding to an acquired log statement are used to replace a log statement in a log file, so that the log file is compressed and encrypted, the size of the log file is reduced, the security of the log file is ensured, no resource waste is caused, and the efficiency of processing the log file is improved.
The above-mentioned drawbacks are the results of the inventor after practical and careful study, and therefore, the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to the above-mentioned problems should be the contribution of the inventor in the process of the present disclosure.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
To facilitate understanding of the present embodiment, an application scenario of the log file processing method disclosed in the embodiment of the present disclosure is introduced first, and as shown in fig. 1, an application scenario schematic diagram of the log file processing method provided in the embodiment of the present disclosure is shown. A user logs in the log processing server 12 through an application client installed in the terminal device 11, where the application client may be a browser of a web page or an application client installed in a terminal device, such as a mobile phone or a tablet computer. The terminal device 11 and the log processing server 12 are communicatively connected through a network, which may be a local area network, a cellular network, a wide area network, and the like. The terminal device 11 may be a portable device (e.g., a mobile phone, a tablet, a notebook, etc.) or a Personal Computer (PC), and the log processing server 12 may be any device capable of providing internet services.
The log processing server 12 can provide log file processing services for users, for example, a user submits different operation requests through a client installed in the terminal device 11, the client generates different program running requests correspondingly based on the received different operation requests from the user and sends the different program running requests to the log file processing server 12, the log file processing server 12 starts different programs correspondingly according to the received different program running requests, during the program running, the log processing server 12 extracts log statements stored in the code segments for the code segments generated during the program running, determines variable data parts and fixed data parts included in the log statements and identifiers corresponding to the log statements, determines the positions of the variable data in the log statements and sets uniform data identifiers based on the data names of the variable data parts in the log statements, the method comprises the steps of taking a log statement with a set unified data identifier as a fixed data part, generating a target log statement by using a variable data part and an identifier to replace the log statement, completing compression and encryption processing of the log statement, storing the fixed data part and the identifier in a log statement mapping table as a basis for restoring a log file, when a user needs to acquire the log file generated in the program operation process, analyzing the target log file by a server to obtain the identifier and the variable data part corresponding to the target log statement, determining the corresponding fixed data part in the log statement mapping table based on the obtained identifier, replacing the unified data identifier set in the fixed data part by using the variable data part obtained by analysis to obtain the log statement, and replacing the target log statement in the log file by using the log statement, the log statements are restored, the log files are further restored, then the server sends the restored log files to the client, and the user can obtain the needed log files, so that the safety of the log files is improved, and the consumption of labor resources is reduced.
In the following, a detailed description is given of a log file processing method disclosed in an embodiment of the present disclosure, an execution subject of the log file processing method provided in the embodiment of the present disclosure is generally a computing device with certain computing capability, and the computing device includes, for example: a terminal device, which may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, or a server or other processing device. In some possible implementations, the log file processing method may be implemented by a processor calling computer readable instructions stored in a memory.
The log file processing method provided by the embodiment of the present disclosure is described below by taking an execution subject as a computer device as an example.
As shown in fig. 2, a flowchart of a log file processing method provided in an embodiment of the present disclosure may include the following steps:
s201: and acquiring a log statement in the log file, wherein the log statement comprises a fixed data part and a variable data part.
In this step, in order to ensure the security of the log file generated during the program running process and avoid the log file collected by the client side from being too large, the log file processing server 12 compresses and encrypts the generated log file.
In specific implementation, in the running process of the program, the engineering file including the log file and other data files may be generated, so the log file processing server needs to first obtain the log file stored in the engineering file. As shown in fig. 3, which is a schematic diagram of a project file provided in an embodiment of the present disclosure, a white frame part indicates a LOG file included therein, and the rest is other included data files, in a specific implementation, a format of the LOG file included in the project file is fixed, and as can be seen in fig. 3, a LOG statement starts with "LOG"; the method comprises the steps that a format of ending appears, preset format storage information can be set based on the format, a log file processing server carries out regular matching on an engineering file by using a regular expression which is the same as the preset format storage information, a stored log file can be obtained, further, log statements in the log file can be determined by using the log file, then the log statements are analyzed by using the regular expression again, a fixed data part and a variable data part in the log statements can be determined, and in addition, when the log statements stored in the log file are obtained, identifiers corresponding to the log statements can be generated in a self-adding mode according to the sequence of the obtained log statements.
Taking the code segment included in the engineering file shown in fig. 3 as an example, the LOG file processing server matches the LOG file in the engineering file according to the preset format storage information ("LOG;") to obtain a LOG file LOG (LS _ INFO, APP _ variation _ MODULE) < "Request video stream ssrc: "< < ssrc <" width: "< < width < <" height: "< < height; (ii) a Further, it can be determined that the log statement in the log file is "Request video stream ssrc: "< < ssrc <" width: "< < width < <" height: "< < height; "; then, analyzing the data by using a regular expression to determine the invariable data' Request video stream ssrc: "< < < <" > width: "< < < <" height: "<; "as a fixed data part, the changed data" ssrc, width, height "is determined as a variable data part, and in addition, assuming that the log statement acquired in this step is the first log statement of the log file, it can be determined that the identifier of this log statement is 1.
S202: the log statement in the log file is replaced with the variable data portion and an identifier of the log statement.
In a specific implementation, the generated log statement identifier and the determined variable data part may be combined into a target log statement, in an implementation, the log statement identifier may be placed in front of the variable data part, and taking the log statement in fig. 3 as an example, the target log statement after combination may be determined to be 1, ssrc, width, height; in another embodiment, the log statement identifier may be placed behind the variable data portion, and the combination manner of the log statement identifier and the variable data portion in the embodiment of the present disclosure is not limited in the embodiment of the present disclosure, and further, the combined target log statement may be used to replace the log statement in the acquired log file.
For example, for the log statement "" Request video stream ssrc: "< < ssrc <" width: "< < width < <" height: "< < height; "replace, can get the goal journal statement after replacing 1, ssrc, width, height.
S203: and storing the identifiers of the fixed data part and the log statement in a log statement mapping table, wherein the log statement mapping table is used for restoring the identifiers of the variable data part and the log statement in the replaced log file into the log statement.
In this step, in order to restore the replaced log file, a log statement mapping table for storing the fixed data portion and the identifier of the log statement needs to be generated.
In specific implementation, a position of variable data in a log statement is determined based on a data name corresponding to the determined variable data portion, and then the variable data portion at the corresponding position in the log statement is replaced with a uniform data identifier, for example, the uniform data identifier may be a placeholder, and further, the replaced log statement is taken as a fixed data portion, and an identifier generated in the process of obtaining the log statement is stored in a log statement mapping table as a whole, taking the log statement included in fig. 3 as an example, a log statement after being replaced with the placeholder is "Request video stream ssrc: "< <% d <" width: "<% d <" height: "< <% d; (ii) a Wherein, "% d" represents a placeholder, the data content in the log statement mapping table may be 1, and "Request video stream ssrc: "< <% d <" width: "<% d <" height: "< <% d; . Further, after the obtained log statement is replaced by the target log statement, the log file processing server normally compiles the engineering file of which the log file has been replaced, and generates a preprocessing program only containing the target log statement. Based on this, even if the preprocessing program is leaked, the analyzed log statement only comprises the identifier and the variable data part, no associated context information exists, and if the generated log statement mapping table is not used as a reference, the obtained identifier and the variable data part have no practical significance, so that the leakage of user information is not caused, and the security of the log file is ensured.
Furthermore, when the user needs to acquire the stored log file, an operation instruction for the replaced log file is submitted to the client, the client sends an operation instruction for the replaced log file to the log file processing server according to the operation instruction, and the log file processing server restores the replaced log statement in response to the received operation instruction for the replaced log file.
As shown in fig. 4, a flowchart for restoring a replaced log statement provided in an embodiment of the present disclosure may include the following steps:
s401: in response to an execution instruction to the replaced log file, an identifier of the log statement is obtained from the replaced log file.
In specific implementation, the log file processing server responds to an operation instruction of the replaced log file, analyzes the generated preprocessing program, can acquire each target log statement in the replaced log file, and matches each acquired target log statement by using a regular expression, and can acquire an identifier of the target log statement, where it is to be noted that the identifier of the target log statement is the identifier of the log statement acquired in the replaced log file. In addition, an identifier representing an unreplaced log statement that coincides with the identifier of each acquired target log statement exists in the generated log statement mapping table.
S402: based on the identifier of the log statement, the fixed data portion of the log statement is obtained from the log statement mapping table.
In a specific implementation, according to the identifier of the obtained log statement, an identifier corresponding to the identifier of the obtained log statement is matched from the generated log statement mapping table, and based on the identifier, the fixed data part of the un-replaced log statement stored in the log statement mapping table can be obtained.
S403: the variable data portion in the replaced log file and the identifier of the log statement are restored to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
In specific implementation, according to the fixed data part of the log statement stored in the log statement mapping table determined in S402, a uniform data identifier position included in the fixed data part may be determined, in addition, according to the identifier of the target log statement included in each extracted target log statement, a variable data part corresponding to the target log statement may be determined, according to the determined uniform data identifier position, each uniform data identifier is replaced by the variable data part, a log statement that is not replaced may be obtained, further, a log file that is not replaced may be obtained, and then the target log statement in the replaced log file is replaced by the log statement of the log file that is not replaced, based on which, the log recovery statement may be implemented.
For example, taking the target log statement obtained after analysis as "1, XXX, 640, 480" as an example, matching the target log statement by using a regular expression may obtain an identifier 1 of the target log statement, matching from the generated log statement mapping table according to the identifier 1 may determine that an identifier consistent with the identifier 1 exists, and further determine that a fixed data portion "" Request video stream ssrc stored in the log statement mapping table according to the identifier 1: "< <% d <" width: "<% d <" height: "< <% d; "then, according to the analysis of the target log statement" 1, XXX, 640, 480 ", obtain the variable data portion" XXX, 640, 480 "in the target log statement, and replace each placeholder with the variable data portion" XXX, 640, 480 "according to the position of the placeholder in the fixed data portion, obtain the log file" Request video stream ssrc: "< < XXX <" width: "< <640< <" height: "< < 480; ", further, using" "Request video stream ssrc: "< < XXX <" width: "< <640< <" height: "< < 480; "replace target log statement" 1, XXX, 640, 480 "to restore the log file. In addition, the log file processor restores the analyzed log file and sends the restored log file to the client, so that the user can obtain the required log file at the client.
The disclosed embodiment provides a log file processing method, wherein a log file processor processes an engineering file generated in the program running process, uses a regular expression to carry out matching, obtains the log file stored in the engineering file, determines log statements in the log file according to the log file, determines a fixed data part, a variable data part and an identifier in the log statements, sets a uniform data identifier at a position corresponding to the variable data part to complete the setting of the fixed data part, then uses the identifier and the variable data to replace the log statements to obtain a target log statement, realizes the compression and encryption of the log file, reduces the size of the log file, improves the file processing efficiency, in addition, generates a log statement mapping table based on the fixed data part and the identifier, when a user has a requirement for obtaining the log file which is not replaced, each target log statement contained in the replaced log file is obtained through analysis, an identifier and a variable data part in the target log statement are further determined, the fixed data part of the corresponding un-replaced log file can be obtained according to the matching of the determined identifier and the identifier in the log statement mapping table, the identifier of the target log file is replaced by the fixed data part, then the uniform data identifier in the fixed data part is replaced by the variable data to obtain the un-replaced log statement, the target log statement in the replaced log file is replaced by the un-replaced log statement to obtain the un-replaced log statement, the un-replaced log file can be obtained on the basis of the obtained un-replaced log statement, the replaced log statement is restored through the generated log statement mapping table, and the encrypted log statement is decrypted, the complexity of log file restoration is reduced.
It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.
Based on the same inventive concept, the embodiment of the present disclosure further provides a log file processing apparatus corresponding to the log file processing method, and as the principle of the apparatus in the embodiment of the present disclosure for solving the problem is similar to the log file processing method described above in the embodiment of the present disclosure, the implementation of the apparatus may refer to the implementation of the method, and repeated details are not described again.
As shown in fig. 5, a schematic diagram of a log file processing apparatus provided in an embodiment of the present disclosure includes:
an obtaining module 501, configured to obtain a log statement in a log file, where the log statement includes a fixed data portion and a variable data portion;
a replacing module 502 for replacing the log statement in the log file with the variable data portion and the identifier of the log statement;
a storing module 503, configured to store the fixed data part and the identifier of the log statement in a log statement mapping table, where the log statement mapping table is used to restore the variable data part and the identifier of the log statement in the replaced log file to the log statement.
In a possible embodiment, the apparatus further comprises:
a restoring module 504, configured to, in response to an operation instruction for the replaced log file, obtain an identifier of the log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
restoring the variable data portion in the replaced log file and the identifier of the log statement to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
In a possible implementation manner, the obtaining module 501 is specifically configured to obtain the log statements in the log file based on preset storage format information of the log statements.
In a possible implementation manner, the storage module 503 is specifically configured to determine, based on a data name of a variable data portion in a log statement, a position of the variable data portion in the log statement;
setting a uniform data identifier at a position corresponding to the variable data part;
and taking the log statement with the set unified data identifier as a fixed data part, and storing the fixed data part and the identifier of the log statement in a log statement mapping table.
In a possible implementation, the replacing module 502 is configured to determine a location of the log statement in the log file based on an identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file by the target log statement based on the position of the log statement in the log file.
In a possible implementation manner, the replacing module 502 is specifically configured to obtain a target log statement in the replaced log file;
and extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, the restoring module 504 is specifically configured to obtain, based on the identifier of the log statement, a variable data portion in the target log statement from the replaced log file, and obtain, from the log statement mapping table, a fixed data portion in the corresponding log statement;
replacing the uniform data identification in the fixed data part with the variable data part to obtain the log statement;
and replacing the target log statement in the replaced log file by using the log statement to restore the log statement in the log file.
The description of the processing flow of each module in the device and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.
An embodiment of the present disclosure further provides a computer device, as shown in fig. 6, which is a schematic structural diagram of the computer device provided in the embodiment of the present disclosure, and the computer device includes:
a processor 61 and a memory 62; the memory 62 stores machine-readable instructions executable by the processor 61, the processor 61 being configured to execute the machine-readable instructions stored in the memory 62, the processor 61 performing the following steps when the machine-readable instructions are executed by the processor 61: s201: acquiring log statements in a log file, wherein the log statements comprise fixed data parts and variable data parts; s202: replacing the log statement in the log file with the variable data portion and the identifier of the log statement S203: and storing the identifiers of the fixed data part and the log statement in a log statement mapping table, wherein the log statement mapping table is used for restoring the identifiers of the variable data part and the log statement in the replaced log file into the log statement.
The memory 62 includes a memory 621 and an external memory 622; the memory 621 is also referred to as an internal memory, and temporarily stores operation data in the processor 61 and data exchanged with the external memory 622 such as a hard disk, and the processor 61 exchanges data with the external memory 622 via the memory 621.
In a possible implementation, the instructions executed by the processor 61 further include:
in response to an execution instruction for the replaced log file, obtaining an identifier of a log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
restoring the variable data portion in the replaced log file and the identifier of the log statement to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
In a possible implementation manner, in the instructions executed by the processor 61, the obtaining a log statement in a log file includes:
and acquiring the log statements in the log file based on the preset storage format information of the log statements.
In a possible implementation, the instructions executed by the processor 61, wherein the storing the fixed data part and the identifier of the log statement in a log statement mapping table, includes:
determining the position of a variable data part in a log statement based on the data name of the variable data part in the log statement;
setting a uniform data identifier at a position corresponding to the variable data part;
and taking the log statement with the set unified data identifier as a fixed data part, and storing the fixed data part and the identifier of the log statement in a log statement mapping table.
In one possible embodiment, the replacing, in the instructions executed by the processor 61, the log statement in the log file with the variable data portion and the identifier of the log statement includes:
determining a location of the log statement in the log file based on the identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file by the target log statement based on the position of the log statement in the log file.
In one possible implementation, the instructions executed by processor 61, the obtaining the identifier of the log statement from the replaced log file, includes:
acquiring a target log statement in the replaced log file;
and extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, in the instructions executed by the processor 61, the restoring the variable data part in the replaced log file and the identifier of the log statement to the log statement specifically includes:
based on the identifier of the log statement, acquiring a variable data part in the target log statement from the replaced log file, and acquiring a fixed data part in the corresponding log statement from the log statement mapping table;
replacing the uniform data identification in the fixed data part with the variable data part to obtain the log statement;
and replacing the target log statement in the replaced log file by using the log statement to restore the log statement in the log file.
The specific execution process of the instruction may refer to the steps of the log file processing method described in the embodiments of the present disclosure, and details are not described here.
The embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the log file processing method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.
The computer program product of the log file processing method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the steps of the log file processing method described in the above method embodiments, which may be referred to in the above method embodiments specifically, and are not described herein again. The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims (10)

1. A log file processing method is characterized by comprising the following steps:
acquiring a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part;
replacing a log statement in the log file with the variable data portion and an identifier of the log statement;
and storing the fixed data part and the identifier of the log statement in a log statement mapping table, wherein the log statement mapping table is used for restoring the variable data part and the identifier of the log statement in the replaced log file into the log statement.
2. The log file processing method according to claim 1, further comprising:
in response to an execution instruction for the replaced log file, obtaining an identifier of a log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
restoring the variable data portion in the replaced log file and the identifier of the log statement to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
3. The log file processing method according to claim 1, wherein the obtaining of the log statement in the log file comprises:
and acquiring the log statements in the log file based on the preset storage format information of the log statements.
4. The log file processing method of claim 3, wherein the storing the fixed data part and the identifier of the log statement in a log statement mapping table comprises:
determining the position of a variable data part in a log statement based on the data name of the variable data part in the log statement;
setting a uniform data identifier at a position corresponding to the variable data part;
and taking the log statement with the set unified data identifier as a fixed data part, and storing the fixed data part and the identifier of the log statement in a log statement mapping table.
5. The log file processing method according to any one of claims 1 to 4, wherein said replacing the log statement in the log file with the variable data portion and the identifier of the log statement comprises:
determining a location of the log statement in the log file based on the identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file by the target log statement based on the position of the log statement in the log file.
6. The log file processing method of claim 5, wherein the obtaining the identifier of the log statement from the replaced log file comprises:
acquiring a target log statement in the replaced log file;
and extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
7. The log file processing method according to claim 4, wherein the restoring the replaced variable data portion in the log file and the identifier of the log statement to the log statement specifically comprises:
based on the identifier of the log statement, acquiring a variable data part in the target log statement from the replaced log file, and acquiring a fixed data part in the corresponding log statement from the log statement mapping table;
replacing the uniform data identification in the fixed data part with the variable data part to obtain the log statement;
and replacing the target log statement in the replaced log file by using the log statement to restore the log statement in the log file.
8. A log file processing apparatus characterized by comprising:
the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring log statements in a log file, and the log statements comprise fixed data parts and variable data parts;
a replacement module for replacing a log statement in the log file with the variable data portion and an identifier of the log statement;
and the storage module is used for storing the fixed data part and the identifier of the log statement in a log statement mapping table, and the log statement mapping table is used for restoring the variable data part and the identifier of the log statement in the replaced log file into the log statement.
9. A computer device, comprising: a processor, a memory storing machine readable instructions executable by the processor, the processor for executing the machine readable instructions stored in the memory, the processor performing the steps of the log file processing method of any one of claims 1 to 7 when the machine readable instructions are executed by the processor.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a computer device, performs the steps of the log file processing method according to any one of claims 1 to 7.
CN202011303567.8A 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium Active CN112286896B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011303567.8A CN112286896B (en) 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011303567.8A CN112286896B (en) 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112286896A true CN112286896A (en) 2021-01-29
CN112286896B CN112286896B (en) 2024-02-06

Family

ID=74399358

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011303567.8A Active CN112286896B (en) 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112286896B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113179265A (en) * 2021-04-26 2021-07-27 湖南蚁坊软件股份有限公司 Log compression encryption method, device, equipment and storage medium
WO2023020187A1 (en) * 2021-08-20 2023-02-23 Oppo广东移动通信有限公司 Data obtaining methods and apparatuses, electronic device and storage medium
DE102022002450A1 (en) 2022-07-05 2024-01-11 Mercedes-Benz Group AG Method for processing log files, data processing system and vehicle

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5790508A (en) * 1996-08-20 1998-08-04 Industrial Technology Research Institute Multidisk changer
CN101047921A (en) * 2006-03-07 2007-10-03 三星电子株式会社 Apparatus and method of providing fast menus in menu screen of a mobile communication terminal
CN103226509A (en) * 2013-04-08 2013-07-31 上海华力微电子有限公司 Method for automatically analyzing system log
CN103337113A (en) * 2013-07-17 2013-10-02 广州广电运通金融电子股份有限公司 Method and device for intelligently analyzing electronic day-to-day journals, as well as processor
CN103412924A (en) * 2013-08-12 2013-11-27 东软集团股份有限公司 Log multi-language query method and system
CN104331487A (en) * 2014-11-13 2015-02-04 携程计算机技术(上海)有限公司 Method and device for processing logs
CN106484742A (en) * 2015-09-01 2017-03-08 北京国双科技有限公司 Log processing method and device
CN109684290A (en) * 2018-12-20 2019-04-26 东软集团股份有限公司 Log storing method, device, equipment and computer readable storage medium
CN109791520A (en) * 2016-09-30 2019-05-21 亚马逊技术股份有限公司 The log recording of the Space Coupling of physical medium perception and playback
CN111190873A (en) * 2019-12-24 2020-05-22 同济大学 Log mode extraction method and system for log training of cloud native system
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN111930701A (en) * 2020-08-13 2020-11-13 工银科技有限公司 Log structured processing method and device

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5790508A (en) * 1996-08-20 1998-08-04 Industrial Technology Research Institute Multidisk changer
CN101047921A (en) * 2006-03-07 2007-10-03 三星电子株式会社 Apparatus and method of providing fast menus in menu screen of a mobile communication terminal
CN103226509A (en) * 2013-04-08 2013-07-31 上海华力微电子有限公司 Method for automatically analyzing system log
CN103337113A (en) * 2013-07-17 2013-10-02 广州广电运通金融电子股份有限公司 Method and device for intelligently analyzing electronic day-to-day journals, as well as processor
CN103412924A (en) * 2013-08-12 2013-11-27 东软集团股份有限公司 Log multi-language query method and system
CN104331487A (en) * 2014-11-13 2015-02-04 携程计算机技术(上海)有限公司 Method and device for processing logs
CN106484742A (en) * 2015-09-01 2017-03-08 北京国双科技有限公司 Log processing method and device
CN109791520A (en) * 2016-09-30 2019-05-21 亚马逊技术股份有限公司 The log recording of the Space Coupling of physical medium perception and playback
CN109684290A (en) * 2018-12-20 2019-04-26 东软集团股份有限公司 Log storing method, device, equipment and computer readable storage medium
CN111190873A (en) * 2019-12-24 2020-05-22 同济大学 Log mode extraction method and system for log training of cloud native system
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN111930701A (en) * 2020-08-13 2020-11-13 工银科技有限公司 Log structured processing method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
唐球 等: "基于差分压缩的大规模日志压缩系统", 《通信学报》, vol. 36, no. 1, pages 197 - 202 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113179265A (en) * 2021-04-26 2021-07-27 湖南蚁坊软件股份有限公司 Log compression encryption method, device, equipment and storage medium
WO2023020187A1 (en) * 2021-08-20 2023-02-23 Oppo广东移动通信有限公司 Data obtaining methods and apparatuses, electronic device and storage medium
DE102022002450A1 (en) 2022-07-05 2024-01-11 Mercedes-Benz Group AG Method for processing log files, data processing system and vehicle
WO2024008460A1 (en) 2022-07-05 2024-01-11 Mercedes-Benz Group AG Method for processing log files, data processing system and vehicle

Also Published As

Publication number Publication date
CN112286896B (en) 2024-02-06

Similar Documents

Publication Publication Date Title
CN112286896A (en) Log file processing method and device, computer equipment and storage medium
US20140344943A1 (en) System and method for securing sensitive data
CN107169726B (en) Two-dimensional code attendance card punching method and device
CN103123588A (en) Method and system for obtaining difference upgrade patch
CN104144081A (en) General application log management method, device and system
CN105339924A (en) Efficient data compression and analysis as a service
KR20170068814A (en) Apparatus and Method for Recognizing Vicious Mobile App
CN110933063B (en) Data encryption method, data decryption method and equipment
CN111949974A (en) Authentication method and device, computer equipment and storage medium
CN110661748A (en) Log encryption method, log decryption method and log encryption device
CN111611408A (en) Public opinion analysis method and device, computer equipment and storage medium
CN114285575B (en) Image encryption and decryption method and device, storage medium and electronic device
CN112287371B (en) Method and device for storing industrial data and computer equipment
CN110795162A (en) Method and device for generating container mirror image file
CN113642025A (en) Interface data processing method, device, equipment and storage medium
CN113904832A (en) Data encryption method, device, equipment and storage medium
CN104657631B (en) The processing method and processing device of the channel information of application
CN109324843B (en) Fingerprint processing system and method and fingerprint equipment
CN108109625B (en) Mobile phone voice recognition internal and external network transmission system and method
CN110187385B (en) Seismic data acquisition method, seismic data processing method and device
CN113434827A (en) Information tracing method, device and system, electronic equipment and storage medium
CN114676451A (en) Electronic document signature method and device, storage medium and electronic equipment
CN112580101A (en) Data decryption method and terminal
CN111783415A (en) Template configuration method and device
CN113327591B (en) Voice processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant