CN112286896B - Log file processing method and device, computer equipment and storage medium - Google Patents

Log file processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN112286896B
CN112286896B CN202011303567.8A CN202011303567A CN112286896B CN 112286896 B CN112286896 B CN 112286896B CN 202011303567 A CN202011303567 A CN 202011303567A CN 112286896 B CN112286896 B CN 112286896B
Authority
CN
China
Prior art keywords
log
statement
identifier
log file
log statement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011303567.8A
Other languages
Chinese (zh)
Other versions
CN112286896A (en
Inventor
吴子敬
陈雪龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baijia Technology Group Co ltd
Original Assignee
Beijing Baijia Technology Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baijia Technology Group Co ltd filed Critical Beijing Baijia Technology Group Co ltd
Priority to CN202011303567.8A priority Critical patent/CN112286896B/en
Publication of CN112286896A publication Critical patent/CN112286896A/en
Application granted granted Critical
Publication of CN112286896B publication Critical patent/CN112286896B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/602Providing cryptographic facilities or services
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/62Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2107File encryption
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • Data Mining & Analysis (AREA)
  • Human Computer Interaction (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The present disclosure provides a log file processing method, apparatus, computer device and storage medium, wherein the method includes: acquiring a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part; replacing the log statements in the log file with identifiers of the variable data portion and the log statements; storing the fixed data portion and the identifier of the log statement in a log statement mapping table; the log statement map is used to restore the variable data portion in the replaced log file and the identifier of the log statement to a log statement. According to the embodiment of the disclosure, the method for replacing the log statement in the log file by using the variable data part and the identifier is used for realizing compression and encryption of the log file, reducing the size of the log file and ensuring the safety of the log file, and in addition, the method for storing the fixed data part and the identifier in the log statement by using the established log statement mapping table is used for ensuring the realization foundation of log statement restoration.

Description

Log file processing method and device, computer equipment and storage medium
Technical Field
The disclosure relates to the technical field of computers, and in particular relates to a log file processing method, a log file processing device, computer equipment and a storage medium.
Background
The log file is used as an important basis for storing the user information, the log file obtained and recorded is equivalent to the stored user information, if key data exist in the stored user information, the log file leakage can cause unnecessary user loss, and therefore encryption processing is needed to be carried out on the generated log file.
At present, in order to ensure the security of the log file generated in the running process of the program, the log file in a text format is generally stored by using an encryption algorithm, but the log file is not compressed, when the log file needs to be acquired, the complete log file can be acquired by using a matched decryption tool or a text which can be referred to, otherwise, the log file is messy.
However, in the above method, since the log file is not compressed, when the generated data volume of the log file is too large, storing the log wastes a large amount of storage space, and in the process of encrypting the log file, a certain amount of calculation force is required to be consumed for processing, so that the resource waste is caused, and the processing efficiency of the log file is reduced.
Disclosure of Invention
The embodiment of the disclosure at least provides a log file processing method, a log file processing device, computer equipment and a storage medium.
In a first aspect, an embodiment of the present disclosure provides a log file processing method, including:
acquiring a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part;
replacing a log statement in the log file with the variable data portion and an identifier of the log statement;
and storing the identifiers of the fixed data part and the log statements in a log statement mapping table, wherein the log statement mapping table is used for restoring the identifiers of the variable data part and the log statements in the replaced log file into the log statements.
In one possible embodiment, the method further comprises:
in response to an operation instruction to the replaced log file, acquiring an identifier of the log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file, the variable data portion of the replaced log file and the identifier of the log statement are restored to the log statement.
In one possible implementation manner, the acquiring the log statement in the log file includes:
and acquiring the log statement in the log file based on the preset storage format information of the log statement.
In a possible implementation manner, the storing the fixed data portion and the identifier of the log statement in a log statement mapping table includes:
determining the position of a variable data part in a log statement based on the data name of the variable data part in the log statement;
a unified data identifier is arranged at a position corresponding to the variable data part;
and taking the log statement with the unified data mark as a fixed data part, and storing the log statement and the identifier of the log statement in a log statement mapping table.
In a possible implementation manner, the replacing the log statement in the log file with the variable data portion and the identifier of the log statement includes:
determining a location of the log statement in the log file based on an identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
And replacing the log statement in the log file with the target log statement based on the position of the log statement in the log file.
In a possible implementation manner, the acquiring the identifier of the log statement from the replaced log file includes:
acquiring a target log statement in the replaced log file;
extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In one possible implementation manner, the restoring the identifier of the variable data portion and the log statement in the replaced log file into the log statement specifically includes:
based on the identifier of the log statement, acquiring a variable data part in the target log statement from the replaced log file, and acquiring a fixed data part in the corresponding log statement from the log statement mapping table;
replacing the unified data identifier in the fixed data part by the variable data part to obtain the log statement;
replacing the target log statement in the replaced log file with the log statement to restore the log statement in the log file.
In a second aspect, an embodiment of the present disclosure further provides a log file processing apparatus, including:
the acquisition module is used for acquiring log sentences in the log file, wherein the log sentences comprise a fixed data part and a variable data part;
a replacement module for replacing a log statement in the log file with the variable data portion and an identifier of the log statement;
and the storage module is used for storing the fixed data part and the identifier of the log statement in a log statement mapping table, and the log statement mapping table is used for restoring the variable data part and the identifier of the log statement in the replaced log file into the log statement.
In one possible embodiment, the apparatus further comprises:
a restoring module, configured to obtain an identifier of the log statement from the replaced log file in response to an operation instruction for the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file, the variable data portion of the replaced log file and the identifier of the log statement are restored to the log statement.
In a possible implementation manner, the acquiring module is specifically configured to acquire the log statement in the log file based on the preset storage format information of the log statement.
In a possible implementation manner, the storage module is specifically configured to determine a location of the variable data portion in the log statement based on a data name of the variable data portion in the log statement;
a unified data identifier is arranged at a position corresponding to the variable data part;
and taking the log statement with the unified data mark as a fixed data part, and storing the log statement and the identifier of the log statement in a log statement mapping table.
In a possible implementation manner, the replacing module is configured to determine a location of the log statement in the log file based on an identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file with the target log statement based on the position of the log statement in the log file.
In a possible implementation manner, the replacing module is specifically configured to obtain a target log statement in the replaced log file;
Extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, the restoring module is specifically configured to obtain, based on the identifier of the log statement, a variable data portion in the target log statement from the replaced log file, and obtain a fixed data portion in the corresponding log statement from the log statement mapping table;
replacing the unified data identifier in the fixed data part by the variable data part to obtain the log statement;
replacing the target log statement in the replaced log file with the log statement to restore the log statement in the log file.
In a third aspect, an optional implementation manner of the disclosure further provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, where the machine-readable instructions, when executed by the processor, perform the steps in the first aspect, or any possible implementation manner of the first aspect, when executed by the processor.
In a fourth aspect, an alternative implementation of the present disclosure further provides a computer readable storage medium having stored thereon a computer program which when executed performs the steps of the first aspect, or any of the possible implementation manners of the first aspect.
The description of the effects of the log file processing apparatus, the computer device, and the computer-readable storage medium is referred to the description of the log file processing method, and is not repeated here.
According to the log file processing method, device, computer equipment and storage medium, based on the log sentences obtained in the log file, the fixed data part, the variable data part and the identifiers corresponding to the fixed data part and the variable data part are determined, the log sentences obtained in the log file are replaced by the variable data part and the identifiers, the fixed data part and the identifiers are stored in the log sentence mapping table for restoring the obtained log sentences, compression and encryption processing in the log file are achieved, the size of the log file and the complexity of restoring the log file are reduced, the security of the log file is improved, compared with the prior art, in order to ensure the security of the log file, an encryption algorithm is used for encrypting the log file, the consumption of calculation force is increased by using the algorithm, the efficiency of log file processing is reduced, and compared with the storage space is wasted, the log sentence mapping method in the log file is replaced by using the variable data part and the identifiers corresponding to the obtained log sentences, the log sentences is achieved, the compression and encryption sentence mapping efficiency of the log file is also reduced, the log sentence is greatly reduced, the security of the log sentence is greatly reduced, the log sentence is restored, the security of the log file is not is restored, and the log sentence is greatly is restored, the security of the log file is ensured, and the security of the log sentence is greatly is not wasted, and the log sentence is restored.
Further, according to the log file processing method provided by the embodiment of the disclosure, the operation instruction of the replaced log file can be responded, the fixed data part of the original log statement is searched in the log statement mapping table according to the identifier of the replaced log statement, the log statement is restored based on the variable data part, the log file is decrypted by referring to the established log statement mapping table, the applicability and the safety of the log file are improved, and the complexity of log file restoration is reduced.
The foregoing objects, features and advantages of the disclosure will be more readily apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the embodiments are briefly described below, which are incorporated in and constitute a part of the specification, these drawings showing embodiments consistent with the present disclosure and together with the description serve to illustrate the technical solutions of the present disclosure. It is to be understood that the following drawings illustrate only certain embodiments of the present disclosure and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort.
Fig. 1 illustrates an application scenario schematic diagram of a log file processing method provided by an embodiment of the present disclosure;
FIG. 2 illustrates a flow chart of a log file processing method provided by an embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of an engineering document provided by an embodiment of the present disclosure;
FIG. 4 illustrates a flow chart for restoring a replaced log statement provided by an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a log file processing apparatus according to an embodiment of the present disclosure;
fig. 6 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. The components of the disclosed embodiments generally described and illustrated herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be made by those skilled in the art based on the embodiments of this disclosure without making any inventive effort, are intended to be within the scope of this disclosure.
Furthermore, the terms first, second and the like in the description and in the claims of embodiments of the disclosure and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein.
Reference herein to "a plurality of" or "a number" means two or more than two. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
According to research, in the prior art, in order to ensure the security of the log file generated in the running process of the program, in the process of generating the log file, an encryption algorithm is utilized to encrypt the log file, under the condition that the log file is leaked, if no matched encryption tool or file can be referred to, the acquired log file is messy, although the security of the log file is improved, the log file can be completed only by consuming a certain amount of calculation force when the encryption operation is performed on the log file, the consumption of calculation force resources is increased, the processing efficiency of the log file is reduced, in addition, in the process, the log file is not compressed, a large amount of storage space is wasted when the data amount of the log file is large, and the storage difficulty of the log file is increased.
Based on the above study, the disclosure provides a log file processing method, a device, a computer device and a storage medium, by using the variable data part and an identifier corresponding to the obtained log statement to replace the log statement method in the log file, the log file is compressed, the log file is encrypted, the size of the log file is reduced, the safety of the log file is ensured, the resource waste is not caused, the processing efficiency of the log file is improved, and in addition, the method for storing the fixed data part and the identifier in the log statement by using the established log statement mapping table ensures the realization basis for restoring the replaced log statement.
The present invention is directed to a method for manufacturing a semiconductor device, and a semiconductor device manufactured by the method.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
For the convenience of understanding the present embodiment, first, an application scenario of a log file processing method disclosed in the present embodiment is described, as shown in fig. 1, which is a schematic application scenario diagram of a log file processing method provided in the present embodiment. The user logs into the log processing server 12 through an application client installed in the terminal device 11, wherein the application client may be a browser of a web page or an application client installed in a terminal device, such as a mobile phone, a tablet computer, etc. The terminal device 11 and the log processing server 12 are in communication connection through a network, which may be a local area network, a cellular network, a wide area network, or the like. The terminal device 11 may be a portable device (e.g., a mobile phone, a tablet, a notebook, etc.), a personal computer (PC, personal Computer), and the log processing server 12 may be any device capable of providing internet services.
The log processing server 12 may provide a log file processing service for a user, for example, the user submits different operation requests through a client installed in the terminal device 11, the client generates different program running requests corresponding to the received different operation requests from the user and transmits the different program running requests to the log file processing server 12, the log file processing server 12 initiates different programs corresponding to the received different program running requests, in the process of running the program, the log processing server 12 extracts log statements stored in the code segments for code segments generated in the process of running the program, determines a variable data part and a fixed data part included in the log statements and identifiers corresponding to the log statements, determines the position of the variable data in the log statements based on the data name of the variable data part in the log statements, sets a unified data identifier, generates target log statements using the variable data part and the identifiers to replace the log statements, completes compression and encryption processing of the log statements, stores the fixed data part and the identifiers in a reduction table as a process of running the program, extracts log statements stored in the code segments, obtains a fixed log statement map file by parsing the fixed data corresponding to the fixed data part of the fixed data, obtains the fixed log statement, obtains a fixed log statement identifier by parsing the fixed log statement file by the fixed data, obtains the fixed log statement identifier by parsing the fixed log statement, and obtains the fixed log statement identifier by setting the fixed data, and the target log statement in the replaced log file is replaced by the log statement, so that the log statement is restored, the log file is restored further, and then the server sends the restored log file to the client, so that a user can acquire the needed log file, the safety of the log file is improved, and the consumption of computational resources is reduced.
In the following, a log file processing method disclosed in the embodiments of the present disclosure is described in detail, where an execution body of the log file processing method provided in the embodiments of the present disclosure is generally a computing device with a certain computing capability, and the computing device includes, for example: the terminal device, or server or other processing device, may be a User Equipment (UE), mobile device, user terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant, PDA), handheld device, computing device, vehicle mounted device, wearable device, etc. In some possible implementations, the log file processing method may be implemented by way of a processor invoking computer readable instructions stored in a memory.
The log file processing method provided by the embodiment of the present disclosure is described below by taking an execution body as a computer device as an example.
As shown in fig. 2, a flowchart of a log file processing method according to an embodiment of the disclosure may include the following steps:
s201: the method comprises the steps of obtaining a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part.
In this step, in order to ensure the security of the log file generated during the running process of the program and to avoid the log file collected at the client side from being excessively large, the log file processing server 12 compresses and encrypts the generated log file.
In particular, during the running process of the program, the engineering files including the log files and other data files may be generated, so that the log file processing server first needs to obtain the log files stored in the engineering files. As shown in fig. 3, a schematic diagram of an engineering file provided in an embodiment of the disclosure is shown, where a white box portion indicates a LOG file included in the engineering file, and the rest is other data files included in the engineering file, and when implemented, a format of the LOG file included in the engineering file is fixed, and in fig. 3, a LOG statement starts with a LOG; the method comprises the steps that the format of the tail end appears, based on the format, preset format storage information can be set, a log file processing server carries out regular matching on engineering files by using regular expressions which are the same as the preset format storage information, a stored log file can be obtained, further, by using the log file, log sentences in the log file can be determined, then the log sentences are analyzed again by using the regular expressions, a fixed data part and a variable data part in the log sentences can be determined, and in addition, when the log sentences stored in the log file are obtained, identifiers corresponding to the log sentences can be generated in a self-increasing mode according to the sequence of the obtained log sentences.
Taking the code segment included in the engineering file shown in fig. 3 as an example, the LOG file processing server matches the LOG file in the engineering file according to the stored information ("LOG") in the preset format, to obtain a LOG file LOG (ls_info, app_trival_module) < < "> Request video stream ssrc: "< < ssrc < <" width: "< < width < <" height: "< < height; the method comprises the steps of carrying out a first treatment on the surface of the Further, it may be determined that the log statement in this log file is "" "Request video stream ssrc: "< < ssrc < <" width: "< < width < <" height: "< < height; "; then, the regular expression is utilized to analyze the data, and the unchanged data 'Request video stream ssrc' is determined: "< < <" width: "< < <" height: "<; "as a fixed data portion, the changed data" ssrc, width, height "is determined as a variable data portion, and in addition, assuming that the log statement obtained in this step is the first log statement of the log file, the identifier of this log statement can be determined to be 1.
S202: the log statements in the log file are replaced with the variable data portion and identifiers of the log statements.
In specific implementation, the generated log statement identifier and the determined variable data portion may be combined into a target log statement, in an implementation manner, the log statement identifier may be placed in front of the variable data portion, and taking the log statement in fig. 3 as an example, it may be determined that the combined target log statement is 1, ssrc, width, height; in another implementation manner, the log statement identifier may be placed at the rear of the variable data portion, and the embodiment of the disclosure is not limited with respect to the combination manner of the log statement identifier and the variable data portion in the embodiment of the disclosure, and further, the log statement in the obtained log file is replaced by the combined target log statement.
For example, for the log statement "" "Request video stream ssrc" included in fig. 3: "< < ssrc < <" width: "< < width < <" height: "< < height; and replacing to obtain the replaced target log statement as 1, ssrc, width and height.
S203: the identifiers of the fixed data portions and the log statements are stored in a log statement mapping table that is used to restore the identifiers of the variable data portions and the log statements in the replaced log file to log statements.
In this step, in order to restore the replaced log file, a log statement mapping table for storing identifiers of the fixed data portion and the log statement needs to be generated.
In specific implementation, based on the determined data name corresponding to the variable data portion, determining the position of the variable data in the log statement, and then replacing the variable data portion at the corresponding position in the log statement with a unified data identifier, for example, the unified data identifier may be a placeholder, further, storing the replaced log statement as a fixed data portion and an identifier correspondingly generated in the process of obtaining the log statement as a whole in a log statement mapping table, taking the log statement included in fig. 3 as an example, and using the placeholder to replace the log statement as "Request video stream ssrc: "<%d <" "width": "<%d <" "height": "<% d; the method comprises the steps of carrying out a first treatment on the surface of the Where "%d" represents the placeholder, the data content in the log statement mapping table may be 1, "Request video stream ssrc: "<%d <" "width": "<%d <" "height": "<% d; . Further, after replacing the obtained log statement with the target log statement, the log file processing server compiles the engineering file of the replaced log file normally to generate a preprocessing program only containing the target log statement. Based on the method, even if the preprocessing program is leaked, the analyzed log statement only comprises an identifier and a variable data part, no associated context information exists, and if a log statement mapping table which is not generated is used as a reference, the acquired identifier and the variable data part have no practical meaning, so that the leakage of user information is not caused, and the safety of the log file is ensured.
Further, when the user needs to acquire the stored log file, submitting an operation instruction for the replaced log file to the client, and sending the operation instruction for the replaced log file to the log file processing server by the client according to the operation instruction, wherein the log file processing server further responds to the received operation instruction for the replaced log file to restore the replaced log statement.
As shown in fig. 4, a flowchart for restoring a replaced log statement provided in an embodiment of the disclosure may include the following steps:
s401: in response to an execution instruction for the replaced log file, an identifier of the log statement is obtained from the replaced log file.
In the specific implementation, the log file processing server responds to an operation instruction of the replaced log file, analyzes the generated preprocessing program, can acquire each target log statement in the replaced log file, matches each acquired target log statement by using a regular expression, and can acquire an identifier of the target log statement, wherein the identifier of the target log statement is an identifier of the log statement acquired in the replaced log file. In addition, there is an identifier representing an unsubstituted log statement in the generated log statement map table in correspondence with the obtained identifier of each target log statement.
S402: based on the identifier of the log statement, a fixed data portion of the log statement is obtained from the log statement mapping table.
In particular, according to the identifier of the obtained log statement, an identifier corresponding to the identifier of the obtained log statement is matched in the generated log statement mapping table, and based on the identifier, the fixed data part of the unsubstituted log statement stored in the log statement mapping table can be obtained.
S403: the variable data portion in the replaced log file and the identifier of the log statement are restored to the log statement based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file.
In the specific implementation, according to the fixed data portion of the log statement stored in the log statement mapping table determined in S402, a unified data identification position included in the fixed data portion may be determined, in addition, according to the extracted identifier of each target log statement included in each target log statement, a variable data portion corresponding to the target log statement may be determined, according to the determined unified data identification position, each unified data identification is replaced by the variable data portion, an unremoved log statement may be obtained, further, an unremoved log file may be obtained, and then, according to the extracted unified data identification position, the target log statement in the replaced log file may be replaced by the log statement of the unremoved log file, based on which the log statement may be restored.
For example, taking the target log statement obtained after parsing as "1, xxx,640, 480", matching the target log statement by using a regular expression, an identifier 1 of the target log statement may be obtained, matching from the generated log statement mapping table according to the identifier 1, determining that an identifier consistent with the identifier 1 exists, and further determining a fixed data portion "" "Request video stream ssrc" stored in the log statement mapping table according to the identifier 1: "<%d <" "width": "<%d <" "height": "<% d; then according to the analysis of the target log statement "1, XXX,640, 480", the variable data portions "XXX,640, 480" in the target log statement are obtained, and each placeholder is replaced by the variable data portions "XXX,640, 480" according to the positions of the placeholders in the fixed data portions, so that a log file "" Request video stream ssrc which is not replaced is obtained: "< < XXX < <" width ": "< <640< <" height: "< <480; "further, using" "" Request video stream ssrc: "< < XXX < <" width ": "< <640< <" height: "< <480; "replace target log statement" 1, xxx,640, 480 "to restore the log file. In addition, after the log file processor restores the log file obtained by analysis, the log file is sent to the client, so that the user can acquire the needed log file at the client.
The embodiment of the disclosure provides a log file processing method, a log file processor processes engineering files generated in the running process of a program, matches with regular expressions, acquires log files stored in the engineering files, determines log sentences in the log files according to the log files, determines fixed data parts, variable data parts and identifiers in the log sentences, completes setting of the fixed data parts by using a method of setting uniform data identifiers at positions corresponding to the variable data parts, then obtains target log sentences by using the identifiers and alternative log sentences of the variable data, realizes compression and encryption of the log files, reduces the size of the log files, improves the efficiency of file processing, generates a log sentence mapping table based on the fixed data parts and the identifiers, further determines the identifiers and the variable data parts in the target log sentences by analyzing the replaced log files when a user has a requirement of acquiring the replaced log files, obtains the fixed data parts of the corresponding non-replaced log files by using the identifiers in the fixed log sentences, and then obtains the alternative log sentences of the non-replaced log sentences by using the fixed data parts of the non-replaced log sentences, and then obtains the alternative log sentences by using the fixed data of the non-replaced log sentences, the encrypted log statement is decrypted, and complexity of log file restoration is reduced.
It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.
Based on the same inventive concept, the embodiments of the present disclosure further provide a log file processing device corresponding to the log file processing method, and since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to that of the log file processing method in the embodiments of the present disclosure, the implementation of the device may refer to the implementation of the method, and the repetition is omitted.
As shown in fig. 5, a schematic diagram of a log file processing apparatus according to an embodiment of the disclosure includes:
an obtaining module 501, configured to obtain a log statement in a log file, where the log statement includes a fixed data portion and a variable data portion;
a replacing module 502, configured to replace a log statement in the log file with the variable data portion and an identifier of the log statement;
a storage module 503, configured to store the fixed data portion and the identifier of the log statement in a log statement mapping table, where the log statement mapping table is configured to restore the variable data portion and the identifier of the log statement in the replaced log file to the log statement.
In one possible embodiment, the apparatus further comprises:
a restoration module 504, configured to obtain, in response to an execution instruction for the replaced log file, an identifier of the log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file, the variable data portion of the replaced log file and the identifier of the log statement are restored to the log statement.
In a possible implementation manner, the obtaining module 501 is specifically configured to obtain the log statement in the log file based on the preset storage format information of the log statement.
In a possible implementation manner, the storage module 503 is specifically configured to determine a location of the variable data portion in the log statement based on a data name of the variable data portion in the log statement;
a unified data identifier is arranged at a position corresponding to the variable data part;
and taking the log statement with the unified data mark as a fixed data part, and storing the log statement and the identifier of the log statement in a log statement mapping table.
In a possible implementation manner, the replacing module 502 is configured to determine, based on an identifier of the log statement, a location of the log statement in the log file;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file with the target log statement based on the position of the log statement in the log file.
In a possible implementation manner, the replacing module 502 is specifically configured to obtain a target log statement in the replaced log file;
extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, the restoration module 504 is specifically configured to obtain, based on the identifier of the log statement, a variable data portion in the target log statement from the replaced log file, and obtain, from the log statement mapping table, a fixed data portion in the corresponding log statement;
replacing the unified data identifier in the fixed data part by the variable data part to obtain the log statement;
Replacing the target log statement in the replaced log file with the log statement to restore the log statement in the log file.
The process flow of each module in the apparatus and the interaction flow between the modules may be described with reference to the related descriptions in the above method embodiments, which are not described in detail herein.
The embodiment of the disclosure further provides a computer device, as shown in fig. 6, which is a schematic structural diagram of the computer device provided by the embodiment of the disclosure, including:
a processor 61 and a memory 62; the memory 62 stores machine readable instructions executable by the processor 61, the processor 61 being configured to execute the machine readable instructions stored in the memory 62, the machine readable instructions when executed by the processor 61, the processor 61 performing the steps of: s201: acquiring a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part; s202: replacing the log statements in the log file with identifiers of the variable data portion and the log statements S203: the identifiers of the fixed data portions and the log statements are stored in a log statement mapping table that is used to restore the identifiers of the variable data portions and the log statements in the replaced log file to log statements.
The memory 62 includes a memory 621 and an external memory 622; the memory 621 is also referred to as an internal memory, and is used for temporarily storing operation data in the processor 61 and data exchanged with the external memory 622 such as a hard disk, and the processor 61 exchanges data with the external memory 622 via the memory 621.
In a possible implementation manner, the instructions executed by the processor 61 further include:
in response to an operation instruction to the replaced log file, acquiring an identifier of a log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file, the variable data portion of the replaced log file and the identifier of the log statement are restored to the log statement.
In a possible implementation manner, in the instructions executed by the processor 61, the obtaining a log statement in the log file includes:
and acquiring the log statement in the log file based on the preset storage format information of the log statement.
In a possible implementation manner, in the instructions executed by the processor 61, the storing the fixed data portion and the identifier of the log statement in the log statement mapping table includes:
determining the position of a variable data part in a log statement based on the data name of the variable data part in the log statement;
a unified data identifier is arranged at a position corresponding to the variable data part;
and taking the log statement with the unified data mark as a fixed data part, and storing the log statement and the identifier of the log statement in a log statement mapping table.
In a possible implementation manner, in the instructions executed by the processor 61, the replacing the log statement in the log file with the variable data portion and the identifier of the log statement includes:
determining a location of the log statement in the log file based on an identifier of the log statement;
combining the identifier of the log statement and the variable data part to obtain a target log statement;
and replacing the log statement in the log file with the target log statement based on the position of the log statement in the log file.
In a possible implementation manner, in the instructions executed by the processor 61, the obtaining the identifier of the log statement from the replaced log file includes:
acquiring a target log statement in the replaced log file;
extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
In a possible implementation manner, in the instructions executed by the processor 61, the restoring the variable data portion in the replaced log file and the identifier of the log statement into the log statement specifically includes:
based on the identifier of the log statement, acquiring a variable data part in the target log statement from the replaced log file, and acquiring a fixed data part in the corresponding log statement from the log statement mapping table;
replacing the unified data identifier in the fixed data part by the variable data part to obtain the log statement;
replacing the target log statement in the replaced log file with the log statement to restore the log statement in the log file.
The specific execution process of the above instruction may refer to the steps of the log file processing method described in the embodiments of the present disclosure, which is not described herein again.
The disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the log file processing method described in the above method embodiments. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The computer program product of the log file processing method provided in the embodiments of the present disclosure includes a computer readable storage medium storing program code, where the program code includes instructions for executing the steps of the log file processing method described in the above method embodiments, and specifically, reference may be made to the above method embodiments, which are not repeated herein. The computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that: the foregoing examples are merely specific embodiments of the present disclosure, and are not intended to limit the scope of the disclosure, but the present disclosure is not limited thereto, and those skilled in the art will appreciate that while the foregoing examples are described in detail, it is not limited to the disclosure: any person skilled in the art, within the technical scope of the disclosure of the present disclosure, may modify or easily conceive changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features thereof; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the disclosure, and are intended to be included within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims (7)

1. A log file processing method, comprising:
acquiring a log statement in a log file, wherein the log statement comprises a fixed data part and a variable data part;
determining a location of the log statement in the log file based on an identifier of the log statement; the identifier is used for identifying the position of the log statement in the log file;
Combining the identifier of the log statement and the variable data part to obtain a target log statement;
replacing the log statement in the log file with the target log statement based on the position of the log statement in the log file;
determining the position of a variable data part in a log statement based on the data name of the variable data part in the log statement;
a unified data identifier is arranged at a position corresponding to the variable data part;
taking the log statement with the unified data mark as a fixed data part and storing the fixed data part and the identifier of the log statement in a log statement mapping table, wherein the log statement mapping table is used for restoring the variable data part in the replaced log file and the identifier of the log statement into the log statement; the restoring the variable data part in the replaced log file and the identifier of the log statement into the log statement specifically comprises:
based on the identifier of the log statement, acquiring a variable data part in a target log statement from the replaced log file, and acquiring a fixed data part in a corresponding log statement from the log statement mapping table;
Replacing the unified data identifier in the fixed data part by the variable data part to obtain the log statement;
replacing the target log statement in the replaced log file with the log statement to restore the log statement in the log file.
2. The log file processing method as set forth in claim 1, further comprising:
in response to an operation instruction to the replaced log file, acquiring an identifier of a log statement from the replaced log file;
acquiring a fixed data part of the log statement from the log statement mapping table based on the identifier of the log statement;
based on the fixed data portion of the log statement and the variable data portion of the log file stored in the replaced log file, the variable data portion of the replaced log file and the identifier of the log statement are restored to the log statement.
3. The method for processing a log file according to claim 1, wherein the obtaining the log statement in the log file comprises:
and acquiring the log statement in the log file based on the preset storage format information of the log statement.
4. The log file processing method as claimed in claim 2, wherein acquiring the identifier of the log statement from the replaced log file comprises:
acquiring a target log statement in the replaced log file;
extracting the identifier of the target log statement from the target log statement, and taking the obtained identifier as the identifier of the log statement.
5. A log file processing apparatus, comprising:
the acquisition module is used for acquiring log sentences in the log file, wherein the log sentences comprise a fixed data part and a variable data part;
a replacement module, configured to determine a location of the log statement in the log file based on an identifier of the log statement; the identifier is used for identifying the position of the log statement in the log file; combining the identifier of the log statement and the variable data part to obtain a target log statement; replacing the log statement in the log file with the target log statement based on the position of the log statement in the log file;
the storage module is used for determining the position of the variable data part in the log statement based on the data name of the variable data part in the log statement; a unified data identifier is arranged at a position corresponding to the variable data part; taking the log statement with the unified data mark as a fixed data part and storing the fixed data part and the identifier of the log statement in a log statement mapping table, wherein the log statement mapping table is used for restoring the variable data part in the replaced log file and the identifier of the log statement into the log statement;
The restoring module is specifically configured to, when the variable data portion in the replaced log file and the identifier of the log statement are restored to the log statement:
based on the identifier of the log statement, acquiring a variable data part in a target log statement from the replaced log file, and acquiring a fixed data part in a corresponding log statement from the log statement mapping table;
replacing the unified data identifier in the fixed data part by the variable data part to obtain the log statement;
replacing the target log statement in the replaced log file with the log statement to restore the log statement in the log file.
6. A computer device, comprising: a processor, a memory storing machine readable instructions executable by the processor for executing machine readable instructions stored in the memory, which when executed by the processor, perform the steps of the log file processing method of any of claims 1 to 4.
7. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when run by a computer device, performs the steps of the log file processing method according to any of claims 1 to 4.
CN202011303567.8A 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium Active CN112286896B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011303567.8A CN112286896B (en) 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011303567.8A CN112286896B (en) 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112286896A CN112286896A (en) 2021-01-29
CN112286896B true CN112286896B (en) 2024-02-06

Family

ID=74399358

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011303567.8A Active CN112286896B (en) 2020-11-19 2020-11-19 Log file processing method and device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112286896B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113179265A (en) * 2021-04-26 2021-07-27 湖南蚁坊软件股份有限公司 Log compression encryption method, device, equipment and storage medium
CN113691610B (en) * 2021-08-20 2024-04-09 Oppo广东移动通信有限公司 Data acquisition method and device, electronic equipment and storage medium
DE102022002450A1 (en) 2022-07-05 2024-01-11 Mercedes-Benz Group AG Method for processing log files, data processing system and vehicle

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5790508A (en) * 1996-08-20 1998-08-04 Industrial Technology Research Institute Multidisk changer
CN101047921A (en) * 2006-03-07 2007-10-03 三星电子株式会社 Apparatus and method of providing fast menus in menu screen of a mobile communication terminal
CN103226509A (en) * 2013-04-08 2013-07-31 上海华力微电子有限公司 Method for automatically analyzing system log
CN103337113A (en) * 2013-07-17 2013-10-02 广州广电运通金融电子股份有限公司 Method and device for intelligently analyzing electronic day-to-day journals, as well as processor
CN103412924A (en) * 2013-08-12 2013-11-27 东软集团股份有限公司 Log multi-language query method and system
CN104331487A (en) * 2014-11-13 2015-02-04 携程计算机技术(上海)有限公司 Method and device for processing logs
CN106484742A (en) * 2015-09-01 2017-03-08 北京国双科技有限公司 Log processing method and device
CN109684290A (en) * 2018-12-20 2019-04-26 东软集团股份有限公司 Log storing method, device, equipment and computer readable storage medium
CN109791520A (en) * 2016-09-30 2019-05-21 亚马逊技术股份有限公司 The log recording of the Space Coupling of physical medium perception and playback
CN111190873A (en) * 2019-12-24 2020-05-22 同济大学 Log mode extraction method and system for log training of cloud native system
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN111930701A (en) * 2020-08-13 2020-11-13 工银科技有限公司 Log structured processing method and device

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5790508A (en) * 1996-08-20 1998-08-04 Industrial Technology Research Institute Multidisk changer
CN101047921A (en) * 2006-03-07 2007-10-03 三星电子株式会社 Apparatus and method of providing fast menus in menu screen of a mobile communication terminal
CN103226509A (en) * 2013-04-08 2013-07-31 上海华力微电子有限公司 Method for automatically analyzing system log
CN103337113A (en) * 2013-07-17 2013-10-02 广州广电运通金融电子股份有限公司 Method and device for intelligently analyzing electronic day-to-day journals, as well as processor
CN103412924A (en) * 2013-08-12 2013-11-27 东软集团股份有限公司 Log multi-language query method and system
CN104331487A (en) * 2014-11-13 2015-02-04 携程计算机技术(上海)有限公司 Method and device for processing logs
CN106484742A (en) * 2015-09-01 2017-03-08 北京国双科技有限公司 Log processing method and device
CN109791520A (en) * 2016-09-30 2019-05-21 亚马逊技术股份有限公司 The log recording of the Space Coupling of physical medium perception and playback
CN109684290A (en) * 2018-12-20 2019-04-26 东软集团股份有限公司 Log storing method, device, equipment and computer readable storage medium
CN111190873A (en) * 2019-12-24 2020-05-22 同济大学 Log mode extraction method and system for log training of cloud native system
CN111708860A (en) * 2020-06-15 2020-09-25 北京优特捷信息技术有限公司 Information extraction method, device, equipment and storage medium
CN111930701A (en) * 2020-08-13 2020-11-13 工银科技有限公司 Log structured processing method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于差分压缩的大规模日志压缩系统;唐球 等;《通信学报》;第36卷(第Z1期);197-202 *

Also Published As

Publication number Publication date
CN112286896A (en) 2021-01-29

Similar Documents

Publication Publication Date Title
CN112286896B (en) Log file processing method and device, computer equipment and storage medium
JP6926429B2 (en) Data processing equipment, data processing methods, and programs
CN107665233B (en) Database data processing method and device, computer equipment and storage medium
CN108512830B (en) Information encryption processing method and device, computer equipment and storage medium
CN106919555B (en) System and method for field extraction of data contained within a log stream
CN104680039B (en) A kind of data guard method and device of application program installation kit
CN103123588A (en) Method and system for obtaining difference upgrade patch
CN108399319B (en) Source code protection method, application server and computer readable storage medium
KR20170068814A (en) Apparatus and Method for Recognizing Vicious Mobile App
Chang et al. Forensic analysis of LINE messenger on android
CN111949974A (en) Authentication method and device, computer equipment and storage medium
CN110968847A (en) File watermark hiding and analyzing method, device, equipment and storage medium
CN109241707A (en) Application program obscures method, apparatus and server
CN103246540A (en) Update method and update device of application program
CN114285575A (en) Image encryption and decryption method and device, storage medium and electronic device
CN112437060A (en) Data transmission method and device, computer equipment and storage medium
CN111737718A (en) Encryption and decryption method and device for jar packet, terminal equipment and storage medium
CN112287371B (en) Method and device for storing industrial data and computer equipment
CN111475690B (en) Character string matching method and device, data detection method and server
CN113642025A (en) Interface data processing method, device, equipment and storage medium
CN110930193B (en) Advertisement conversion rate evaluation method, advertisement conversion rate evaluation device, computer equipment and storage medium
CN112436943A (en) Request deduplication method, device, equipment and storage medium based on big data
CN110990056A (en) Reverse analysis method, device, electronic equipment and storage medium
CN113778826B (en) Log processing method and device
CN114036364B (en) Method, apparatus, device, medium, and system for identifying crawlers

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant