CN113407508A - Method, system, equipment and medium for compressing log file - Google Patents
Method, system, equipment and medium for compressing log file Download PDFInfo
- Publication number
- CN113407508A CN113407508A CN202110656324.0A CN202110656324A CN113407508A CN 113407508 A CN113407508 A CN 113407508A CN 202110656324 A CN202110656324 A CN 202110656324A CN 113407508 A CN113407508 A CN 113407508A
- Authority
- CN
- China
- Prior art keywords
- file
- compressed
- log
- storage medium
- log file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000007906 compression Methods 0.000 claims abstract description 38
- 230000006835 compression Effects 0.000 claims abstract description 33
- 230000004044 response Effects 0.000 claims abstract description 11
- 230000001360 synchronised effect Effects 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 14
- 238000012795 verification Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 7
- 230000004048 modification Effects 0.000 claims description 7
- 238000012986 modification Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 description 9
- 230000000694 effects Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000010076 replication Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000002035 prolonged effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 230000006837 decompression Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000007730 finishing process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a system, equipment and a storage medium for compressing log files, wherein the method comprises the following steps: in response to compressing the log file, confirming a currently used synchronization mode through a configuration file; determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names; coding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium; and modifying the index file of the log file according to the address of the storage medium. According to the method, different keywords are adopted according to different synchronization modes, so that corresponding log files are matched, diversified log file compression modes are formed, and the applicability is stronger; check information is added in the compression process of the log file, so that human errors can be avoided.
Description
Technical Field
The present invention relates to the field of log file compression, and more particularly, to a method, a system, a computer device, and a readable medium for log file compression.
Background
The high availability of the system is always an important index for measuring the reliability of a software system, mysql is used as a common database, and the disaster tolerance capability of the mysql is a very important part for measuring the high availability of the system. The disaster recovery of the database is usually realized by a high availability environment composed of a plurality of database nodes, and in order to solve the Data synchronization among the nodes, the existing technologies include an asynchronous Replication technology, a semi-synchronous Replication technology and a Group synchronization technology (MySQL Group Replication), which are realized based on binlog of MySQL, which is a binary log and is a binary log file, and the file records all DML (Data management Language) operations of MySQL. Data recovery, incremental backup, primary master replication and master slave replication, etc. may be performed through binlog logs.
Binlog records the modification operation of the database in the running process of the database, and the Binlog is copied and played back to the standby node, so that the data can be synchronized at the standby node. The Binlog records detailed changes of the database, and the size of the Binlog file is rapidly increased under the condition of large data volume, so that hard disk resources are greatly consumed. Meanwhile, if mysql has a problem, the binlog needs to be saved when recovery is needed, which results in that only log files of a short time can be saved in the mysql environment under the condition of a large resource amount, so that in some cases, when mysql is recovered or a new node is added, part of the binlog is deleted, and thus, the transactions corresponding to the GTID (global transaction identifier) are deleted and synchronization fails.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method, a system, a computer device, and a computer-readable storage medium for compressing log files, in which different keywords are adopted according to different synchronization modes, and corresponding log files are matched according to the keywords for compression, so that a diversified way of compressing log files is formed, and applicability is stronger; check information is added in the compression process of the log file, so that human errors can be avoided; and moreover, the occupation of hard disk resources is effectively reduced, the log storage time can be effectively prolonged under the environment of limited hard disk resources, and the effect of mysql synchronization in actual use is improved.
Based on the above purpose, an aspect of the embodiments of the present invention provides a method for compressing a log file, including the following steps: in response to compressing the log file, confirming a currently used synchronization mode through a configuration file; determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names; coding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium; and modifying the index file of the log file according to the address of the storage medium.
In some embodiments, the determining a keyword according to the currently used synchronization method includes: and acquiring the mapping relation of the synchronous mode, and determining the keywords corresponding to the synchronous mode according to the mapping relation.
In some embodiments, the matching the file name to be compressed according to the keyword includes: and matching a rule according to the keywords, and determining the name of the file to be compressed according to the rule.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing medium access control information in the length of the first buffer area of the log file.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing the service identifier in the last byte of the log file.
In some embodiments, the writing the compressed file into a preset storage medium includes: reading the medium access control information of the compressed file, and judging whether a first compressed file which is the same as the medium access control information of the compressed file exists in the storage medium; and in response to the existence of a first compressed file in the storage medium, the first compressed file being identical to the medium access control information of the compressed file, placing the compressed file and the first compressed file in the same storage queue.
In some embodiments, the modifying the index file of the log file according to the address of the storage medium includes: and acquiring the address of the storage queue, and replacing the address of the index file corresponding to the log file by using the address.
In another aspect of the embodiments of the present invention, a system for compressing a log file is provided, including: the synchronization module is configured to respond to the compression of the log file and confirm the synchronization mode used currently through the configuration file; the matching module is configured to determine keywords according to the currently used synchronization mode, match the names of the log files to be compressed according to the keywords, and determine the corresponding log files according to the names; the compression module is configured to encode the log file, add check information to generate a compressed file, and write the compressed file into a preset storage medium; and the modification module is configured to modify the index file of the log file according to the address of the storage medium.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method as above.
In a further aspect of the embodiments of the present invention, a computer-readable storage medium is also provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has the following beneficial technical effects: different keywords are adopted according to different synchronization modes, and corresponding log files are matched according to the keywords for compression, so that a diversified mode for compressing the log files is formed, and the applicability is stronger; check information is added in the compression process of the log file, so that human errors can be avoided; and moreover, the occupation of hard disk resources is effectively reduced, the log storage time can be effectively prolonged under the environment of limited hard disk resources, and the effect of mysql synchronization in actual use is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a diagram illustrating a method for compressing a log file according to an embodiment of the present invention;
FIG. 2 is a schematic hardware structure diagram of an embodiment of a computer device for log file compression according to the present invention;
FIG. 3 is a schematic diagram of an embodiment of a computer storage medium for log file compression according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In a first aspect of the embodiments of the present invention, an embodiment of a method for compressing a log file is provided. Fig. 1 is a schematic diagram illustrating an embodiment of a method for compressing a log file according to the present invention.
As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, responding to the compression of the log file, and confirming the synchronization mode used currently through the configuration file;
s2, determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names;
s3, encoding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium; and
and S4, modifying the index file of the log file according to the address of the storage medium.
And in response to compressing the log file, confirming the currently used synchronization mode through the configuration file. When the log file needs to be compressed, whether the current environment is a synchronous environment is judged, if not, the process is ended, and if so, the type of the current synchronous environment is judged. The synchronization mode currently used by MySQL may be confirmed by a MySQL configuration file or a MySQL attribute. And judging whether the current execution flow starts compression, if so, reading parameters, and if not, entering the ending flow. And if the current flow starts compression, selecting the log files to be compressed according to different synchronization modes.
And determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names.
In some embodiments, the determining a keyword according to the currently used synchronization method includes: and acquiring the mapping relation of the synchronous mode, and determining the keywords corresponding to the synchronous mode according to the mapping relation. A mapping relationship between each synchronization pattern and a keyword may be set, and when a certain synchronization pattern is input, a corresponding keyword may be determined by the mapping relationship.
Keywords corresponding to different synchronization modes can be preset in the configuration file, and the name of the log file to be compressed corresponding to each keyword is set. For example, the "regs" field may be used to perform regular matching on the file name, so as to obtain the file to be processed. In addition, master-slave copy can be set to compress the binlog and the replayog log files, and if the copy is MGR (state machine copy), the binlog and the group _ reproduction _ application log files can be compressed.
In some embodiments, the matching the file name to be compressed according to the keyword includes: and matching a rule according to the keywords, and determining the name of the file to be compressed according to the rule.
The code to set the rules in the configuration file is as follows:
wherein the 'msrep' rule binlog and relaylog are compressed, and the 'mgrep' rule processes the binlog and the group _ reproduction _ applicator log. The configuration for selecting the compressed file is expandable, the embodiment of the invention is set as a default matching rule for master-slave copy and MGR, and meanwhile, other log files needing to be compressed can be modified in the configuration file to realize the functions of expanding and customizing the compressed file by a user, namely, the user in the code is a rule customized by the user to compress the mysql-slow-query.
And coding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing medium access control information in the length of the first buffer area of the log file.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing the service identifier in the last byte of the log file.
When the number of log files is large, the log files are generally required to be backed up to a special storage device, mysql can check some data when the data is restored, such as server id (service identifier) in binlog, errors of human factors are easy to occur when the data is transmitted between devices.
In order to prevent information leakage, the byte code is encrypted, for example, simple byte offset is used, mac information is written in the length of the first buffer, then log file bytes are read, and server id is added to the last bit of the bytes for verification, so that file damage caused in the transmission process is prevented, verification is performed when data is recovered, and similar errors are avoided. The mac information is used for verifying whether the device generating the compressed file and the device restored by the user are the same device, and the server id field is used for verifying whether the server id in the file is consistent with the server id in the mysql configuration, so that the binlog can be normally used by the mysql. The user can define the compression algorithm to be used, such as "lz 77" and "lzma" in the above example, according to the characteristics of the file, and can also use the customized compression algorithm to process, only the "method" field needs to be modified.
And modifying the index file of the log file according to the address of the storage medium.
In some embodiments, the writing the compressed file into a preset storage medium includes: reading the medium access control information of the compressed file, and judging whether a first compressed file which is the same as the medium access control information of the compressed file exists in the storage medium; and in response to the existence of a first compressed file in the storage medium, the first compressed file being identical to the medium access control information of the compressed file, placing the compressed file and the first compressed file in the same storage queue.
In some embodiments, the modifying the index file of the log file according to the address of the storage medium includes: and acquiring the address of the storage queue, and replacing the address of the index file corresponding to the log file by using the address.
The decompression synchronization process comprises the following steps:
1. when decompression and synchronization are needed, judging whether the current environment is a synchronous environment, if not, entering a finishing process, and if so, judging the type of the current synchronous environment;
2. analyzing current log information, checking whether a log file is complete, if the log file is complete, checking the synchronization type, the equipment information and the server id of the log file to judge whether the log file is matched with the current environment, if not, entering an ending process, and if so, entering a decompressing process;
3. decompressing the current compressed file through a preset compression algorithm, and modifying the index file.
According to the embodiment of the invention, different keywords are adopted according to different synchronization modes, and corresponding log files are matched according to the keywords for compression, so that a diversified mode of compressing the log files is formed, and the applicability is stronger; check information is added in the compression process of the log file, so that human errors can be avoided; and moreover, the occupation of hard disk resources is effectively reduced, the log storage time can be effectively prolonged under the environment of limited hard disk resources, and the effect of mysql synchronization in actual use is improved.
It should be particularly noted that, the steps in the embodiments of the method for compressing a log file described above may be mutually intersected, replaced, added, or deleted, and therefore, these methods for compressing a log file, which are reasonably arranged and combined, should also belong to the scope of the present invention, and should not limit the scope of the present invention to the embodiments.
In view of the above object, according to a second aspect of the embodiments of the present invention, there is provided a system for compressing a log file, including: the synchronization module is configured to respond to the compression of the log file and confirm the synchronization mode used currently through the configuration file; the matching module is configured to determine keywords according to the currently used synchronization mode, match the names of the log files to be compressed according to the keywords, and determine the corresponding log files according to the names; the compression module is configured to encode the log file, add check information to generate a compressed file, and write the compressed file into a preset storage medium; and the modification module is configured to modify the index file of the log file according to the address of the storage medium.
In some embodiments, the matching module is configured to: and acquiring the mapping relation of the synchronous mode, and determining the keywords corresponding to the synchronous mode according to the mapping relation.
In some embodiments, the matching module is configured to: and matching a rule according to the keywords, and determining the name of the file to be compressed according to the rule.
In some embodiments, the compression module is configured to: and writing medium access control information in the length of the first buffer area of the log file.
In some embodiments, the compression module is configured to: and writing the service identifier in the last byte of the log file.
In some embodiments, the compression module is configured to: reading the medium access control information of the compressed file, and judging whether a first compressed file which is the same as the medium access control information of the compressed file exists in the storage medium; and in response to the existence of a first compressed file in the storage medium, the first compressed file being identical to the medium access control information of the compressed file, placing the compressed file and the first compressed file in the same storage queue.
In some embodiments, the modification module is configured to: and acquiring the address of the storage queue, and replacing the address of the index file corresponding to the log file by using the address.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, responding to the compression of the log file, and confirming the synchronization mode used currently through the configuration file; s2, determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names; s3, encoding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium; and S4, modifying the index file of the log file according to the address of the storage medium.
In some embodiments, the determining a keyword according to the currently used synchronization method includes: and acquiring the mapping relation of the synchronous mode, and determining the keywords corresponding to the synchronous mode according to the mapping relation.
In some embodiments, the matching the file name to be compressed according to the keyword includes: and matching a rule according to the keywords, and determining the name of the file to be compressed according to the rule.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing medium access control information in the length of the first buffer area of the log file.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing the service identifier in the last byte of the log file.
In some embodiments, the writing the compressed file into a preset storage medium includes: reading the medium access control information of the compressed file, and judging whether a first compressed file which is the same as the medium access control information of the compressed file exists in the storage medium; and in response to the existence of a first compressed file in the storage medium, the first compressed file being identical to the medium access control information of the compressed file, placing the compressed file and the first compressed file in the same storage queue.
In some embodiments, the modifying the index file of the log file according to the address of the storage medium includes: and acquiring the address of the storage queue, and replacing the address of the index file corresponding to the log file by using the address.
According to the embodiment of the invention, different keywords are adopted according to different synchronization modes, and corresponding log files are matched according to the keywords for compression, so that a diversified mode of compressing the log files is formed, and the applicability is stronger; check information is added in the compression process of the log file, so that human errors can be avoided; and moreover, the occupation of hard disk resources is effectively reduced, the log storage time can be effectively prolonged under the environment of limited hard disk resources, and the effect of mysql synchronization in actual use is improved.
Fig. 2 is a schematic hardware structural diagram of an embodiment of the computer device for compressing log files according to the present invention.
Taking the apparatus shown in fig. 2 as an example, the apparatus includes a processor 201 and a memory 202, and may further include: an input device 203 and an output device 204.
The processor 201, the memory 202, the input device 203 and the output device 204 may be connected by a bus or other means, and fig. 2 illustrates the connection by a bus as an example.
The memory 202, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for compressing log files in the embodiments of the present application. The processor 201 executes various functional applications of the server and data processing, namely, implements the method of log file compression of the above-described method embodiment, by running the nonvolatile software program, instructions and modules stored in the memory 202.
The memory 202 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the method of log file compression, and the like. Further, the memory 202 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 202 may optionally include memory located remotely from processor 201, which may be connected to local modules via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 203 may receive information such as a user name and a password that are input. The output device 204 may include a display device such as a display screen.
Program instructions/modules corresponding to one or more methods of log file compression are stored in the memory 202 and, when executed by the processor 201, perform the method of log file compression in any of the method embodiments described above.
Any embodiment of the computer device executing the method for compressing the log file can achieve the same or similar effects as any corresponding embodiment of the method.
The invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the method as above.
Fig. 3 is a schematic diagram of an embodiment of a computer storage medium for compressing a log file according to the present invention. Taking the computer storage medium as shown in fig. 3 as an example, the computer-readable storage medium 3 stores a computer program 31 that executes a method of log file compression when executed by a processor.
The log file compression method comprises the following steps: in response to compressing the log file, confirming a currently used synchronization mode through a configuration file; determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names; coding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium; and modifying the index file of the log file according to the address of the storage medium.
In some embodiments, the determining a keyword according to the currently used synchronization method includes: and acquiring the mapping relation of the synchronous mode, and determining the keywords corresponding to the synchronous mode according to the mapping relation.
In some embodiments, the matching the file name to be compressed according to the keyword includes: and matching a rule according to the keywords, and determining the name of the file to be compressed according to the rule.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing medium access control information in the length of the first buffer area of the log file.
In some embodiments, the encoding the log file and adding the verification information to generate the compressed file includes: and writing the service identifier in the last byte of the log file.
In some embodiments, the writing the compressed file into a preset storage medium includes: reading the medium access control information of the compressed file, and judging whether a first compressed file which is the same as the medium access control information of the compressed file exists in the storage medium; and in response to the existence of a first compressed file in the storage medium, the first compressed file being identical to the medium access control information of the compressed file, placing the compressed file and the first compressed file in the same storage queue.
In some embodiments, the modifying the index file of the log file according to the address of the storage medium includes: and acquiring the address of the storage queue, and replacing the address of the index file corresponding to the log file by using the address.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and the program of the method for log file compression can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods as described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method for compressing log files, comprising the steps of:
in response to compressing the log file, confirming a currently used synchronization mode through a configuration file;
determining keywords according to the currently used synchronization mode, matching the names of the log files to be compressed according to the keywords, and determining the corresponding log files according to the names;
coding the log file, adding check information to generate a compressed file, and writing the compressed file into a preset storage medium; and
and modifying the index file of the log file according to the address of the storage medium.
2. The method of claim 1, wherein determining the key according to the currently used synchronization mode comprises:
and acquiring the mapping relation of the synchronous mode, and determining the keywords corresponding to the synchronous mode according to the mapping relation.
3. The method according to claim 1, wherein the matching the file name to be compressed according to the keyword comprises:
and matching a rule according to the keywords, and determining the name of the file to be compressed according to the rule.
4. The method of claim 1, wherein encoding the log file and adding verification information to generate a compressed file comprises:
and writing medium access control information in the length of the first buffer area of the log file.
5. The method of claim 4, wherein encoding the log file and adding the verification information to generate a compressed file comprises:
and writing the service identifier in the last byte of the log file.
6. The method according to claim 4, wherein writing the compressed file into a preset storage medium comprises:
reading the medium access control information of the compressed file, and judging whether a first compressed file which is the same as the medium access control information of the compressed file exists in the storage medium; and
and in response to the storage medium having a first compressed file which is the same as the medium access control information of the compressed file, placing the compressed file and the first compressed file in the same storage queue.
7. The method of claim 6, wherein modifying the index file of the log file according to the address of the storage medium comprises:
and acquiring the address of the storage queue, and replacing the address of the index file corresponding to the log file by using the address.
8. A system for log file compression, comprising:
the synchronization module is configured to respond to the compression of the log file and confirm the synchronization mode used currently through the configuration file;
the matching module is configured to determine keywords according to the currently used synchronization mode, match the names of the log files to be compressed according to the keywords, and determine the corresponding log files according to the names;
the compression module is configured to encode the log file, add check information to generate a compressed file, and write the compressed file into a preset storage medium; and
and the modification module is configured to modify the index file of the log file according to the address of the storage medium.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110656324.0A CN113407508B (en) | 2021-06-11 | 2021-06-11 | Method, system, equipment and medium for compressing log file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110656324.0A CN113407508B (en) | 2021-06-11 | 2021-06-11 | Method, system, equipment and medium for compressing log file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113407508A true CN113407508A (en) | 2021-09-17 |
| CN113407508B CN113407508B (en) | 2023-04-25 |
Family
ID=77683697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110656324.0A Active CN113407508B (en) | 2021-06-11 | 2021-06-11 | Method, system, equipment and medium for compressing log file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113407508B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1842021A (en) * | 2005-03-28 | 2006-10-04 | 华为技术有限公司 | Log information storage method |
| CN102290091A (en) * | 2011-09-02 | 2011-12-21 | 南京博同科技有限公司 | Complete protection method for moving hard disk log files |
| CN102970158A (en) * | 2012-11-05 | 2013-03-13 | 广东睿江科技有限公司 | Log storage and processing method and log server |
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| CN109857714A (en) * | 2018-12-24 | 2019-06-07 | 浪潮电子信息产业股份有限公司 | Journal obtaining method, device, electronic equipment and computer readable storage medium |
| US10740300B1 (en) * | 2017-12-07 | 2020-08-11 | Commvault Systems, Inc. | Synchronization of metadata in a distributed storage system |
| CN111708673A (en) * | 2020-06-15 | 2020-09-25 | 北京优特捷信息技术有限公司 | Log data compression method, device, equipment and storage medium |
-
2021
- 2021-06-11 CN CN202110656324.0A patent/CN113407508B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1842021A (en) * | 2005-03-28 | 2006-10-04 | 华为技术有限公司 | Log information storage method |
| CN102290091A (en) * | 2011-09-02 | 2011-12-21 | 南京博同科技有限公司 | Complete protection method for moving hard disk log files |
| CN102970158A (en) * | 2012-11-05 | 2013-03-13 | 广东睿江科技有限公司 | Log storage and processing method and log server |
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| US10740300B1 (en) * | 2017-12-07 | 2020-08-11 | Commvault Systems, Inc. | Synchronization of metadata in a distributed storage system |
| CN109857714A (en) * | 2018-12-24 | 2019-06-07 | 浪潮电子信息产业股份有限公司 | Journal obtaining method, device, electronic equipment and computer readable storage medium |
| CN111708673A (en) * | 2020-06-15 | 2020-09-25 | 北京优特捷信息技术有限公司 | Log data compression method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 张春玲等: "基于虚拟日志压缩的数据同步方案", 《计算机工程》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113407508B (en) | 2023-04-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220188003A1 (en) | Distributed Storage Method and Device | |
| CN109783014B (en) | Data storage method and device | |
| CN112636992B (en) | Dynamic routing method, device, equipment and storage medium | |
| CN111400246B (en) | Asynchronous file import method, device, computer equipment and storage medium | |
| CN111984465A (en) | Database remote backup method, device, medium and electronic equipment | |
| CN111339551B (en) | Data verification method and related device and equipment | |
| CN112527562A (en) | Data backup method and device of distributed storage system and computer equipment | |
| CN110989934B (en) | Block chain link point data storage method, block chain system and block chain node | |
| CN113468118B (en) | File increment storage method, device and storage medium based on blockchain | |
| CN108133026B (en) | Multi-data processing method, system and storage medium | |
| CN111651417B (en) | Log processing method and device | |
| CN113391972A (en) | Interface testing method and device | |
| CN111813346A (en) | Method, system, device and medium for establishing Ceph distributed storage based on cloud platform | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| CN113824755A (en) | Method, system and related device for processing block chain data | |
| CN113407508B (en) | Method, system, equipment and medium for compressing log file | |
| CN116069728A (en) | Method, system, equipment and storage medium for checking directory consistency | |
| CN115878381A (en) | Data recovery method and device based on SRM disc, storage medium and electronic device | |
| CN114625697A (en) | File system recovery method and device, electronic equipment and storage medium | |
| CN109491699B (en) | Resource checking method, device, equipment and storage medium of application program | |
| CN114722261A (en) | Resource processing method and device, electronic equipment and storage medium | |
| CN113505153A (en) | Memorandum backup method based on iOS system and related equipment | |
| CN111770158A (en) | Cloud platform recovery method and device, electronic equipment and computer readable storage medium | |
| CN110795388A (en) | Binary file processing method, system, equipment and readable storage medium | |
| CN118092982B (en) | Multi-cluster operation and maintenance method, equipment and medium for cloud native application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |