CN108874639A - Log archiving method, electronic device and readable storage medium storing program for executing - Google Patents
Log archiving method, electronic device and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN108874639A CN108874639A CN201810259905.9A CN201810259905A CN108874639A CN 108874639 A CN108874639 A CN 108874639A CN 201810259905 A CN201810259905 A CN 201810259905A CN 108874639 A CN108874639 A CN 108874639A
- Authority
- CN
- China
- Prior art keywords
- preset
- compression algorithm
- store path
- daily record
- target store
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of log archiving method, electronic device and readable storage medium storing program for executing, this method to include：Daily record data to be archived in default application program is read, and daily record data is stored into corresponding preset cache；When meeting preset streaming contractive condition, target store path is set by preset rules, and the daily record data in preset cache is written by corresponding target store path based on the preset compression algorithm for supporting data supplementing；The preset compression algorithm for supporting data supplementing includes at least one of SNAPPY compression algorithm, LZ4 compression algorithm, ZIP compression algorithm.The present invention is not necessarily to do customized development for different business systems, realize that the versatility of log filing is stronger, and, when reaching preset streaming contractive condition, daily record data to be archived can be write direct into corresponding target store path based on the preset compression algorithm for supporting data supplementing, without the backup of archive log data, the problem of avoiding multiple duplication, memory space is saved.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of log archiving methods, electronic device and readable storage
Medium.
Background technique
Existing log processing software is broadly divided into two classes, and one kind is using Storm and Spark Streaming as representative
Processing frame in real time, needs to do operation system customized development, and often the development cycle is longer, and institute is functional to need to develop again,
Versatility is not strong.Another kind of is that commercialization profession log processing product such as Splunk carries backup functionality when handling journal file,
A data can not only be filed when filing, but multiple backups are all filed, will cause the waste of memory space.
Summary of the invention
The purpose of the present invention is to provide a kind of log archiving method, electronic device and readable storage medium storing program for executing, it is intended to realize
The versatility of log is filed, and avoids the problem that multiple duplication.
To achieve the above object, the present invention provides a kind of electronic device, and the electronic device includes memory, processor,
The log filing system that can be run on the processor is stored on the memory, the log filing system is by the place
Reason device realizes following steps when executing：
Daily record data to be archived in default application program is read, and the daily record data is stored to corresponding default slow
In depositing；
When meeting preset streaming contractive condition, target store path is set by preset rules, and be based on preset branch
Corresponding target store path is written in daily record data in the preset cache by the compression algorithm for holding data supplementing；Wherein, institute
Stating the preset compression algorithm for supporting data supplementing includes SNAPPY compression algorithm, LZ4 compression algorithm, in ZIP compression algorithm
At least one.
Preferably, it is described based on the preset compression algorithm for supporting data supplementing by the daily record data in the preset cache
Corresponding target store path, which is written, includes：
It generates and shows compress mode selection interface, include Fast Compression option, master in the compress mode selection interface
Stream format compression options and universal compressed option；
If user chooses Fast Compression option in the compress mode selection interface, utilize LZ4 compression algorithm by institute
Corresponding target store path is written in the daily record data stated in preset cache；
If user chooses mainstream format compression option in the compress mode selection interface, ZIP compression algorithm is utilized
Corresponding target store path is written into daily record data in the preset cache；
If user chooses universal compressed option in the compress mode selection interface, utilize SNAPPY compression algorithm will
Corresponding target store path is written in daily record data in the preset cache.
Preferably, it is described based on the preset compression algorithm for supporting data supplementing by the daily record data in the preset cache
Corresponding target store path, which is written, includes：
If file is not present in the target store path, based on the preset compression algorithm for supporting data supplementing to institute
The daily record data stated in preset cache is compressed, and the new file in the target store path is formed；
If existing compressed file in the target store path, based on the preset compression algorithm for supporting data supplementing
Daily record data in the preset cache is directly compressed in the write-in target store path in already present compressed file.
Preferably, described to include by preset rules setting target store path：
It is ordered according to the data type of machine name and/or the daily record data where compression time, default application program
Name generates pathname, and using the pathname of generation as target store path.
In addition, to achieve the above object, the present invention also provides a kind of log archiving method, the log archiving method packet
It includes：
Daily record data to be archived in default application program is read, and the daily record data is stored to corresponding default slow
In depositing；
When meeting preset streaming contractive condition, target store path is set by preset rules, and be based on preset branch
Corresponding target store path is written in daily record data in the preset cache by the compression algorithm for holding data supplementing；Wherein, institute
Stating the preset compression algorithm for supporting data supplementing includes SNAPPY compression algorithm, LZ4 compression algorithm, in ZIP compression algorithm
At least one.
Preferably, it is described based on the preset compression algorithm for supporting data supplementing by the daily record data in the preset cache
Corresponding target store path, which is written, includes：
It generates and shows compress mode selection interface, include Fast Compression option, master in the compress mode selection interface
Stream format compression options and universal compressed option；
If user chooses Fast Compression option in the compress mode selection interface, utilize LZ4 compression algorithm by institute
Corresponding target store path is written in the daily record data stated in preset cache；
If user chooses mainstream format compression option in the compress mode selection interface, ZIP compression algorithm is utilized
Corresponding target store path is written into daily record data in the preset cache；
If user chooses universal compressed option in the compress mode selection interface, utilize SNAPPY compression algorithm will
Corresponding target store path is written in daily record data in the preset cache.
Preferably, it is described based on the preset compression algorithm for supporting data supplementing by the daily record data in the preset cache
Corresponding target store path, which is written, includes：
If file is not present in the target store path, based on the preset compression algorithm for supporting data supplementing to institute
The daily record data stated in preset cache is compressed, and the new file in the target store path is formed；
If existing compressed file in the target store path, based on the preset compression algorithm for supporting data supplementing
Daily record data in the preset cache is directly compressed in the write-in target store path in already present compressed file.
Preferably, described to include by preset rules setting target store path：
It is ordered according to the data type of machine name and/or the daily record data where compression time, default application program
Name generates pathname, and using the pathname of generation as target store path.
Preferably, the preset streaming contractive condition is that the daily record data size in the preset cache reaches default appearance
Threshold value is measured, alternatively, cache-time of the daily record data in the preset cache reaches preset time threshold.
Further, to achieve the above object, the present invention also provides a kind of computer readable storage medium, the computers
Readable storage medium storing program for executing is stored with log filing system, and the log filing system can be executed by least one processor, so that institute
At least one processor is stated to execute such as the step of above-mentioned log archiving method.
Log archiving method, electronic device and readable storage medium storing program for executing proposed by the present invention read log number to be archived
According to rear, daily record data to be archived is stored into corresponding preset cache；When meeting preset streaming contractive condition, by pre-
If rule settings target store path, and based on the preset compression algorithm for supporting data supplementing by the day in the preset cache
Corresponding target store path is written in will data.Due to doing customized development without being directed to different business systems, realize log
The versatility of filing is stronger, moreover, when reaching preset streaming contractive condition, it can be based on the preset pressure for supporting data supplementing
Daily record data to be archived is write direct corresponding target store path and kept away without the backup of archive log data by compression algorithm
The problem of having exempted from multiple duplication saves memory space.
Detailed description of the invention
Fig. 1 is the running environment schematic diagram of 10 preferred embodiment of log filing system of the present invention；
Fig. 2 is the flow diagram of one embodiment of log archiving method of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that described herein, specific examples are only used to explain the present invention, not
For limiting the present invention.Based on the embodiments of the present invention, those of ordinary skill in the art are not before making creative work
Every other embodiment obtained is put, shall fall within the protection scope of the present invention.
It should be noted that the description for being related to " first ", " second " etc. in the present invention is used for description purposes only, and cannot
It is interpreted as its relative importance of indication or suggestion or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include at least one of the features.In addition, the skill between each embodiment
Art scheme can be combined with each other, but must be based on can be realized by those of ordinary skill in the art, when technical solution
Will be understood that the combination of this technical solution is not present in conjunction with there is conflicting or cannot achieve when, also not the present invention claims
Protection scope within.
The present invention provides a kind of log filing system.Referring to Fig. 1, being 10 preferred embodiment of log filing system of the present invention
Running environment schematic diagram.
In the present embodiment, the log filing system 10 is installed and is run in electronic device 1.The electronic device 1
It may include, but be not limited only to, memory 11, processor 12 and display 13.Fig. 1 illustrates only the electronics with component 11-13
Device 1, it should be understood that being not required for implementing all components shown, the implementation that can be substituted is more or less
Component.
The memory 11 is the readable computer storage medium of at least one type, and the memory 11 is in some implementations
It can be the internal storage unit of the electronic device 1, such as the hard disk or memory of the electronic device 1 in example.The memory
11 are also possible to the External memory equipment of the electronic device 1 in further embodiments, such as are equipped on the electronic device 1
Plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card,
Flash card (Flash Card) etc..Further, the memory 11 can also both include the storage inside of the electronic device 1
Unit also includes External memory equipment.The memory 11 for store the application software for being installed on the electronic device 1 and respectively
Class data, such as the program code etc. of the log filing system 10.The memory 11 can be also used for temporarily storing
Data through exporting or will export.
The processor 12 can be in some embodiments a central processing unit (Central Processing Unit,
CPU), microprocessor or other data processing chips, for running the program code stored in the memory 11 or processing number
According to, such as execute the log filing system 10 etc..
The display 13 can be light-emitting diode display, liquid crystal display, touch-control liquid crystal display in some embodiments
And OLED (Organic Light-Emitting Diode, Organic Light Emitting Diode) touches device etc..The display 13 is used
In being shown in the information handled in the electronic device 1 and for showing visual user interface, such as log filing
Path, progress of log filing etc..The component 11-13 of the electronic device 1 is in communication with each other by system bus.
Log filing system 10 includes that at least one is stored in the computer-readable instruction in the memory 11, this is at least
One computer-readable instruction can be executed by the processor 12, to realize each embodiment of the application.
Wherein, following steps are realized when above-mentioned log filing system 10 is executed by the processor 12：
Step S1 reads daily record data to be archived in default application program, and the daily record data is stored to correspondence
Preset cache in.
In the present embodiment, log filing system detects the journal file of each default application program, when detecting
When default application program has daily record data to need to file, the journal file number to be archived of each default application program output is read
According to, and as where each default application program machine name and/or data type by log file data write-in to be archived not
Same caching.The machine name is identification number, the admin name etc. where each default application program, and data type can be by
Data type classification itself, such as numeric type, byte type, text-type, data type can also be carried out by the type of service of data
It distinguishes, such as loan transaction data, insurance business data.For example, the day that default application program generates during running on machine A
Will file data can be cached by the type of service of data, the loan industry generated during running on machine A such as application program
Business log file data can be cached into preset first caching, and insurance business log file data can be cached to preset second
In caching.Wherein, first caching, that the second caching is only used for filing capacity is smaller or temporarily file lesser journal file number
According to can be carried out to the log file data filed in the first caching, the second caching after triggering subsequent streaming contractive condition
Further streaming compression processing.Further, since being only used for the lesser log text of filing capacity in first caching, the second caching
Number of packages evidence therefore, both can be to log file data by normal in first caching, the second caching when archive log file data
Compression algorithm such as RLE compression algorithm, Huffman Compress Algorithm, Rice compression algorithm, Lempel-Ziv (LZ77) compression are calculated
Method etc. is filed again after being compressed, and can also directly be filed to log file data, will not be very on memory space influence
Greatly.
Step S2 sets target store path by preset rules, and based on pre- when meeting preset streaming contractive condition
If the compression algorithm of support data supplementing corresponding target store path is written into the daily record data in the preset cache；Its
In, the preset compression algorithm for supporting data supplementing includes SNAPPY compression algorithm, LZ4 compression algorithm, ZIP compression algorithm
At least one of.
In the present embodiment, when the log file data filed in caching (such as first caching, second caching) trigger it is pre-
If streaming contractive condition when, then to the log file data further progress streaming compression processing filed in the caching.This is pre-
If streaming contractive condition can be the log file data size in the caching and reach preset capacity threshold value (such as 100M),
Can be cache-time of the log file data of caching in the caching reach preset time threshold (1 hour) or other
It what is the need for the trigger condition etc. that the log file data of caching is compressed to save memory space.In triggering streaming compression
After condition, the target store path of the log file data is generated by preset rules first, it in one embodiment, can be preparatory
The mapping relations of different machines name and/or data type and target store path are determined, such as the loan industry on machine A
It is a, the then loan transaction that application program generates during running on machine A that business daily record data, which can preset target store path,
After log file data triggers streaming contractive condition in the buffer, it may be determined that the corresponding target of loan transaction log file data
Store path is a；In another embodiment, also next life can be named according to " compression time, machine name, data type "
At target store path, wherein " machine name " is the machine name where application program, and " data type " is the data of daily record data
Type.Such as application program run on machine A during the loan transaction log file data that generates, compression time is 2017
It then produced target store path " 171123/ machine A/ loan transaction journal file " November 23, generates target in name and deposit
When storing up path, the sequence between this three of compression time, machine name, data type can carry out converting different sequences as needed,
Such as " machine name/data type/compression time ", " compression time/data type/machine name ", in this way, can be to journal file
Detailed store path division is carried out by the various relevant informations of journal file when data carry out compression filing, convenient for subsequent to day
The retrieval of will file and transfer operation.
It, can be based on preset compression algorithm to journal file number after the target store path of log file data has been determined
According to being compressed and be written in target store path simultaneously.In the present embodiment, preset compression algorithm is preset support data
Additional compression algorithm, that is, support can supplemental data compression algorithm, support can the compression algorithm of supplemental data include but unlimited
In Snappy compression algorithm, LZ4 compression algorithm, ZIP compression algorithm etc..
Further, based on the preset compression algorithm for supporting data supplementing by the daily record data in the preset cache
When corresponding target store path is written, further include：
It generates and shows compress mode selection interface, include Fast Compression option, master in the compress mode selection interface
Stream format compression options and universal compressed option；Such as Fast Compression option, mainstream lattice in the compress mode selection interface
Tick boxes are provided with before formula compression options and universal compressed option, user can choose according to self-demand to be needed to daily record data
The mode compressed.
If user chooses Fast Compression option in the compress mode selection interface, utilize LZ4 compression algorithm by institute
Corresponding target store path is written in the daily record data stated in preset cache.LZ4 compression algorithm is a quick lossless compression
Algorithm, the compression speed of monokaryon are more than 400MB/s, and the decompression speed of monokaryon is more than 1GB/s.LZ4 compression algorithm is also applied for more
The case where core, compression speed can increase exponentially, and decompression speed nearly reaches RAM rate limitation.Therefore, if in the present embodiment
User needs to carry out Fast Compression, then corresponding mesh is written in the daily record data in the preset cache using LZ4 compression algorithm
Store path is marked, to realize the Fast Compression to daily record data.
If user chooses mainstream format compression option in the compress mode selection interface, ZIP compression algorithm is utilized
Corresponding target store path is written into daily record data in the preset cache.ZIP compression algorithm is a computer documents
Compression algorithm, ZIP format belongs to one of the compressed format of several mainstreams.
If user chooses universal compressed option in the compress mode selection interface, utilize SNAPPY compression algorithm will
Corresponding target store path is written in daily record data in the preset cache.Wherein, Snappy be a C++ be used to compress
With the development kit of decompression, Snappy will not be collapsed when inputting in face of damage or malice, can be provided in various scenes
High speed compression speed and reasonable compression ratio have preferable robustness, therefore, if user is led in the present embodiment
With compression, then corresponding target store path is written into the daily record data in the preset cache using SNAPPY compression algorithm,
To realize the compression for carrying out fast and stable to daily record data.
Above-mentioned compression algorithm such as Snappy compression algorithm, LZ4 compression algorithm, ZIP compression algorithm can support data supplementing,
In the present embodiment, log file data is compressed based on above-mentioned compression algorithm and target store path is written.Specifically, if
It is existing in target store path to have compressed file, i.e., before this compresses log file data, to the day
The relevant history log file data of will file data (such as machine name, data type is identical) progress over-voltage, which is contractd, has been stored in this
In target store path, then using above-mentioned support data supplementing compression algorithm for example Snappy compression algorithm, LZ4 compression algorithm,
ZIP compression algorithm is directly appended to existing compression text in target store path after compressing to current log file data
In part, without being first stored in individual space for compressed log file data as temporary file, then successively will again
Compressed log file data decompresses and decompresses compressed file existing in target store path, by the log after decompression
The file that file data and existing compressed file solution extrude is integrated into a whole file again, then come compress to be formed it is final complete
Whole compressed package.In the present embodiment using support can supplemental data compression algorithm, directly compressed log file data is chased after
It is added in existing compressed file, does not need individual space to save temporary file, not only log file data is compressed
Filing can save a large amount of memory spaces, and can save the space of a large amount of storage temporary files.For example, by taking ZIP compresses as an example, benefit
Log file data is compressed with ZIP compression algorithm (such as using general winzip, winrar tool), if target
It is existing in store path to have file such as ZIP compressed package, then log file data is directly appended to already present ZIP compressed package
In, specifically when realizing data supplementing, packet code is beaten again into existing ZIP compressed package using Java and adds data.
Compression filing can be carried out to log file data in the present embodiment, a large amount of disk spaces have been saved, so that being consumed
1/5th when storage medium capacity is uncompressed, and it is not necessarily to the backup of archive log, the problem of avoiding multiple duplication.Separately
Outside, target store path is generated by rule in the present embodiment, target is written while compressing to log file data and deposits
Store up path in, if avoid using first compress be written again by the way of caused by generation machine power off etc. accidents, it is easy to cause text
Part is lost or the case where damage, and the log filing in the present embodiment based on streaming compression can be influence drop caused by failure
To minimum.
Compared with prior art, after the present embodiment reads daily record data to be archived, daily record data to be archived is deposited
Storage is into corresponding preset cache；When meeting preset streaming contractive condition, target store path is set by preset rules, and
Corresponding target is written in the daily record data in the preset cache based on the preset compression algorithm for supporting data supplementing to store
Path.Due to doing customized development without being directed to different business systems, realize that the versatility of log filing is stronger, moreover, working as
It, can be based on the preset compression algorithm for supporting data supplementing by daily record data to be archived when reaching preset streaming contractive condition
Corresponding target store path is write direct, without the backup of archive log data, the problem of avoiding multiple duplication, saving is deposited
Store up space.
Further, in an optional embodiment, on the basis of the above embodiments, 10 quilt of log filing system
When the processor 12 executes, further include：
If file is not present in the target store path, based on the preset compression algorithm for supporting data supplementing to institute
The daily record data stated in preset cache is compressed, and forms the new file in the target store path, and by the new text of formation
Presetting database is written in the preset attribute information (such as compression time, data type) of part, so as to it is subsequent when user need using
When filed daily record data, log archive file needed for can first passing through the preset data library searching whether there is, then carry out
Decompression processing.
Further, in an optional embodiment, on the basis of the above embodiments, 10 quilt of log filing system
When the processor 12 executes, further include：
A target store path is automatically generated every preset time (as per hour), by the log number in preset time
It, can basis when being decompressed to filed daily record data convenient for subsequent user in the target store path generated according to write-in
Time is screened to decompress the archive log data of different time.
As shown in Fig. 2, Fig. 2 is the flow diagram of one embodiment of log archiving method of the present invention, the log archiving method
Include the following steps：
Step S10 reads daily record data to be archived in default application program, and the daily record data is stored to correspondence
Preset cache in.
In the present embodiment, the journal file of each default application program is detected, when detecting default application program
When thering is daily record data to need to file, the log file data to be archived of each default application program output is read, and by each
Different cachings is written in log file data to be archived by machine name and/or data type where default application program.It should
Machine name is identification number, the admin name etc. where each default application program, and data type can be by data class itself
Type is distinguished, such as numeric type, byte type, text-type, data type can also be distinguished by the type of service of data, is such as provided a loan
Business datum, insurance business data etc..For example, the log file data that default application program generates during running on machine A
It can be cached by the type of service of data, the loan transaction journal file generated during being run on machine A such as application program
Data can be cached into preset first caching, and insurance business log file data can be cached into preset second caching.Its
In, it is smaller or temporarily file lesser log file data that first caching, the second caching are only used for filing capacity, when triggering
After subsequent streaming contractive condition, the log file data filed in the first caching, the second caching can further be flowed
Formula compression processing.Further, since it is only used for the filing lesser log file data of capacity in first caching, the second caching, because
This, in first caching, the second caching when archive log file data, both can be calculated log file data by common compression
Method such as RLE compression algorithm, Huffman Compress Algorithm, Rice compression algorithm, Lempel-Ziv (LZ77) compression algorithm etc. is pressed
File again after contracting, directly log file data can also be filed, influencing on memory space will not be very big.
Step S20 sets target store path by preset rules, and be based on when meeting preset streaming contractive condition
Corresponding target store path is written in daily record data in the preset cache by the preset compression algorithm for supporting data supplementing；
Wherein, the preset compression algorithm for supporting data supplementing includes SNAPPY compression algorithm, LZ4 compression algorithm, ZIP compression calculation
At least one of method.
In the present embodiment, when the log file data filed in caching (such as first caching, second caching) trigger it is pre-
If streaming contractive condition when, then to the log file data further progress streaming compression processing filed in the caching.This is pre-
If streaming contractive condition can be the log file data size in the caching and reach preset capacity threshold value (such as 100M),
Can be cache-time of the log file data of caching in the caching reach preset time threshold (1 hour) or other
It what is the need for the trigger condition etc. that the log file data of caching is compressed to save memory space.In triggering streaming compression
After condition, the target store path of the log file data is generated by preset rules first, it in one embodiment, can be preparatory
The mapping relations of different machines name and/or data type and target store path are determined, such as the loan industry on machine A
It is a, the then loan transaction that application program generates during running on machine A that business daily record data, which can preset target store path,
After log file data triggers streaming contractive condition in the buffer, it may be determined that the corresponding target of loan transaction log file data
Store path is a；In another embodiment, also next life can be named according to " compression time, machine name, data type "
At target store path, wherein " machine name " is the machine name where application program, and " data type " is the data of daily record data
Type.Such as application program run on machine A during the loan transaction log file data that generates, compression time is 2017
It then produced target store path " 171123/ machine A/ loan transaction journal file " November 23, generates target in name and deposit
When storing up path, the sequence between this three of compression time, machine name, data type can carry out converting different sequences as needed,
Such as " machine name/data type/compression time ", " compression time/data type/machine name ", in this way, can be to journal file
Detailed store path division is carried out by the various relevant informations of journal file when data carry out compression filing, convenient for subsequent to day
The retrieval of will file and transfer operation.
It, can be based on preset compression algorithm to journal file number after the target store path of log file data has been determined
According to being compressed and be written in target store path simultaneously.In the present embodiment, preset compression algorithm is preset support data
Additional compression algorithm, that is, support can supplemental data compression algorithm, support can the compression algorithm of supplemental data include but unlimited
In Snappy compression algorithm, LZ4 compression algorithm, ZIP compression algorithm etc..
Further, based on the preset compression algorithm for supporting data supplementing by the daily record data in the preset cache
When corresponding target store path is written, further include：
It generates and shows compress mode selection interface, include Fast Compression option, master in the compress mode selection interface
Stream format compression options and universal compressed option；Such as Fast Compression option, mainstream lattice in the compress mode selection interface
Tick boxes are provided with before formula compression options and universal compressed option, user can choose according to self-demand to be needed to daily record data
The mode compressed.
If user chooses Fast Compression option in the compress mode selection interface, utilize LZ4 compression algorithm by institute
Corresponding target store path is written in the daily record data stated in preset cache.LZ4 compression algorithm is a quick lossless compression
Algorithm, the compression speed of monokaryon are more than 400MB/s, and the decompression speed of monokaryon is more than 1GB/s.LZ4 compression algorithm is also applied for more
The case where core, compression speed can increase exponentially, and decompression speed nearly reaches RAM rate limitation.Therefore, if in the present embodiment
User needs to carry out Fast Compression, then corresponding mesh is written in the daily record data in the preset cache using LZ4 compression algorithm
Store path is marked, to realize the Fast Compression to daily record data.
If user chooses mainstream format compression option in the compress mode selection interface, ZIP compression algorithm is utilized
Corresponding target store path is written into daily record data in the preset cache.ZIP compression algorithm is a computer documents
Compression algorithm, ZIP format belongs to one of the compressed format of several mainstreams.
If user chooses universal compressed option in the compress mode selection interface, utilize SNAPPY compression algorithm will
Corresponding target store path is written in daily record data in the preset cache.Wherein, Snappy be a C++ be used to compress
With the development kit of decompression, Snappy will not be collapsed when inputting in face of damage or malice, can be provided in various scenes
High speed compression speed and reasonable compression ratio have preferable robustness, therefore, if user is led in the present embodiment
With compression, then corresponding target store path is written into the daily record data in the preset cache using SNAPPY compression algorithm,
To realize the compression for carrying out fast and stable to daily record data.
Above-mentioned compression algorithm such as Snappy compression algorithm, LZ4 compression algorithm, ZIP compression algorithm can support data supplementing,
In the present embodiment, log file data is compressed based on above-mentioned compression algorithm and target store path is written.Specifically, if
It is existing in target store path to have compressed file, i.e., before this compresses log file data, to the day
The relevant history log file data of will file data (such as machine name, data type is identical) progress over-voltage, which is contractd, has been stored in this
In target store path, then using above-mentioned support data supplementing compression algorithm for example Snappy compression algorithm, LZ4 compression algorithm,
ZIP compression algorithm is directly appended to existing compression text in target store path after compressing to current log file data
In part, without being first stored in individual space for compressed log file data as temporary file, then successively will again
Compressed log file data decompresses and decompresses compressed file existing in target store path, by the log after decompression
The file that file data and existing compressed file solution extrude is integrated into a whole file again, then come compress to be formed it is final complete
Whole compressed package.In the present embodiment using support can supplemental data compression algorithm, directly compressed log file data is chased after
It is added in existing compressed file, does not need individual space to save temporary file, not only log file data is compressed
Filing can save a large amount of memory spaces, and can save the space of a large amount of storage temporary files.For example, by taking ZIP compresses as an example, benefit
Log file data is compressed with ZIP compression algorithm (such as using general winzip, winrar tool), if target
It is existing in store path to have file such as ZIP compressed package, then log file data is directly appended to already present ZIP compressed package
In, specifically when realizing data supplementing, packet code is beaten again into existing ZIP compressed package using Java and adds data.
Compression filing can be carried out to log file data in the present embodiment, a large amount of disk spaces have been saved, so that being consumed
1/5th when storage medium capacity is uncompressed, and it is not necessarily to the backup of archive log, the problem of avoiding multiple duplication.Separately
Outside, target store path is generated by rule in the present embodiment, target is written while compressing to log file data and deposits
Store up path in, if avoid using first compress be written again by the way of caused by generation machine power off etc. accidents, it is easy to cause text
Part is lost or the case where damage, and the log filing in the present embodiment based on streaming compression can be influence drop caused by failure
To minimum.
Compared with prior art, after the present embodiment reads daily record data to be archived, daily record data to be archived is deposited
Storage is into corresponding preset cache；When meeting preset streaming contractive condition, target store path is set by preset rules, and
Corresponding target is written in the daily record data in the preset cache based on the preset compression algorithm for supporting data supplementing to store
Path.Due to doing customized development without being directed to different business systems, realize that the versatility of log filing is stronger, moreover, working as
It, can be based on the preset compression algorithm for supporting data supplementing by daily record data to be archived when reaching preset streaming contractive condition
Corresponding target store path is write direct, without the backup of archive log data, the problem of avoiding multiple duplication, saving is deposited
Store up space.
Further, in an optional embodiment, on the basis of the above embodiments, this method further includes：
If file is not present in the target store path, based on the preset compression algorithm for supporting data supplementing to institute
The daily record data stated in preset cache is compressed, and forms the new file in the target store path, and by the new text of formation
Presetting database is written in the preset attribute information (such as compression time, data type) of part, so as to it is subsequent when user need using
When filed daily record data, log archive file needed for can first passing through the preset data library searching whether there is, then carry out
Decompression processing.
Further, in an optional embodiment, on the basis of the above embodiments, this method further includes：
A target store path is automatically generated every preset time (as per hour), by the log number in preset time
It, can basis when being decompressed to filed daily record data convenient for subsequent user in the target store path generated according to write-in
Time is screened to decompress the archive log data of different time.
In addition, the computer-readable recording medium storage has the present invention also provides a kind of computer readable storage medium
Log filing system, the log filing system can be executed by least one processor, so that at least one described processor is held
Such as the step of log archiving method in above-described embodiment, step S10, S20 etc. of the log archiving method was embodied row
Journey is as described above, and details are not described herein.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the device that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or device institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or device.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to be realized by hardware, but very much
In the case of the former be more preferably embodiment.Based on this understanding, technical solution of the present invention is substantially in other words to existing
The part that technology contributes can be embodied in the form of software products, which is stored in a storage
In medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, calculate
Machine, server, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
Preferred embodiments of the present invention have been described above with reference to the accompanying drawings, not thereby limiting the scope of the invention.On
It is for illustration only to state serial number of the embodiment of the present invention, does not represent the advantages or disadvantages of the embodiments.It is patrolled in addition, though showing in flow charts
Sequence is collected, but in some cases, it can be with the steps shown or described are performed in an order that is different from the one herein.
Without departing from the scope and spirit of the invention, there are many variations to implement the present invention by those skilled in the art,
It can be used for another embodiment for example as the feature of one embodiment and obtain another embodiment.It is all to use technology of the invention
Made any modifications, equivalent replacements, and improvements within design, should all be within interest field of the invention.
Claims (10)
1. a kind of electronic device, which is characterized in that the electronic device includes memory, processor, is stored on the memory
There is the log filing system that can be run on the processor, is realized such as when the log filing system is executed by the processor
Lower step：
Daily record data to be archived in default application program is read, and the daily record data is stored to corresponding preset cache
In；
When meeting preset streaming contractive condition, target store path is set by preset rules, and be based on preset support number
Corresponding target store path is written into the daily record data in the preset cache according to additional compression algorithm；Wherein, described pre-
If support data supplementing compression algorithm include SNAPPY compression algorithm, LZ4 compression algorithm, in ZIP compression algorithm at least
One.
2. electronic device as described in claim 1, which is characterized in that described to be calculated based on the preset compression for supporting data supplementing
Corresponding target store path is written in daily record data in the preset cache by method：
It generates and shows compress mode selection interface, include Fast Compression option, mainstream lattice in the compress mode selection interface
Formula compression options and universal compressed option；
It, will be described pre- using LZ4 compression algorithm if user chooses Fast Compression option in the compress mode selection interface
If corresponding target store path is written in the daily record data in caching；
If user chooses mainstream format compression option in the compress mode selection interface, utilize ZIP compression algorithm by institute
Corresponding target store path is written in the daily record data stated in preset cache；
It, will be described using SNAPPY compression algorithm if user chooses universal compressed option in the compress mode selection interface
Corresponding target store path is written in daily record data in preset cache.
3. electronic device as claimed in claim 1 or 2, which is characterized in that described based on the preset pressure for supporting data supplementing
Corresponding target store path is written in daily record data in the preset cache by compression algorithm：
If file is not present in the target store path, based on the preset compression algorithm for supporting data supplementing to described pre-
If the daily record data in caching is compressed, the new file in the target store path is formed；
If existing compressed file in the target store path, based on the preset compression algorithm for supporting data supplementing by institute
The daily record data in preset cache is stated directly to compress in the write-in target store path in already present compressed file.
4. electronic device as claimed in claim 3, which is characterized in that described to set target store path packet by preset rules
It includes：
Life is named according to the data type of machine name and/or the daily record data where compression time, default application program
At pathname, and using the pathname of generation as target store path.
5. a kind of log archiving method, which is characterized in that the log archiving method includes：
Daily record data to be archived in default application program is read, and the daily record data is stored to corresponding preset cache
In；
When meeting preset streaming contractive condition, target store path is set by preset rules, and be based on preset support number
Corresponding target store path is written into the daily record data in the preset cache according to additional compression algorithm；Wherein, described pre-
If support data supplementing compression algorithm include SNAPPY compression algorithm, LZ4 compression algorithm, in ZIP compression algorithm at least
One.
6. log archiving method as claimed in claim 5, which is characterized in that described based on the preset pressure for supporting data supplementing
Corresponding target store path is written in daily record data in the preset cache by compression algorithm：
It generates and shows compress mode selection interface, include Fast Compression option, mainstream lattice in the compress mode selection interface
Formula compression options and universal compressed option；
It, will be described pre- using LZ4 compression algorithm if user chooses Fast Compression option in the compress mode selection interface
If corresponding target store path is written in the daily record data in caching；
If user chooses mainstream format compression option in the compress mode selection interface, utilize ZIP compression algorithm by institute
Corresponding target store path is written in the daily record data stated in preset cache；
It, will be described using SNAPPY compression algorithm if user chooses universal compressed option in the compress mode selection interface
Corresponding target store path is written in daily record data in preset cache.
7. such as log archiving method described in claim 5 or 6, which is characterized in that described to be based on preset support data supplementing
Compression algorithm corresponding target store path be written into the daily record data in the preset cache include：
If file is not present in the target store path, based on the preset compression algorithm for supporting data supplementing to described pre-
If the daily record data in caching is compressed, the new file in the target store path is formed；
If existing compressed file in the target store path, based on the preset compression algorithm for supporting data supplementing by institute
The daily record data in preset cache is stated directly to compress in the write-in target store path in already present compressed file.
8. log archiving method as claimed in claim 7, which is characterized in that described to set target store path by preset rules
Including：
Life is named according to the data type of machine name and/or the daily record data where compression time, default application program
At pathname, and using the pathname of generation as target store path.
9. such as log archiving method described in claim 5 or 6, which is characterized in that the preset streaming contractive condition is institute
The daily record data size stated in preset cache reaches preset capacity threshold value, alternatively, the daily record data is in the preset cache
Cache-time reach preset time threshold.
10. a kind of computer readable storage medium, which is characterized in that be stored with log on the computer readable storage medium and return
Shelves system realizes the log filing as described in any one of claim 5 to 9 when the log filing system is executed by processor
The step of method.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810259905.9A CN108874639A (en) | 2018-03-27 | 2018-03-27 | Log archiving method, electronic device and readable storage medium storing program for executing |
| PCT/CN2018/102084 WO2019184218A1 (en) | 2018-03-27 | 2018-08-24 | Log archiving method, electronic device, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810259905.9A CN108874639A (en) | 2018-03-27 | 2018-03-27 | Log archiving method, electronic device and readable storage medium storing program for executing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108874639A true CN108874639A (en) | 2018-11-23 |
Family
ID=64326523
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810259905.9A Pending CN108874639A (en) | 2018-03-27 | 2018-03-27 | Log archiving method, electronic device and readable storage medium storing program for executing |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108874639A (en) |
| WO (1) | WO2019184218A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111354050A (en) * | 2020-02-25 | 2020-06-30 | 安徽以萨数据技术有限公司 | Android-based picture compression method and device and storage medium |
| CN113220635A (en) * | 2021-05-11 | 2021-08-06 | 深圳市星火数控技术有限公司 | File archiving method, device, equipment and computer readable storage medium |
| CN113326237A (en) * | 2021-06-16 | 2021-08-31 | 深圳市明源云科技有限公司 | Log data processing method and device, terminal device and storage medium |
| CN113656358A (en) * | 2020-05-12 | 2021-11-16 | 网联清算有限公司 | Database log file processing method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1535011A (en) * | 2003-03-27 | 2004-10-06 | 乐金电子（沈阳）有限公司 | Image signal treatment device of digital television |
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
| KR20130136730A (en) * | 2012-06-05 | 2013-12-13 | 주식회사 퓨쳐시스템 | Method and system for archiving and querying semi-structured log |
| CN104424314A (en) * | 2013-09-06 | 2015-03-18 | Sap欧洲公司 | Database operation for columnar table database |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| US9665625B2 (en) * | 2014-06-25 | 2017-05-30 | International Business Machines Corporation | Maximizing the information content of system logs |
| CN106850761A (en) * | 2016-12-30 | 2017-06-13 | 江苏天联信息科技发展有限公司 | Journal file storage method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331487B (en) * | 2014-11-13 | 2019-03-12 | 上海携程商务有限公司 | The processing method and processing device of log |
| CN106815227A (en) * | 2015-11-27 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of storage information |
-
2018
- 2018-03-27 CN CN201810259905.9A patent/CN108874639A/en active Pending
- 2018-08-24 WO PCT/CN2018/102084 patent/WO2019184218A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1535011A (en) * | 2003-03-27 | 2004-10-06 | 乐金电子（沈阳）有限公司 | Image signal treatment device of digital television |
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
| KR20130136730A (en) * | 2012-06-05 | 2013-12-13 | 주식회사 퓨쳐시스템 | Method and system for archiving and querying semi-structured log |
| CN104424314A (en) * | 2013-09-06 | 2015-03-18 | Sap欧洲公司 | Database operation for columnar table database |
| US9665625B2 (en) * | 2014-06-25 | 2017-05-30 | International Business Machines Corporation | Maximizing the information content of system logs |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN106850761A (en) * | 2016-12-30 | 2017-06-13 | 江苏天联信息科技发展有限公司 | Journal file storage method and device |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111354050A (en) * | 2020-02-25 | 2020-06-30 | 安徽以萨数据技术有限公司 | Android-based picture compression method and device and storage medium |
| CN113656358A (en) * | 2020-05-12 | 2021-11-16 | 网联清算有限公司 | Database log file processing method and system |
| CN113220635A (en) * | 2021-05-11 | 2021-08-06 | 深圳市星火数控技术有限公司 | File archiving method, device, equipment and computer readable storage medium |
| CN113220635B (en) * | 2021-05-11 | 2022-02-15 | 深圳市星火数控技术有限公司 | File archiving method, device, equipment and computer readable storage medium |
| CN113326237A (en) * | 2021-06-16 | 2021-08-31 | 深圳市明源云科技有限公司 | Log data processing method and device, terminal device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019184218A1 (en) | 2019-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108874639A (en) | Log archiving method, electronic device and readable storage medium storing program for executing | |
| CN103733195B (en) | Manage the storage of the data for the search based on range | |
| US9823981B2 (en) | Backup and restore strategies for data deduplication | |
| JP5732536B2 (en) | System, method and non-transitory computer-readable storage medium for scalable reference management in a deduplication-based storage system | |
| US20110087640A1 (en) | Data Compression and Storage Techniques | |
| CN104199816A (en) | Managing storage of individually accessible data units | |
| CN109522020A (en) | Android system installation kit method for reducing body weight, device, terminal and storage medium | |
| CN111125033B (en) | Space recycling method and system based on full flash memory array | |
| US10248656B2 (en) | Removal of reference information for storage blocks in a deduplication system | |
| CN1858746A (en) | Apparatus and method for storing and managing additional data in file system | |
| US9965487B2 (en) | Conversion of forms of user data segment IDs in a deduplication system | |
| US10656860B2 (en) | Tape drive library integrated memory deduplication | |
| EP2946295A1 (en) | Grouping chunks of data into a compression region | |
| CN101178726A (en) | Method to efficiently use the disk space while unarchiving | |
| CN111104377A (en) | File management method, electronic device and computer-readable storage medium | |
| CN105493080A (en) | Method and apparatus for context aware based data de-duplication | |
| CN111625531A (en) | Merging device based on programmable device, data merging method and database system | |
| WO2024078122A1 (en) | Database table scanning method and apparatus, and device | |
| CN100377158C (en) | Method for creating library file | |
| WO2011137684A1 (en) | Search method and device based on information records of embedded system | |
| CN112765112A (en) | Installation package packing and unpacking method | |
| CN115543918A (en) | File snapshot method, system, electronic equipment and storage medium | |
| CN115269511A (en) | Generic electronic file packaging method and device | |
| CN107846327A (en) | A kind of processing method and processing device of network management performance data | |
| US6510499B1 (en) | Method, apparatus, and article of manufacture for providing access to data stored in compressed files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181123 |
|
| RJ01 | Rejection of invention patent application after publication |