CN111984618A - Log compression method and device - Google Patents
Log compression method and device Download PDFInfo
- Publication number
- CN111984618A CN111984618A CN202010842915.2A CN202010842915A CN111984618A CN 111984618 A CN111984618 A CN 111984618A CN 202010842915 A CN202010842915 A CN 202010842915A CN 111984618 A CN111984618 A CN 111984618A
- Authority
- CN
- China
- Prior art keywords
- log
- buffer
- compression
- compressed
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a log compression method and a device, wherein the log compression method comprises the following steps: writing the newly generated log into a log buffer area in the memory; under the condition that the log buffer area is filled, compressing the logs temporarily stored in the log buffer area to obtain a compressed log file; and storing the compressed log file and emptying a log buffer. According to the log compression method, the newly generated log is stored in the log buffer area of the memory, the stored log is compressed after the log buffer area is filled up to obtain a compressed log file, and then the compressed log file is stored in the memory. Because the IO process is only invoked when the compressed log is stored, there is no need to use the IO process as many times as in the prior art. Because the compressed log file is a file which is already compressed, the time for the IO process to execute the storage of the compressed log file is shorter when the IO process is occupied.
Description
Technical Field
The application relates to the technical field of computer data processing, in particular to a log compression method and device.
Background
When the database runs to provide services to the outside, the server records various operation behaviors to form a log. Under the conditions of long-time running and frequent trigger operation, the server can form a large amount of log files; the log file is stored in the hard disk of the server, so that a large amount of hard disk resources are consumed, and in order to write the log into the hard disk, the IO service of the hard disk needs to be frequently occupied, and the use of the normal business on the IO service of the hard disk is crowded.
In order to solve the problems that the log storage occupies a large amount of resources and affects the operation of the server, the existing method is to regularly compress the accumulated log files stored in the hard disk, store and compress the files. Compressing a large number of log files stored in the hard disk, and occupying the hard disk for reading data and occupying the hard disk for writing data (i.e. IO operations); compressing a large number of log files after storage occupies a large number of CPU resources in a short time, and reduces the timeliness of the server in processing normal services.
Disclosure of Invention
To solve the above technical problem or at least partially solve the above technical problem, the present application provides a log compression method and apparatus.
In one aspect, the present application provides a log compression method, including:
writing the newly generated log into a log buffer area in the memory;
under the condition that the log buffer area is filled, compressing the logs temporarily stored in the log buffer area to obtain a compressed log file;
and storing the compressed log file and emptying the log buffer area.
The log buffer comprises a plurality of buffer sub-regions;
writing the newly generated log to a log buffer in the memory includes: writing newly generated logs of different types into different buffer subareas, or writing newly generated logs corresponding to different applications into different buffer subareas;
under the condition that the log buffer is full, compressing the logs temporarily stored in the log buffer comprises the following steps: under the condition that any buffer subarea is full, compressing the log temporarily stored in any buffer subarea;
the emptying the log buffer comprises: emptying the buffer subarea.
Simultaneously starting a plurality of compression processing processes or compression processing threads;
under the condition that any buffer subarea is full, compressing the log temporarily stored in any buffer subarea, wherein the compressing comprises the following steps: and compressing the log temporarily stored in any buffer subarea by using the compression processing process or the compression processing thread in an idle state.
Further comprising:
and determining the number of the started compression processing processes or the number of the started compression processing threads according to the number of the buffer subareas and the log generation rate.
Storing the compressed log file, including: and storing the compressed log file by adopting an IO process.
A log compression apparatus comprising:
the log cache unit is used for writing the newly generated log into a log buffer area in the memory;
the log compression unit is used for compressing the logs temporarily stored in the log buffer area under the condition that the log buffer area is filled up to obtain a compressed log file;
a storage unit for storing the compressed log file;
and the log clearing unit is used for clearing the log buffer area after the storage unit stores the compressed log file.
The log buffer comprises a plurality of buffer sub-regions;
the log cache unit writes newly generated logs of different types into different buffer subareas, or writes newly generated logs corresponding to different applications into different buffer subareas;
the log compression unit compresses the logs temporarily stored in any buffer subarea under the condition that any buffer subarea is full;
and the log emptying unit empties the log buffer subarea after the storage unit stores the compressed log file.
The log compression unit simultaneously starts a plurality of compression processing processes or compression processing processes, and,
and compressing the log temporarily stored in any buffer subarea by using the compression processing process or the compression processing process in an idle state.
According to the log compression method, the newly generated log is stored in the memory log buffer area, the stored log is compressed after the log buffer area is filled up to obtain a compressed log file, and then the compressed log file is stored in the memory. Because the IO process is only invoked when the compressed log is stored, there is no need to use the IO process as many times as in the prior art. Because the compressed log file is a compressed file, the file size of the compressed log file is relatively small, the time for the IO process to execute the compressed log file storage is also short when the IO process is occupied, the single log compression can occupy shorter CPU service time, the influence on normal service processing is reduced, and the timeliness requirement of the server on the normal service processing is met.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a flowchart of a log compression method provided in an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a log compression apparatus according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;
wherein: 11-a log cache unit, 12-a log compression unit, 13-a storage unit and 14-a log clearing unit; 21-processor, 22-memory, 23-communication interface, 24-bus system.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
The embodiment of the application provides a method for compressing a CPU log, which can reduce the occupation of IO to a memory and reduce the occupation of the time for compressing the log; the log compression method provided by the embodiment of the application is mainly applied to the server which can generate a large number of operation logs in a short time so as to improve the performance of the server.
Of course, the log compression method in the embodiment of the present application may also be applied to client devices, such as electronic devices such as personal computers; however, the log compression method provided by the embodiment of the present application is applied to such devices, and cannot effectively improve the device performance, considering that the single-use startup time of such devices is short (such devices are not generally in a long-term live operation state), and the operation log amount generated in each startup use is also small.
Fig. 1 is a flowchart of a log compression method provided in an embodiment of the present application. As shown in fig. 1, the log compression method provided by the embodiment of the present application includes steps S101 to S104.
S101: and writing the newly generated log into a log buffer in the memory.
In the embodiment of the application, in order to reduce occupation of memory IO, a special log buffer area is arranged in the memory; the log buffer is a buffer which can only be read by a log writing process and a log compression process, the log writing process only has writing authority to the log buffer, and the log compression process only has reading authority and deleting authority to the log buffer, but has no rewriting authority.
After the CPU generates a new log according to the operation, the CPU determines the unused memory address in the log buffer area through addressing, and writes the newly generated log into the unused memory address to form a log linked list.
S102: judging whether the log buffer area is filled; if yes, go to step S103; if not, the process continues to S101.
In this embodiment, after executing S102 each time, the server may determine the filling state of the log buffer.
If the journal buffer is already filled, the journal buffer will not be available for storing newly generated journals if it is not emptied at this time. According to the production job request, the log stored in the log buffer cannot be emptied directly without being stored, so step S103 is triggered to be executed.
If the log buffer is not filled, then step S103 is not executed, and the log generated subsequently may be directly written into the unwritten memory address, so that S101 is executed continuously.
S103: and compressing the logs temporarily stored in the log buffer area to obtain a compressed log file.
In step S103, the log temporarily stored in the log buffer is compressed, and the buffered log is read, and the data is operated by using a compression process according to a specific compression algorithm, so as to obtain a compressed log file.
In the embodiment of the present application, the compression algorithm may adopt a compression algorithm provided by a zlib compression function library. The user can set the compression level (e.g. level 1 to 10 in zlib) according to the importance of the log, the storage capacity of the memory and the CPU usage of the server, to be able to seek the optimum between the server performance loss and the storage space occupation.
S104: and storing the compressed log file and emptying a log buffer.
After the compressed log file is generated in step S103, the server calls the IO process, and stores the generated compressed log file in the memory.
After the compressed log file is correctly stored, the temporarily stored logs in the log buffer are backed up in the memory, and the temporarily stored logs can be emptied, so that the log buffer can cache new logs.
According to the log compression method provided by the embodiment of the application, the newly generated log is stored in the log buffer area of the memory, the stored log is compressed after the log buffer area is filled, a compressed log file is obtained, and then the compressed log file is stored in the memory.
Because the IO process is only invoked when the compressed log is stored, there is no need to use the IO process as many times as in the prior art. Because the compressed log file is a file which is already compressed, the file size of the compressed log file is relatively small, and the time for the IO process to execute the storage of the compressed log file is also short when the IO process is occupied.
In addition, under the condition that the log buffer is full, namely the stored logs are compressed, and the capacity of the log buffer is small (in the conventional application, the capacity of the log buffer is about 10M), so that the single-time log compression can occupy small CPU service time, the influence on normal business processing is reduced, and the requirement of the server on timeliness of normal business processing is met.
In a specific application of the embodiment of the present application, in order to achieve management of the log in a refined manner, the log buffer may be divided into a plurality of mutually independent buffer sub-areas.
Correspondingly, the writing of the newly generated log into the log buffer in step S101 may be: and storing the newly generated log into the corresponding buffer subarea according to the log type of the newly generated log. For example, in one application, the log is divided into a read log, a write log, a rewrite log, and the buffer sub-area may be divided into a write log sub-area, a rewrite log sub-area, or a buffer sub-area. The corresponding newly generated log is written into the corresponding log buffer.
In other embodiments of the present application, the log buffer may also be divided into buffer sub-areas according to different applications, that is, different applications correspond to different buffer sub-areas, and a log corresponding to an application is stored in the corresponding buffer sub-area.
Based on the foregoing case that the log buffer is divided into a plurality of mutually independent buffer sub-areas, in this embodiment of the application, step S103 may be subdivided as: and when any buffer subarea is full, compressing the log temporarily stored in the buffer subarea. That is, as long as one buffer sub-area is filled with the corresponding type of log, the compression operation is performed on the buffer sub-area, regardless of whether other buffer sub-areas in the log buffer are filled.
Accordingly, the emptying of the log buffer in step S104 is then refined to empty the buffer sub-area in which the log has been compressed and to store the corresponding compressed log file.
In the case that the log buffer area may be divided into a plurality of mutually independent buffer sub-areas, and each buffer sub-area is compressed independently, the electronic device in the embodiment of the present application may start a plurality of compression processing processes, and perform a compression operation on the log by using the plurality of compression processing processes.
Specifically, in the embodiment of the present application, when any buffer sub-area is full, a compression processing process in an idle state is used to perform a compression processing operation on a log that is full in a certain sub-buffer area. In practical applications, the time from emptying to refilling of each buffer subarea is much longer than the time of the compression process progress compression process, so that the number of the compression process progress started can be less than the number of the buffer subareas.
In a specific application of the embodiment of the present application, the number of compression processing processes to be started may be determined according to the number of buffer sub-areas and a log generation rate. For example, in the case of a small amount of server data processing, a part of the processes that have been created may be terminated, and only one compression processing process may be reserved; and under the condition that the server is in large data throughput, a compression processing process which meets the number of application requirements (for example, each buffer subarea is ensured to be emptied quickly, but the logs are not in a secondary cache of the CPU) can be started according to the log generation rate.
In actual operation, an administrator may set the number of compression processing processes according to the application of the server, so as to optimize the performance of the server according to the running hardware devices and environment.
In the embodiment of the application, when the plurality of buffer subareas are compressed through the log compression process, each log compression process can judge whether the buffer subareas are filled up one by one in a polling mode. If the log compression process polls and determines that a buffer subarea is full, a mutual exclusion lock is added for marking the buffer subarea again, and the log compression is started; other log compression processes in the idle state check that a certain buffer subarea is in the locked state, and the buffer subarea cannot be processed.
In the foregoing embodiment, the server starts a plurality of compression processing processes for performing compression processing on the logs in the buffer subarea where the logs are filled up; in other embodiments, the server may also start one or more compression processing processes, and start a plurality of compression processing threads in each compression processing process to compress the log temporarily stored in the cache sub-area.
In the specific application of the application, the log is compressed to generate the compressed log file, the speed of storing one compressed log file is high, and the storage time is short compared with the compression time, so that in the embodiment of the application, only one IO process can be started to store the compressed log file.
Besides providing the log compression method, the embodiment of the application also discloses a log compression device. The log compression device and the log compression method adopt the same inventive concept. The following describes a structure of a log compression apparatus provided in an embodiment of the present application.
Fig. 2 is a schematic structural diagram of a log compression apparatus according to an embodiment of the present application. As shown in fig. 2, the log compression apparatus provided in the embodiment of the present application includes a log caching unit 11, a log compression unit 12, a storage unit 13, and a log clearing unit 14.
The log cache unit 11 is configured to write the newly generated log into a log buffer in the memory. The log buffer is a buffer that can only be read by the log buffer unit 11 and the log compression unit 12, and the log buffer unit 11 has only write authority to the log buffer and the log compression unit 12 has read authority and delete authority to the log buffer, but has no rewrite authority.
After the CPU generates a new log according to the operation, the log cache unit 11 determines an unused memory address in the log buffer area by addressing, and writes the newly generated log into the unused memory address. If the journal buffer is in a locked state at this time, the CPU may store the newly generated journal in another buffer in the memory, or directly in the secondary cache.
The log compression unit 12 is configured to compress the log temporarily stored in the log buffer area to obtain a compressed log file when the log buffer area is full. In this embodiment of the application, the log compression unit 12 may adopt a compression algorithm provided by a zlib compression function library. The user can set the compression level (e.g., levels 1-10 in the zlib compression function library) based on the importance of the log, the storage capacity of the memory, and the CPU usage of the server to be able to seek the optimum between server performance loss and storage footprint.
The storage unit 13 is used to store the compressed log file. In the embodiment of the present application, the storage unit 13 is disposed on a nonvolatile memory, so as to be able to store the compressed log file even when the device is powered off, and ensure that the compressed log file is not lost.
The log emptying unit 14 is configured to empty the log buffer after the storage unit 13 stores the compressed log file. After the compressed log file is correctly stored, the temporarily stored logs in the log buffer area are backed up in the memory, and the temporarily stored logs can be emptied, so that the log buffer area can cache new logs.
The log compression device provided by the embodiment of the application stores the newly generated log in the log buffer area of the memory, compresses the stored log after the log buffer area is filled to obtain a compressed log file, and stores the compressed log file in the memory.
Because the IO process is only invoked when the compressed log is stored, there is no need to use the IO process as many times as in the prior art. Because the compressed log file is a file which is already compressed, the file size of the compressed log file is relatively small, and the time for the IO process to execute the storage of the compressed log file is also short when the IO process is occupied.
In addition, under the condition that the log buffer is full, namely the stored logs are compressed, and the capacity of the log buffer is small (in the conventional application, the capacity of the log buffer is about 10M), so that the single-time log compression can occupy small CPU service time, the influence on normal business processing is reduced, and the requirement of the server on timeliness of normal business processing is met.
In a specific application of the embodiment of the present application, the log buffer includes a plurality of buffer sub-areas.
Correspondingly, the log cache unit 11 may write the newly generated logs of different types to different buffer sub-areas, or write the logs corresponding to the newly generated logs of different applications to different buffer sub-areas.
In the case that the log buffer includes a plurality of buffer subareas and the buffer subareas are independent of each other, the log compression unit 12 compresses the log temporarily stored in any one of the buffer subareas when any one of the buffer subareas is full.
In addition, in the case where the log buffer includes a plurality of buffer subareas, and the respective buffer subareas are independent of each other, the log clearing unit 14 clears the log buffer subarea after the storage unit 13 stores the compressed log file.
In this embodiment of the present application, the log compressing unit 12 may start a plurality of compression processing processes or compression processing threads simultaneously, and use one compression processing process or compression processing thread in an idle state to compress the log temporarily stored in any buffer sub-area.
Specifically, when any buffer is full, the log compression unit 12 searches for a compression process in an idle state, so as to perform a compression process on a log full in a sub-buffer by using the process.
Under certain application conditions, the number of compression process processes that are open may be less than the number of buffer sub-sections, because the time from emptying to refilling of each buffer sub-section is much longer than the time of the compression process.
In practical applications, the log compression unit 12 may determine the number of compression processing processes to be started according to the number of buffer sub-areas and the log generation rate. For example, in the case where the data processing amount is small at the server, the log compression unit 12 may terminate a part of the already created process; in the case of a server with a large data throughput, the log compression unit 12 may start a compression process according to the log generation rate, where the compression process satisfies the number of application requirements (for example, it is ensured that each buffer sub-area can be emptied quickly, but the log is not in the second-level cache of the CPU).
In a specific application of the present application, when the log compression unit 12 compresses a plurality of buffer sub-regions through a log compression process, each log compression process can determine whether the buffer sub-regions are filled up one by one in a polling manner. If the log compression process polls to determine that a buffer subarea is full, marking the buffer subarea to be in a locking state, and starting to compress the log in the buffer subarea; other log compression processes in the idle state check that a certain buffer subarea is in the locking state, and the buffer subarea cannot be processed.
In this application, it takes a long time to compress the log to generate a compressed log file, and the rate of storing one compressed log file is fast, and the storage time is short compared to the compression time, so in this embodiment of the application, the number of the storage units 13 is only one (in essence, the storage units are instantiated as a storage process).
Besides providing the log compression method and the log compression device, the embodiment of the application also provides the electronic equipment capable of realizing the method. As mentioned above, electronic devices are mostly servers.
Fig. 3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As shown in fig. 3, the first server comprises at least one processor 21, at least one memory 22 and at least one communication interface 23. And a communication interface 23 for information transmission with an external device.
The various components in the first server are coupled together by a bus system 24. Understandably, the bus system 24 is used to enable connective communication between these components. The bus system 24 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 24 in fig. 3.
It will be appreciated that the memory 22 in this embodiment may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. In some embodiments, memory 22 stores elements, executable units or data structures, or a subset thereof, or an expanded set thereof, as follows: an operating system and an application program.
The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic tasks and processing hardware-based tasks. The application programs include various application programs such as a media player (MediaPlayer), a Browser (Browser), etc. for implementing various application tasks. The program for implementing the log compression method provided by the embodiment of the present disclosure may be included in an application program.
In the embodiment of the present disclosure, the processor 21 is configured to call a program or an instruction stored in the memory 22, which may be specifically a program or an instruction stored in an application program, and the processor 21 is configured to execute the steps of the log compression method provided by the embodiment of the present disclosure.
The method for solving the skylight problem provided by the embodiment of the present disclosure may be applied to the processor 21, or implemented by the processor 21. The processor 21 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 21. The Processor 21 may be a general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The steps of the log compression method provided by the embodiment of the present disclosure may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software units in the decoding processor. The software elements may be located in ram, flash, rom, prom, or eprom, registers, among other storage media that are well known in the art. The storage medium is located in a memory 22, and the processor 21 reads the information in the memory 22 and performs the steps of the method in combination with its hardware.
The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores a program or an instruction, and the program or the instruction causes a computer to execute the steps of each embodiment of the log compression method, and details are not repeated here in order to avoid repeated description.
It is noted that, herein, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other elements in the process, method, article, or apparatus that comprise the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A log compression method, comprising:
writing the newly generated log into a log buffer area in the memory;
under the condition that the log buffer area is filled, compressing the logs temporarily stored in the log buffer area to obtain a compressed log file;
and storing the compressed log file and emptying the log buffer area.
2. The log compression method of claim 1, wherein the log buffer comprises a plurality of buffer sub-sections;
writing the newly generated log to a log buffer in the memory includes: writing newly generated logs of different types into different buffer subareas, or writing newly generated logs corresponding to different applications into different buffer subareas;
under the condition that the log buffer is full, compressing the logs temporarily stored in the log buffer comprises the following steps: under the condition that any buffer subarea is full, compressing the log temporarily stored in any buffer subarea;
the emptying the log buffer comprises: emptying the buffer subarea.
3. The log compression method according to claim 2, wherein a plurality of compression processing processes or compression processing threads are started simultaneously;
under the condition that any buffer subarea is full, compressing the log temporarily stored in any buffer subarea, wherein the compressing comprises the following steps: and compressing the log temporarily stored in any buffer subarea by using the compression processing process or the compression processing thread in an idle state.
4. The log compression method of claim 3, further comprising:
and determining the number of the started compression processing processes or the number of the compression processing threads according to the number of the buffer subareas and the log generation rate.
5. The log compression method as recited in any one of claims 1 to 4,
storing the compressed log file, including: and storing the compressed log file by adopting an IO process.
6. A log compression apparatus, comprising:
the log cache unit is used for writing the newly generated log into a log buffer area in the memory;
the log compression unit is used for compressing the logs temporarily stored in the log buffer area under the condition that the log buffer area is filled up to obtain a compressed log file;
a storage unit for storing the compressed log file;
and the log clearing unit is used for clearing the log buffer area after the storage unit stores the compressed log file.
7. The log compression apparatus of claim 6, wherein the log buffer comprises a plurality of buffer sub-regions;
the log cache unit writes newly generated logs of different types into different buffer subareas, or writes newly generated logs corresponding to different applications into different buffer subareas;
the log compression unit compresses the logs temporarily stored in any buffer subarea under the condition that any buffer subarea is full;
and the log emptying unit empties the log buffer subarea after the storage unit stores the compressed log file.
8. The log compression apparatus of claim 7,
the log compression unit simultaneously starts a plurality of compression processing processes or compression processing threads, and,
and compressing the log temporarily stored in any buffer subarea by using the compression processing process or the compression processing thread in an idle state.
9. An electronic device comprising a processor and a memory;
the processor is adapted to perform the steps of the method of any one of claims 1 to 5 by calling a program or instructions stored in the memory.
10. A computer-readable storage medium, characterized in that it stores a program or instructions for causing a computer to carry out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010842915.2A CN111984618A (en) | 2020-08-20 | 2020-08-20 | Log compression method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010842915.2A CN111984618A (en) | 2020-08-20 | 2020-08-20 | Log compression method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111984618A true CN111984618A (en) | 2020-11-24 |
Family
ID=73443861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010842915.2A Pending CN111984618A (en) | 2020-08-20 | 2020-08-20 | Log compression method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111984618A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746665A (en) * | 2021-07-29 | 2021-12-03 | 深圳市明源云科技有限公司 | Log data processing method, apparatus, computer program product and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030076535A1 (en) * | 2001-10-23 | 2003-04-24 | Prosi Rainer Friedrich | Banded compositor for variable data |
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
| US7779021B1 (en) * | 2004-03-09 | 2010-08-17 | Versata Development Group, Inc. | Session-based processing method and system |
| CN102053923A (en) * | 2009-11-05 | 2011-05-11 | 北京金山软件有限公司 | Storage method and storage device for logbook data |
| US20130117235A1 (en) * | 2011-11-07 | 2013-05-09 | Sap Ag | Implicit Group Commit When Writing Database Log Entries |
| US20140095553A1 (en) * | 2012-09-28 | 2014-04-03 | Oracle International Corporation | Techniques for moving data files without interrupting access |
| US20140279917A1 (en) * | 2013-03-14 | 2014-09-18 | Oracle International Corporation | Techniques To Parallelize CPU and IO Work of Log Writes |
-
2020
- 2020-08-20 CN CN202010842915.2A patent/CN111984618A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030076535A1 (en) * | 2001-10-23 | 2003-04-24 | Prosi Rainer Friedrich | Banded compositor for variable data |
| US7779021B1 (en) * | 2004-03-09 | 2010-08-17 | Versata Development Group, Inc. | Session-based processing method and system |
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
| CN102053923A (en) * | 2009-11-05 | 2011-05-11 | 北京金山软件有限公司 | Storage method and storage device for logbook data |
| US20130117235A1 (en) * | 2011-11-07 | 2013-05-09 | Sap Ag | Implicit Group Commit When Writing Database Log Entries |
| US20140095553A1 (en) * | 2012-09-28 | 2014-04-03 | Oracle International Corporation | Techniques for moving data files without interrupting access |
| US20140279917A1 (en) * | 2013-03-14 | 2014-09-18 | Oracle International Corporation | Techniques To Parallelize CPU and IO Work of Log Writes |
Non-Patent Citations (1)
| Title |
|---|
| 赵伟华: "《JAVA网络编程》", 西安电子科技大学出版社, pages: 137 - 138 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113746665A (en) * | 2021-07-29 | 2021-12-03 | 深圳市明源云科技有限公司 | Log data processing method, apparatus, computer program product and storage medium |
| CN113746665B (en) * | 2021-07-29 | 2022-04-15 | 深圳市明源云科技有限公司 | Log data processing method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7707232B2 (en) | Implementation for collecting unmanaged memory | |
| KR101357397B1 (en) | Method for tracking memory usages of a data processing system | |
| US8307148B2 (en) | Flash management techniques | |
| EP3108371B1 (en) | Modified memory compression | |
| US9323460B2 (en) | Assigning priorities to data for hybrid drives | |
| CN114096953A (en) | Memory management method and device, electronic equipment and computer readable medium | |
| KR20140035416A (en) | Memory manager with enhanced application metadata | |
| US8775749B2 (en) | Demand based memory management of non-pagable data storage | |
| US20080209264A1 (en) | Method for Automatic Dump Assurance | |
| KR20140034246A (en) | Memory management model and interface for new applications | |
| KR20060129873A (en) | Method for executing garbage collection of mobile terminal | |
| EP3283965A1 (en) | Reducing memory commit charge when compressing memory | |
| CN115407943A (en) | Memory dump file generation method, device and equipment and readable storage medium | |
| CN112214388A (en) | Memory monitoring method, device, equipment and computer readable storage medium | |
| EP3265918B1 (en) | Using memory compression to reduce memory commit charge | |
| CN111984618A (en) | Log compression method and device | |
| Han et al. | A hybrid swapping scheme based on per-process reclaim for performance improvement of android smartphones (August 2018) | |
| US20050144389A1 (en) | Method, system, and apparatus for explicit control over a disk cache memory | |
| US10417121B1 (en) | Monitoring memory usage in computing devices | |
| CN116719753A (en) | Data processing apparatus, data processing method, and computer-readable storage medium | |
| US20100299672A1 (en) | Memory management device, computer system, and memory management method | |
| US7681009B2 (en) | Dynamically updateable and moveable memory zones | |
| CN112162780B (en) | Application operation control method and device and electronic equipment | |
| CN112948336B (en) | Data acceleration method, cache unit, electronic device and storage medium | |
| CN115543859A (en) | Wear leveling optimization method, device, equipment and medium for multi-partition SSD |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |