JP2007257209A - Log data compression program and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that it takes a long time to decompress all log records in compressing and restoring log records, and that a data area is wasted not in compressing any log record at all. <P>SOLUTION: This log data compression program is provided to make a computer function as follows: an update log file as the object of storage; a reflection time DB in which a data reflection time per unit quantity is stored; a decompression time DB in which a data decompression time per unit quantity is stored; a reflection time calculation means for calculating the total sum of the data reflection time from the leading log of log data; a decompression time calculation means for calculating the total sum of the data decompression time from the tailing log of the storage object log data; and a compression processing means for comparing the total sum of the reflection time with the total sum of the decompression time, and for acquiring the data of the next log number about the time whose total sum is smaller, and for comparing the log number of the calculation object of the reflection time with the log number of the calculation object of the decompression time, and for, when the log number of the reflection time is equal to the log number of the decompression time, performing compression processing to the log data included in the total sum of the decompression time. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a method for compressing log data, and more particularly to a method for determining a data range to be compressed.

  In a data system in which data is updated, if the amount of data handled is enormous, updating the master data database each time data is updated is insufficient in processing time, which is virtually impossible. It is possible. Therefore, it is a common practice to store updated data, which is differential information, as a log file and update the master data database in a batch using the log file at an arbitrary time. .

  When a disaster or the like occurs in the above data system and the database data used in the data system is restored, the log data is recorded in the master data using the master data immediately before and the log file after the master data. Restore the latest data system by reflecting the files sequentially.

Further, there is a method of compressing history data and restoring it at the time of use (see, for example, Patent Document 1). Here, the process of decompressing the compressed data and the process of using the decompressed data are processed in parallel.
JP 2004-185460 A

  The creation and storage of master data requires a substantial amount of cost, and there is a desire to reduce as much as possible. Here, the log file has a huge amount of data, and if the storage area of the log file can be secured, the update period of the master data can be extended, so the log file is compressed. It is desirable to do. On the other hand, if a database failure occurs, it is necessary to restore the data. However, if the log file is compressed in that case, the data decompression process will take time and the data can be restored quickly. Absent. Therefore, there is a need for a data compression method that enables rapid data recovery and that can effectively use the storage area of the log file.

  Update log file that stores log records, reflection time database that stores data reflection time for each log type, decompression time database that stores data decompression time per unit data volume, log record from one end of log file to be compressed Reflection time calculation means for obtaining the sum of the reflection times required for each time, decompression time calculation means for obtaining the sum of the decompression times required for each log in reverse order from the log record at the other end of the compression target log file, the reflection time and the decompression time Time comparison means for comparing the sum, means for obtaining data of the next log record for a time with a small sum of the reflection time and the decompression time, and a log record number for which the reflection time is to be calculated is a target for the calculation of the decompression time When the log record number exceeds the log record number included in the total decompression time There compressing log data by compression means for performing compression processing.

  Further, when the total of the decompression time is calculated by the decompression time calculating means, the total of the second reflection time for the log record is also calculated, and there is a log record added in the update log file. When the data compression process is performed for the log record, the reflection time calculation unit adds the total of the second reflection time to the total reflection time for the current data compression process, thereby adding log data. Compress.

  According to the present invention, since the decompression process is performed on the next data reflected data during the time required to reflect the data, the time required independently for the data decompression is unnecessary. On the other hand, since the data can be compressed, the amount of data that can be stored in the log file storage area increases.

Embodiments of the present invention will be described below with reference to the drawings.
Example 1
FIG. 1 is a system configuration diagram in the present embodiment.

  The system according to this embodiment includes a processing server 10, a data server 20, an update log file 30, a reflection time database 40, a decompression time database 50, a compression status database 60, and master data 70, which are connected by a network. ing.

  The data server 20 is a process of registering data in the update log file 30 for data information obtained from the processing server 10, a process of updating data to the master data 70 based on the update log file 30, and the latest database information. It is a device that performs processing to be provided to the processing server 10. The log file reflection process and the log file decompression process can be performed in parallel.

  The master data 70 refers to the latest data structure of the database held by the data server at an arbitrary time. Although the data server 20 has the latest database, when the latest database is lost due to a disaster or the like, the latest database is restored by the master data 70 and the update log file.

  The processing server 10 refers to the data information of the data server 20 based on processing instructions from the outside, performs processing, and outputs processing results to the outside. When the data information is updated at that time, It is a device that performs processing for transmitting updated data information to the data server 20. One or more processing servers 10 are connected to the data server 20.

  FIG. 2 is a configuration diagram of the update log file 30.

  The update log file 30 is a log file that stores update data obtained from the processing server 10 in chronological order.

  The SEC number 31 is a number uniquely assigned to a log record.

  The date and time 32 is the date and time when a log related to the update occurs due to the data update.

  The log type 33 is a classification of the data type of the log record.

  The content 34 is a substantial content of the log record.

  When the data information is updated in the processing server 10, the updated date and time 32 and the log type 33 corresponding to the data information are classified, and the data information is sent to the data server 20. In the data server 20, the SEC number 31 to be registered is set and stored in the update log file 30. At the same time, the database information is updated.

  FIG. 3 is a configuration diagram of the reflection time database 40.

  The reflection time database 40 is stored in advance for the relationship between the log type of log data and the time required for reflection of the log data in the database when the database is restored. The log type 41 is the same information as the log type 33 of the update log file 30. The reflection time 42 is a time required for reflecting the log type 41 in the database.

  FIG. 4 is a configuration diagram of the decompression time database 50.

  The decompression time database 50 is stored in advance for the time required for decompression when decompressing the compressed log data when the database is restored. The number of bytes 51 indicates the size of data to be handled. The decompression time 52 stores the time required for decompression according to the number of bytes.

  FIG. 5 is a configuration diagram of the compression status database 60.

  The compression status database 60 is a database that manages the state of the target log record group when the update log file 30 is compressed.

  The start SEC number 61 is an SEC number corresponding to the first log record when managing a plurality of log records as a group in the update log file 30.

  The end SEC number 62 is an SEC number corresponding to the last log record in the case where a plurality of log records are managed as one unit in the update log file 30.

  The pair ID 63 is a log data compression process, which will be described later. In one compression process, two log record groups, a log record group to be compressed and a log record group not to be compressed, are generated as a pair. This is an ID for managing a pair of log records as a unit.

  The compression flag 64 is information for distinguishing whether or not the corresponding log record group is a compression target. In this embodiment, corresponding to the pair ID 63, “0” is assigned to a log record group that is not a compression target, and “1” is assigned to a log record group that is a compression target.

  The reflection time 65 stores the time required to reflect the entire group of log records composed of the one set at the time of database recovery.

  The decompression time 66 stores the time required for decompressing the entire log record group composed of one compressed unit.

  The file name 67 is the name of a file to be saved when saving the log record group consisting of one compressed group.

  Next, processing when the update log file 30 is compressed will be described.

  The compression of the update log file 30 needs to be performed in consideration of the reflection time.

  Here, it is possible to perform other processing for the time required to reflect the data information. Therefore, if the time required for the reflection can be devoted to the decompression time of other data information, it is possible to prevent a delay in the data information reflection processing time due to waiting for the decompression process. Therefore, it is necessary to compress other data corresponding to the time for reflecting the data information in the database.

  Note that the compression processing of the update log file 30 is performed, for example, when a day of work is completed. For example, when the migration process to the master data 70 is performed once every seven days, update logs are accumulated for seven days. By compressing the update log for 7 days every day, the data area can be effectively used.

  FIG. 6 is a flowchart for specifying the compressed update log file 30.

  For convenience of explanation, a schematic diagram of the update log file 30 is shown in FIG.

The data compression program determines whether or not there is a newly added update log file 30 after performing the previous compression process (ST01).
If there is no newly added update log file 30 (ST01: No), the compression process is terminated. On the other hand, when there is a newly added update log file 30 (ST01: Yes), the processing after ST02 is performed.

  First, the reflection time [ht] is calculated as a variable for calculating the total reflection time, the decompression time [kt] is calculated as a variable for calculating the total decompression time, and the total time required for reflecting the log record group to be compressed is calculated. The reflection time [Kht] is initialized as a variable to be performed (ST02). As a result of initialization, the reflection time [ht], the decompression time [kt], and the reflection time [Kht] are 0 in this embodiment.

  Next, the head SEC number of the newly added target log file is searched from the compression status database 60, and the SEC number immediately before the head SEC number is extracted (ST03). In FIG. 10, (a1) indicates the head SEC number, and (a2) indicates the SEC number immediately before the head SEC number.

  Based on the record compression flag of the start SEC number and end SEC number of the compression status database 60 including the extracted SEC number, it is determined whether or not it is a compression target (ST04). In FIG. 10, in (a2), the immediately preceding SEC number is a determination target of whether or not it is a compression target.

  When it is a compression target (ST04: Yes), the reflection time of the record including the corresponding record in the compression status database 60 is added to the reflection time [ht] (ST05).

  The present invention prevents the delay in data reflection processing and the reduction in data compression efficiency by making the reflection processing time and the decompression processing time equal to each other when the other processing is completed. Is the purpose. Therefore, if the decompression log file generated by the previous reflection process and decompression process is subjected to the reflection process alone, the decompression process is not performed during that time, and the data reflection process is delayed. In addition, since there is a target to be decompressed corresponding to the reflection processing time, more log files can be compressed. Therefore, by adding the reflection time of the previous log file to the reflection time of the current log file, it is possible to set the decompression process for the reflection time. Therefore, the reflection time is added in ST05.

  In FIG. 10, the reflection time 65 corresponding to the log record group including the immediately preceding SEC number is added in (a3).

  Next, the start SEC number of the log record to be compressed this time in the update log file 30 is set as the initial value hs [0] of the reflected SEC number hs [x] (ST06).

  Further, the tail SEC number of the log record to be compressed this time in the update log file 30 is set as the initial value ks [M] of the decompressed SEC number ks [y] (ST07). In FIG. 10, the state (a4) corresponds.

  The initial value is set as described above, and then the actual compression target extraction process is performed.

  FIG. 7 is a flowchart (2) for specifying the compressed update log file 30.

  For convenience of explanation, FIG. 11 shows a schematic diagram of the current compression target of the update target log file.

  The data compression program determines whether or not the reflected SEC number hs [x] is equal to the decompressed SEC number ks [y] (ST11).

  When the reflected SEC number hs [x] is equal to the decompressed SEC number ks [y] (ST11: Yes), the log record group to be compressed is determined, and the processing after ST12 is performed.

  On the other hand, when the reflected SEC number hs [x] and the decompressed SEC number ks [y] are not equal (ST11: No), the processing after ST19 is performed to determine whether or not the SEC number should be compressed.

  The data compression program compares the reflection time [ht] with the decompression time [kt] (ST19).

  When the reflection time [ht] is smaller than the decompression time (ST19: Yes), the log type 33 of the update log file 30 and the log type 41 of the reflection time database 40 and the reflection for the log record corresponding to the reflection SEC number hs [x]. Based on the time 42, the reflection time [htx] required for the log record is calculated (ST25), and the reflection time [htx] is added to the reflection time [ht] (ST26).

  In FIG. 11, (b1) corresponds to the sum of [ht] and [htx] as a new [ht].

  Then, hs [x + 1] obtained by adding 1 to the reflected SEC number hs [x] is set as a new reflected SEC number hs [x] (ST27), and the processes after ST11 are performed.

  In FIG. 11, (b2) corresponds to hs [x + 1] obtained by adding 1 to hs [x] as a new reflection SEC number hs [x].

  If the reflection time [ht] is equal to or longer than the decompression time [kt] (ST19: No), the decompression time [kty] corresponding to the decompression SEC number ks [y] is calculated (ST20). This obtains the number of bytes of the log record corresponding to the decompression SEC number ks [y] from the contents 34 of the update log file 30 and searches the decompression time 52 corresponding to the number of bytes 51 of the decompression time database 50 based on the number of bytes. To find out.

  The thawing time [kt] obtained in ST20 and the thawing time [kt], which is the sum of the thawing times, are added to calculate a new thawing time [kt] (ST21).

  In FIG. 11, (c1) corresponds to the sum of [kt] and [kty] as a new [kt].

  Next, calculation of the reflection time [Khty] corresponding to the log record of the decompressed SEC number ks [y] is performed based on the log type 33 of the update log file 30, the log type 41 of the reflection time database 40, and the reflection time 42 (ST22). ), The reflection time [Khty] is added to the reflection time [Kht] to obtain a new reflection time [Kht] (ST23).

  Then, the decompressed SEC number ks [y-1] obtained by subtracting 1 from the decompressed SEC number ks [y] is set as a new decompressed SEC number ks [y] (ST24), and the processes after ST11 are performed.

  In FIG. 11, (c2) corresponds to ks [y−1] obtained by subtracting 1 from ks [y] as a new reflection SEC number ks [y].

  Next, when the reflected SEC number hs [x] is equal to the decompressed SEC number ks [y] (ST11: Yes), the following processing is performed.

  For the reflection SEC number hs [x] that is the current target, the data compression program performs processing to be added to the reflection time, the log type 33 of the update log file 30, the log type 41 of the reflection time database 40, and the reflection time 42. Based on the above, the reflection time [htx] required for the log record is calculated (ST12). Thereafter, the reflection time [htx] is added to the reflection time [ht] (ST13).

  In FIG. 11, (d) corresponds, and the reflected SEC number hs [x] and the decompressed SEC number ks [y] are in the same state. Further, the reflection time [htx] is added to the reflection time [ht].

  In the present embodiment, when the reflection time and the decompression time are the same value, the reflection time is prioritized, but a method in which the decompression time is prioritized is also possible.

  Next, log records from the SEC number hs [x + 1] obtained by adding 1 to the current reflected SEC number hs [x] to the initial value [ksM] of the decompressed SEC number are extracted (ST14).

  If there is a record (ST15: Yes), the log file from the reflected SEC number hs [x + 1] to the initial value [ksM] of the decompressed SEC number is compressed (ST16), and a compressed file is created (ST17). On the other hand, if there is no record (ST15: No), there is no compression target, so a compression status management process (ST18) is performed.

  FIG. 8 is a flowchart for registration in the compression status database 60.

  The data compression program newly sets the pair ID 63 (ST41). By this processing, a data record for storing the log record group to be compressed and a data record for storing the log record group to be compressed are created.

  Next, the start SEC number hs [0] is registered as the start SEC number 61, the reflection SEC number hs [x] is registered as the end SEC number 62 (ST42), the compression flag 64 is set to 0 (ST43), and the reflection time [ ht] is registered in the reflection time 65 (ST44). This completes registration of the uncompressed log record group.

  Next, the reflection number hs [x + 1] obtained by adding 1 to the reflection SEC number is set as the start SEC number 61, ks [M] is registered as the end SEC number 62 (ST45), and the compression flag 65 is set to 1 (ST46). The decompression time [kt] is registered in the decompression time 66 (ST47), the reflection time [Kht] is registered in the reflection time 65 (ST48), and the compressed file name 67 is registered (ST49). This completes the registration of the log record group to be compressed.

  Note that the order of ST42 to ST49 in FIG. 8 is arbitrary.

  FIG. 9 is a flowchart of processing at the time of data recovery.

  At the time of data reflection, two data records corresponding to the pair ID 63 in the compression status database 60 are extracted (ST31).

  Next, the log data group corresponding to the compression flag 64 of 0 is registered in the “reflection” processing SEC number (ST32). This is to register the start SEC number 61 to the end SEC number 62. Further, the file name 67 having the compression flag 64 of 1 is registered in the “decompression” process SEC number (ST33).

  Each data information reflection process and compressed data information decompression process are performed (ST34). The processing completion timing of each processing is almost the same time.

  Next, the presence / absence of the next pair ID 63 is determined (ST35).

  If the next pair ID 63 exists (ST35: Yes), the decompressed log data is registered in the “reflection” processing SEC number (ST36).

  On the other hand, when the next pair ID 63 does not exist (ST35: No), the “reflect” processing is performed on the decompressed log data performed in ST34 (ST37), and the recovery is complete, so the processing ends.

  FIG. 12 is a diagram illustrating an example of a hardware configuration of the data server 20 described above.

  The data server 20 shown in the figure has a CPU 21, memory 22, input means 23, output means 24, storage means 25, network connection means 26, etc., which are connected to a bus 27. The configuration shown in the figure is an example, and the present invention is not limited to this.

  The CPU 21 is a central processing unit that controls the entire data server 20.

  The memory 22 is a memory such as a RAM that temporarily stores programs and data stored in the storage means 25 when executing programs, updating data, and the like. The CPU 21 uses the program and data read out to the memory 22 to execute various processes including the process related to the data compression process described with reference to FIGS.

  The input means 23 is, for example, a keyboard, a mouse, a touch panel, or the like.

  The output unit 24 is, for example, a display (display device).

  The storage unit 25 is, for example, a hard disk device or the like, and includes a program (including a program for causing the data server 20 to execute processing related to the above-described data compression processing operation) and data (the above-described update log file 30, reflection time database 40, decompression). A time database 50, a compression status database 60, and master data 70) are stored.

  The network connection means 27 is configured to be connected to a network (Internet, LAN, etc.) and to transmit / receive programs and data to / from an external information processing apparatus.

  Moreover, the recording medium which recorded the said program or the download of a program is also possible.

  The program and data for executing the various processes described above may be read out, stored in the memory 22 and executed, and the program and data may be stored in a network (Internet or the like) connected by the network connection means 27. Alternatively, an external program or a program or data stored in a server may be downloaded.

  Although the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and it is needless to say that various improvements and changes may be made without departing from the gist of the present invention.

FIG. 1 is a system configuration diagram in the present embodiment. FIG. 2 is a configuration diagram of the update log file 30. FIG. 3 is a configuration diagram of the reflection time database 40. FIG. 4 is a configuration diagram of the decompression time database 50. FIG. 5 is a configuration diagram of the compression status database 60. FIG. 6 is a flowchart for specifying the compressed update log file 30. FIG. 7 is a flowchart (2) for specifying the compressed update log file 30. FIG. 8 is a flowchart for registration in the compression status database 60. FIG. 9 is a flowchart of processing at the time of data recovery. FIG. 10 is a schematic diagram of the update log file 30. FIG. 11 is a schematic diagram of the current compression target of the update target log file. FIG. 12 is a diagram illustrating an example of a hardware configuration of the data server 20 described above.

Explanation of symbols

10 processing server 20 data server

Claims (3)

A log data compression program for compressing log data,
On the computer,
An update log file that stores log records,
Reflection time database that stores data reflection time for each log type,
Decompression time database that stores data decompression time per unit data amount,
Reflection time calculation means for obtaining the sum of reflection times required for each log in order from the log record at one end of the compression target log file,
Decompression time calculation means for obtaining the sum of the decompression times required for each log in reverse order from the log record at the other end of the compression target log file;
A time comparison means for comparing the sum of the reflection time and the thawing time;
A means of obtaining data for the next log record for times when the sum is low,
Compression means for performing compression processing on log records included in the sum of the decompression times when the log record number for which the reflection time is to be calculated is equal to or greater than the log record number for which the decompression time is to be calculated;
Log data compression program characterized by functioning as When calculating the total of the thawing time by the thawing time calculating means, also calculate the total of the second reflection time for the log record,
When there is a log record added in the update log file and data compression processing is performed for the log record,
2. The log data compression program according to claim 1, wherein the reflection time calculation means adds the total of the second reflection times to the total reflection time for the current data compression processing. A log data compression method for compressing log data,
For log records,
Reflection time calculation step for obtaining the total reflection time required for each log in order from the log record at one end of the compression target log file based on the data reflection time for each log type stored in advance,
Decompression time calculation step for obtaining the total of the decompression time required for each log in reverse order from the log record at the other end of the compression target log file based on the data decompression time per unit data amount stored in advance,
Comparing the sum of the reflection time and the thawing time;
Obtaining data for the next log record for times when the sum is low,
When the log record number for which the reflection time is calculated is equal to or greater than the log record number for which the decompression time is calculated, a step of performing compression processing on the log records included in the total of the decompression time;
A log data compression method comprising: