JP5432596B2 - Log file management device, log file management system, log file management method and program thereof - Google Patents

Description

  The present invention relates to a technique for creating a shared file that generates and accumulates a commit log indicating the update contents of each record data based on a record data update operation, and synchronizes a trigger for deleting a commit log that is no longer needed. .

  Conventionally, in database management in a distributed system, for each record set in which record data of the same schema is collected, identification information is added to update difference information, which is difference information from the previous update of record data to be updated (hereinafter referred to as “difference information”). , "Commit log" file) was prepared. However, in this method, when an update from a client apparatus to a plurality of record sets occurs, a commit log file is provided for each record set, so the number of I / O (Input / Output) times associated with access to the commit log file is reduced. There is a problem that the network load increases to increase the response of the database update.

  To solve the above problem, a commit log file is not prepared for each record set in which record data of the same schema is collected, but a single commit log file is shared and written by a plurality of record sets. Has been proposed. In this method, when updates to a plurality of record sets occur at a time, it is possible to collectively update the information to be updated, and it is possible to improve the response of the database update. However, since the commit log is generated for each update, if it is left unattended, it becomes enormous.

  Therefore, there is a method for deleting a commit log after a predetermined time has elapsed. In this method, a snapshot of a record set is acquired every predetermined period, and the commit log is deleted with a period before the acquired period as a predetermined period. As a result, the record set can be restored with the acquired snapshot and the remaining commit log file.

  There is also a method of deleting old commit logs exceeding a predetermined amount when a predetermined amount of commit logs are accumulated. A snapshot is acquired when a predetermined amount of commit logs are accumulated, and the previous commit log is deleted from the acquired point, so that a record set can be restored with this snapshot and the remaining commit log file ( For example, refer nonpatent literature 1).

  In addition, even in a configuration in which a plurality of replication destination replica storage devices having a database in which the same contents as the record data of the master storage device having the replication source database are replicated are arranged, a certain period of time has elapsed as described above. By notifying the time point or the time point when a predetermined amount is accumulated between the master storage device and the replica storage device, the commit log can be deleted.

"Series ORACLE MASTER Silver DBA point explanation 3rd control file and redo log file management", Tomoko Onodera, IT Self Strategic Research Institute, [online], 2004 August 27, [April 16, 2009 search ], Internet, <URL: http://jibun.atmarkit.co.jp/Iskill01/rensai/omsdb03/omsdb01.html>

  However, when the commit logs of a plurality of record aggregates are combined into one file as described above, the timing for acquiring a snapshot from each record aggregate is not necessarily the same. There is a risk that even the commit log that is originally necessary will be deleted if triggered by the accumulation of fixed amounts. Therefore, it is necessary to determine which commit log can delete the past one.

  Further, in the configuration of the commit log as described above, when the configuration is such that there are a plurality of replication destination replica storage devices for one replication source master storage device, the replica storage device has record data that the master storage device has. The update of the record data in the replica storage device may be delayed depending on the communication status and load status. When a delay occurs in the synchronization of the replica storage device, the master storage device needs to consider the update status of the replica storage device in the process of deleting the commit log. Similarly, even if the replica storage device wants to delete unnecessary commit logs, the replica storage device replaces the master storage device when the master storage device fails and cannot operate. It is necessary to execute commit log deletion considering the update status of replica storage devices other than itself.

  In view of the above-described problems, the present invention provides a log file management apparatus and system for determining and triggering a trigger for deleting a commit log that is no longer needed by centralizing and managing a commit log generated by updating record data It is an object to provide a method and a program.

  In order to solve the above-described problem, the present invention generates and accumulates a commit log indicating the update contents of each record data based on the update operation of the record data for a record set that is a set of record data A log file management device that creates and manages the shared file, a commit log storage unit that stores the shared file, a request reception unit that receives an update request for the record data from a client device, and a basis of the update request The commit log is generated, and the commit log generation unit that persists the commit log to the shared file and the record data stored in the record data storage unit are acquired as a checkpoint at an arbitrary timing, and are persisted. For each record set, compare the point in time when the checkpoint was acquired and An update execution unit that deletes the previous commit log from which the old checkpoint was acquired as an unnecessary commit log, and the record data at the predetermined timing is acquired from the record data storage unit as a checkpoint and is made permanent. A checkpoint acquisition unit stored in a storage unit, and a commit log control unit that deletes a commit log before the checkpoint acquisition from the commit log accumulated in the shared file based on an instruction from the update execution unit. It is characterized by providing.

  According to the present invention, when a commit log generated by a record data update request is made permanent in a shared commit log file and a checkpoint is acquired at an arbitrary timing, a checkpoint is acquired for each record set. Since the commit logs before the time when the oldest checkpoint is acquired are deleted as unnecessary commit logs, it is possible to quickly identify and delete an appropriate delete point.

Further, the present invention generates and accumulates a commit log indicating the update contents of each record data, based on the update operation of the record data, for a record set that is a set of record data shared between nodes. A log file management system for creating and managing a shared file, a master log file management device that receives an update request for the record data from a client device and updates the record data, and a log file management for the master A replica log file management device that replicates and updates the same record data in response to the update of the record data of the device, and each of the log file management devices stores a commit log storage unit that stores the shared file And an update request for the record data from the client device A request receiving unit that receives the commit log, generates a commit log based on the update request, and makes the commit log permanent to the shared file; and a commit log accumulated in the shared file for each record set Update execution unit for instructing to delete the commit log exceeding the predetermined range as an unnecessary commit log when the commit log exceeds the predetermined range; and checkpoint acquisition unit that stores the checkpoint storage unit obtains the record data stored in the record data storage unit as a checkpoint, the basis of an instruction from the update execution unit, stored committed to the shared file Commit log controller that deletes unnecessary commit logs from the log and other log file management devices And when the log file management device is the master log file management device, the log file management device is further connected to each of the replica log file management devices. In response to the record data update in the master log file management device, the replication request is sent to request that the same record data be replicated and updated, and the replication request is sent from each replica log file management device. A replication control unit that receives a response indicating that the persistence of the commit log relating to the record data update based on is completed, and the log file management device of the master and the replica has the same value as the value indicating the predetermined range It is characterized by using.

  According to the present invention, the master log file management device (hereinafter referred to as “master”) can delete from the latest commit log in the same range as the replica log file management device (hereinafter referred to as “replica”). By using, it is possible to delete commit logs outside the range set in advance by the master and replica independently, so it is possible to delete previous commit logs without querying the replica It becomes. Therefore, there is no communication for inquiries regarding deletion between the master and each replica, so that the load on the network can be reduced.

Further, the present invention generates and accumulates a commit log indicating the update contents of each record data, based on the update operation of the record data, for a record set that is a set of record data shared between nodes. A log file management system for creating and managing a shared file, a master log file management device that receives an update request for the record data from a client device and updates the record data, and a log file management for the master A replica log file management device that replicates and updates the same record data in response to the update of the record data of the device, and the log file management device includes a commit log storage unit that stores the shared file; Receiving an update request for the record data from the client device. A request accepting unit, a commit log generating unit that generates the commit log based on the update request and makes the commit log permanent to the shared file, and a commit log accumulated in the shared file for each record set and updating execution unit for instructing deletion of unnecessary commit log out of a checkpoint acquisition unit that stores the checkpoint storage unit obtains the record data stored in the record data storage unit as a checkpoint, the update Communication between a commit log control unit that deletes a commit log before the oldest checkpoint acquisition time from the commit log accumulated in the shared file based on an instruction from the execution unit, and another log file management device A communication control unit that performs the log management of its own log file management device. When it is a file management device, the log file management device further replicates the same record data to each of the replica log file management devices in response to the update of the record data in the master log file management device. A replication control unit that sends a replication request for updating, and receives a response from the respective replica log file management devices indicating that the commit log persistence related to the record data update based on the replication request has been completed. The update execution unit of the master log file management device selects the oldest commit log among the commit logs of the replica log file management device, and does not need a commit log generated before the commit log. Delete as commit log Remove the commit log to its own log file management apparatus has with log file management apparatus F notification of the replica to the effect that, update execution part of the log file management apparatus of the replica on the basis of the said notification The commit log is deleted.

According to the present invention, since the master manages the application status of the commit log that the master and the replica have, and notifies the application status from the master to the replica, there is no need to make an inquiry about the application status between the master and each replica, It is possible to reduce the load on the network.

According to the present invention, the log file management apparatus can appropriately specify a deletion point for deleting a commit log and quickly delete it.
Further, the log file management system can reduce the load on the network by deleting the commit log without depending on other nodes.

It is a block diagram of the log file management apparatus which concerns on 1st Embodiment. It is a figure which shows the relationship between a record set and a checkpoint. It is a flowchart which shows the flow of a process of the log file management apparatus which concerns on 1st Embodiment. It is a figure explaining the determination method of the commit log which may be deleted. It is a figure explaining the method of deleting a commit log on the basis of commit log ID. It is a figure which shows the method of deleting a commit log on the basis of time. It is a figure explaining the method of swapping a commit log file. It is a figure which shows the list | wrist of deletion determination of the commit log file in 2nd Embodiment. It is a figure which shows the relationship between the client apparatus and log file management apparatus in 3rd Embodiment. It is a block diagram of the log file management system which concerns on 4th Embodiment. It is a flowchart which shows the flow of a process in the log file management system which concerns on 4th Embodiment. It is a figure explaining the resending request | requirement from a replica to a master. It is a block diagram of the log file management system which concerns on 4th Embodiment. It is a figure which shows application of the commit log in 5th Embodiment. It is a flowchart which shows the flow of a process in the log file management system which concerns on 5th Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a configuration diagram of a log file management apparatus according to the first embodiment. As shown in FIG. 1, the log file management device 100 includes a request receiving unit 2 that receives a record data update request from the client device 1, a commit log generating unit 3 that generates a commit log in response to the update request, and unnecessary. An update execution unit 4 that determines the timing of deleting a commit log that has been deleted and instructs the deletion, a commit log control unit 5 that deletes a commit log that is no longer needed, and obtains record data at a predetermined timing as a checkpoint A checkpoint acquisition unit 6 (hereinafter referred to as “checkpoint acquisition”), a record data storage unit 7 that stores record data for each record set, a commit log storage unit 8 that stores a commit log, and a checkpoint file (Hereinafter referred to as “checkpoint file”) It consists Kkupointo storage unit 9. Hereinafter, each component will be described in detail.

  The request receiving unit 2 receives a record data update request from the client device 1. Here, the request received from the client apparatus 1 includes update and reference of record data. In the embodiment of the present invention, update of record data is treated as a process. Update target record data is stored for each record set. A record set indicates a table when a database is taken as an example.

  Based on the update request, the commit log generation unit 3 generates a commit log in which a unique ID (Identification) is added to update difference information that is difference information from the previous update of record data to be updated.

  The update execution unit 4 instructs the commit log control unit 5 to make the commit log permanent (store) in the commit log storage unit 7. Furthermore, the checkpoint acquisition unit 6 is instructed to acquire the checkpoint at an arbitrary timing, and the time when the checkpoint is acquired is compared for each record set, and the commit log before the oldest checkpoint is acquired. The commit log is determined to be unnecessary, and the commit log control unit 5 is instructed to delete this commit log. Furthermore, the record data corresponding to the deleted commit log is updated with respect to the record data in the record data storage unit 7.

Based on the instruction from the update execution unit 4, the commit log control unit 5 makes the commit log permanent in the commit log storage unit 8 and deletes the commit log that is no longer needed.
The checkpoint acquisition unit 6 acquires a checkpoint according to an instruction from the update execution unit 4 and stores it in the checkpoint storage unit 9.

  The update execution unit 4, the commit log control unit 5, and the checkpoint acquisition unit 6 are realized by program processing by a CPU (Central Processing Unit) included in the log file management apparatus 100 and a dedicated circuit. Furthermore, the record data storage unit 7, the commit log storage unit 8, the checkpoint storage unit 9, and a storage unit (not shown) for storing the program for realizing the functions of the log file management device 100 include a RAM (Random It is composed of a storage medium such as an access memory (ROM), a read only memory (ROM), a hard disk drive (HDD), and a flash memory.

  Here, the timing of checkpoint acquisition is an arbitrary timing. This arbitrary timing is the amount of the commit log that is made permanent when the number of times the commit log is generated exceeds a predetermined threshold. When a predetermined capacity is exceeded in the commit log storage unit 8, when a predetermined time has elapsed since the previous checkpoint acquisition, when the amount of record data stored in the record data storage unit 7 exceeds a predetermined capacity, Or the timing set arbitrarily.

  FIG. 2 is a diagram illustrating the relationship between the record set in the record data storage unit 7 and the checkpoints in the checkpoint storage unit 9. In FIG. 2, one checkpoint file is associated with one record set, but a structure having a plurality of checkpoint files may be used. When there are a plurality of files, there are both a case of having a plurality of files for each part of the record set according to the schema information and a case of holding a plurality of checkpoint files acquired at different timings (holding a plurality of generations).

  Next, the flow of processing in the log file management apparatus 100 will be described with reference to the flowchart shown in FIG. 3 (see FIG. 1 as appropriate). First, the update request unit 2 receives a record data update request from the client device 1 (S1). Next, on the basis of this update request, the commit log generation unit 3 generates a commit log, and the update execution unit 4 instructs the commit log control unit 5 to make the commit log permanent, and the commit log storage unit 7 makes it permanent. (S2). Next, the update execution unit 4 instructs the checkpoint acquisition unit 6 to acquire a checkpoint at a predetermined timing, and acquires a checkpoint for each record set (S3).

  Next, the update execution unit 4 compares the checkpoint acquisition time for each record set, and determines the oldest checkpoint acquisition time (S4). Next, the update execution unit 4 instructs the commit log control unit 5 to delete the commit log before the oldest checkpoint acquisition time and deletes it (S5). Next, the update execution unit 4 updates the record data corresponding to the deleted commit log on the record data in the record data storage unit 7 (S6). Here, the process of S6 may be executed in synchronization with the deletion timing or may be executed asynchronously. This completes the deletion of the commit log that is no longer needed.

  Here, the commit log is sequentially applied to the record data storage unit 7 in the recovery order for recovering the database according to the permanent order. For this reason, the commit log has a sequential number indicating the order of persistence and / or the generated time as the persistence information together with update information (record data and the address of the record set). Here, the sequential number is called a commit log ID.

  Next, a method for determining a commit log that may be deleted will be described in detail. FIG. 4 is a diagram illustrating a method for determining a commit log that may be deleted. As illustrated in FIG. 4, it is assumed that the update execution unit 4 acquires a checkpoint file for each record set at timings t1, t2, t3, and t4. The record set A acquires a checkpoint file (CPA) at the timing t1, and the commit log ID at that time is “25”. The record set B obtains the checkpoint file (CPB) at the timing t4, and the commit log ID at that time is “69”. The record set C acquires a checkpoint file (CPC) at the timing t3, and the commit log ID at that time is “52”. The record set D obtains a checkpoint file (CPD) at the timing t4, and the commit log ID at that time is “33”. Here, the time is t0 <t1 <t2 <t3 <t4 <current order, and the larger the commit log ID, the newer the log.

  For example, if it is considered that the commit log is to be deleted at this time, the record set A can be deleted before the commit log ID = 25 (or time t1), and the record set B can be deleted with the commit log ID = 69 ( Or before time t4), the record set C is before commit log ID = 52 (or time t3), the record set D is before commit log ID = 33 (or time t2), and the oldest record set is Since it is the record set A, it is determined that the commit log before commit log ID = 25 (or time t1) may be deleted. If the commit log ID = 33 or earlier is deleted, the record set A cannot be restored to the current state.

  A method for performing the determination more simply will be described. FIG. 5 is a diagram for explaining commit log deletion based on the commit log ID. In FIG. 5, record set information is managed in a unidirectional list structure with a commit log ID as a key, and the top of the list represents the oldest data.

  First, since the commit log is not made permanent at the initial startup, the list is created in an arbitrary order with the commit log ID = 0 for each record set (in this case, the order created in the list is a problem) Anything is fine. Next, when a checkpoint is acquired in the record set A, the elements of the record set A that were created in an arbitrary order are deleted from the list, and the commit log ID = at the end of the list (the latest in time series) At 25, record set A is added. Record set D, record set C, record set B and records in any order are deleted in chronological order, and addition is repeated from the end. If this list structure is created, if you want to delete commit logs at any time, you can refer to the top of this list and delete the commit logs before the commit log ID indicated by the record set.

  In the above, the elements of all record sets are added at the time of initial activation, but a method of adding elements at the timing when each record set acquires a checkpoint for the first time is also conceivable. In this case, when determining whether to delete the commit log, the number of elements in the list is checked, and the commit log is deleted only when the number matches the number of record sets. In the case of the example shown in FIG. 5, if the previous data is deleted based on the record set C, the commit log ID = 26 to 51 among the data necessary to restore the record set A is the record set C. Commit log IDs = 34 to 51 are lost among the data necessary for restoring the records, and the record set A and the record set C cannot be restored.

  The time can also be used for determination of deletion. This case is basically the same as the case of using the commit log ID, but the determination element changes to the time. FIG. 6 is a diagram illustrating a method of deleting a commit log based on time. In the example shown in FIG. 6, when the base time that can be deleted is determined, all the commit logs before the base time are deleted. Assuming that the commit log file is added in chronological order from the beginning of the file, if the commit log is written in one file, the time from the beginning to the base time is deleted.

  According to the first embodiment, the commit log generated by the record data update request is associated with time-sequential sequential identification information and made permanent in the shared commit log file. When a checkpoint is acquired, the point in time when the checkpoint was made permanent is compared for each record set, and the oldest commit log before the point when the checkpoint was made permanent is deleted as an unnecessary commit log. Therefore, it is possible to quickly identify and delete an appropriate deletion point.

(Second Embodiment)
In the second embodiment, an example in which the commit log file is logically composed of a plurality of files in the commit log storage unit 7 will be described. If the commit log file spans multiple files, swap processing is performed to sequentially write the commit log to multiple commit log files, and files containing only the commit log before the base point indicating the timing to delete are deleted. In the file containing the base point, the part from the head to the position of the base point is deleted. The basic idea is the same as in the first embodiment, but it is assumed that commit log deletion is deleted in units of files.

  FIG. 7 is a diagram for explaining a method of swapping commit log files. In FIG. 7, three commit log files are prepared. When the commit log ID is 0 to 20, the commit log file 1 is used. When the commit log ID is 21 to 78, the commit log file 2 is used. The subsequent files are respectively stored in the commit log file 3.

  It is assumed that a commit log file 1 is generated at the time of initial activation, a file that files a commit log is swapped to the commit log file 2 at the timing t1, and is swapped to the commit log file 3 at the timing t6.

  FIG. 8 is a diagram illustrating a list of commit log file deletion determinations according to the second embodiment. FIG. 8A shows a case where the commit log is deleted based on the time, and FIG. 8B shows a case where the commit log is deleted based on the commit log ID.

  For example, when commit log deletion is performed at the current time shown in FIG. 7, if determination is performed as in the first embodiment, it is determined that commit log ID = 25 or earlier, that is, commit logs before time t2 can be deleted. Can do. The difference from the first embodiment is that each commit log file is searched in units of files, and the range of commit logs included in the file is confirmed. Commit log file management information may be created in the commit log storage unit 7 and the range of commit logs included in the commit log file may be confirmed with reference to this management information. The commit log management information is information that associates a commit log file with a range of commit logs (ID) included in the file.

  The commit log file 1 includes commit log IDs with commit log IDs 0 to 20 (time t0 to t1), and it is determined that the commit log file 1 may be deleted this time. Accordingly, here, only the commit log file 1 is deleted. When the commit log file 2 is examined, since the commit logs after the commit log ID 25 are included, the commit log file 2 can be left as it is.

  According to the second embodiment, the commit log can be deleted in units of the commit log file, and the commit log is faster than deleting a part of the file when the commit log file is composed of one. Can be deleted.

(Third embodiment)
FIG. 9 is a diagram illustrating a relationship between the client device 1 and the log file management device 100 according to the third embodiment. In the third embodiment, an example in which data of one record set is distributed to a plurality of log file management apparatuses 100 will be described. For example, it is assumed that one table (record set) is physically divided and stored by prefecture. From the client device 1, it is not conscious that the tables are individually stored separately, and is seen as one table. In other words, each log file management apparatus 100 holds different data. Further, since it is considered that the log files are physically distributed in different log file management apparatuses 100, one commit log storage unit 7 is arranged in each log file management apparatus 100.

  Even in the above-described case, the deletion condition of the commit log is not limited depending on each log file management device 100 as long as recovery is possible for each log file management device 100. The same determination as in the second embodiment may be performed by each log file management apparatus 100.

  According to the third embodiment, log file management apparatuses 100 can be installed in a distributed manner, and commit logs can be deleted at high speed without depending on other log file management apparatuses 100.

(Fourth embodiment)
FIG. 10 is a configuration diagram of a log file management system according to the fourth embodiment.
As shown in FIG. 10, a log file management system 200 receives a record data update request from the client device 1 and performs a record data update process (hereinafter referred to as “master 40”), Corresponding to the update of the record data of the master log file management device, a replica log file management device (hereinafter referred to as “replica 50”) that replicates and updates the same record data is provided.

  Each of the master 40 and the replica 50 includes a request reception unit 2 that receives a record data update request from the client device 1, a commit log generation unit 3 that generates a commit log in response to the update request, and deletes a commit log that is no longer needed. The update execution unit 14 that determines the timing to perform deletion and instructs the deletion, the commit log control unit 15 that deletes the unnecessary commit log, and the amount of the commit log within a predetermined range (hereinafter referred to as “window width”) A checkpoint acquisition unit 6 that performs a checkpoint acquisition at the time of reaching, a record data storage unit 7 that stores record data for each record set, a commit log storage unit 8 that stores a commit log, and a checkpoint file Checkpoint storage unit 9 to store and replication to control replication request It provided with Shon control unit 16, a communication control unit 17 that communicates with the other log file management apparatus.

  In the following, parts having the same functions as those in the first embodiment are denoted by the same reference numerals, and duplicate descriptions thereof are omitted.

  Further, the master 40 transmits a replication request for replicating and updating the same record data to each replica 50 in response to the update of the record data in the master 40, and from each replica 50 to the replication request. The replication control unit 16 receives a response to the effect that the commit log persistence related to the record data update based on the completion is completed.

  Here, the timing of checkpoint acquisition is when the amount of commit logs reaches the predetermined window width. This is made permanent when the predetermined window width is exceeded by the number of times the commit log is generated. When the amount of committed logs exceeds a predetermined capacity in the commit log storage unit 8, when a predetermined time has elapsed since the previous checkpoint acquisition, or the amount of record data stored in the record data storage unit 7 Is when the specified capacity is exceeded. It is also possible to acquire a checkpoint at an arbitrary timing.

  When the update execution unit 14 of the master 40 determines whether the amount of commit logs is within a predetermined window width, the window of the amount of commit logs used in each replica 50 is used as the window width of the amount of commit logs. Use the same value as the width. Further, the update execution unit 14 of the master 40 notifies each replica 50 that the persistence of the commit log related to the record data has been completed from a majority of replicas 50 out of the total number of replicas 50 in response to the replication request. When received, the commit log relating to the record data is regarded as a permanent commit log, and it is determined whether or not the amount of the commit log deemed permanent is within a predetermined window width.

  Since the relationship between the record set and the check points in the fourth embodiment is the same as that in the first embodiment, description thereof is omitted here.

  Next, the flow of processing in the log file management system 200 will be described with reference to the flowchart shown in FIG. 11 (see FIG. 10 as appropriate). First, the update request unit 2 of the master 40 receives a record data update request from the client device 1 (S11). Next, on the basis of this update request, the commit log generation unit 3 generates a commit log, and the update execution unit 14 instructs the commit log control unit 15 to make the commit log permanent and makes the commit log storage unit 7 permanent. (S12).

  Next, the replication control unit 16 of the master 40 transmits a replication request to each replica 50 (S13). When the notification that the persistence of the commit log related to the record data is completed is received from each replica 50 from the majority of replicas out of the total number of replicas 50 (Yes in S14), the update execution unit 13 of the master 40 Assuming that the commit log of record data is made permanent, this commit log is counted (S16). On the other hand, when the notification cannot be received from the majority of replicas (No in S14), it is determined that the update execution unit 13 error of the master 40 is present (S15).

  Next, the update execution unit 14 of the master 40 determines whether or not the amount of commit logs in its own commit log storage unit 8 has reached a predetermined window width (S17). When the predetermined window width is reached (Yes in S17), the checkpoint acquisition unit 6 performs checkpoint acquisition (Yes in S17). On the other hand, when the predetermined window width is not reached (No in S17), the record data is updated to the record data storage unit (S20). Next, the master update execution unit 14 instructs the commit log control unit 15 to delete the commit log that is outside the predetermined window width (S19). Next, the update execution unit 14 updates the record data corresponding to the deleted commit log to the record data storage unit 7 (S20). The process of S20 may be executed in synchronization with the deletion or may be executed asynchronously. This completes the deletion of the commit log that is no longer needed.

  In the configuration including the master 40 and the replica 50 according to the fourth embodiment, the master 40 receives an update request from the client device 1 and transmits a replication request (hereinafter referred to as “replication request”) to the replica 50 (S13 described above). ). On the replica 50 side, the commit log generated by the replica 50 in response to the replication request is made permanent in the commit log storage unit 8. Here, assuming that the replica 50 has the same device configuration as that of the master 40, it does not matter whether or not the record data storage unit 7 reflects the update of the record data. The reason is that the update may be reflected in the record data storage unit 7 in real time, or the record data from the commit log of the new master 40 at the timing when the master 40 goes down due to failure or the like and the replica 50 becomes the new master 40. This is because the storage unit 7 may be updated. That is, it is only necessary that the same commit log on the replica 50 side as that on the master 40 side is made permanent in the commit log storage unit 8.

  However, the update order to the record data storage unit 7 in the replica 50, that is, the order of permanent commit log needs to be the same as that of the master 40. This is because an inconsistency may occur when the application order of the update data is changed. For example, it is assumed that the record set A is newly inserted when the commit log ID is 1, and the record data of the record set A is changed when the commit log ID is 2. If this order is changed in the replica 50, it is applied from the record data change of the record set A. However, since the record set A does not exist yet, an error occurs.

  Therefore, each time the master 40 makes a replication request, a monotonically increasing serial number is assigned to the commit log, and the replica 50 determines whether or not the order has changed as described above based on this serial number. Do. Here, the serial number may be used as the commit log ID as it is, or the serial number and the commit log ID may be different systems.

  In the replica 50, the serial number of the last committed commit log is stored in a serial number storage unit (not shown) and the extent to which the commit log storage unit 8 has been made permanent is stored. When a copy request is received, the serial number and the serial number in the serial number storage unit are compared to determine whether there is a missing tooth. For example, when the serial number “3” is stored in the serial number storage unit, it is expected that the replication request No. 4 will come from the master 40 next, but the actual number came from the fifth or later. In the case, it is determined that the tooth is missing. In this case, there is a possibility that No. 4 could not arrive due to a communication failure or the like. Therefore, the replica 50 requests retransmission of the copy request, waits for the arrival of No. 4, and then the No. 4 and No. 5 replication processes ( Update to the commit log storage unit 8). Here, when it is determined that a plurality of serial numbers are missing, a re-transmission request is issued accordingly. Re-transmission requests may be issued one by one or may be issued together. On the other hand, since the master 40 does not know when the retransmission request from the replica 50 comes, the master 40 cannot determine the timing for deleting the commit log.

  Therefore, the master 40 uses the same value as the window width of the amount of commit logs used in each replica 50 as the window width of the amount of commit logs to be held, and commits outside the window width range from the replica 50 side. If a log re-transmission is requested, it is not already present, so a response is made to the replica 50 side so as to rise from the initial state. Upon receiving such a response, the replica 50 receives the start-up request from the initial state, re-transmits the checkpoint and commit log of the master 40 from the master 40, and uses this to use the same state as the master 40. Perform recovery to.

  In the state as described above, the window width (Window size) determined by the master 40 from the latest commit log is, for example, for a predetermined period (for example, the past one hour) or a predetermined number (for example, 100 for the latest). ), And it is determined that previous commit logs can be deleted. It is also possible to perform deletion determination together with a retransmission request for a copy request.

  FIG. 12 is a diagram for explaining a retransmission request from the replica 50 to the master 40. As shown in FIG. 12A, the master determines the amount of commit log to be held as the window size. In this case, the master holds the commit logs 1 to 10, but since the window size is “5”, the commit logs 1 to 5 are commit logs that may be deleted. In FIG. 12B, the replica 1 holds the commit logs 1 to 7, but the commit logs 8 to 10 are retransmitted from the master 40 to be updated. In FIG. 12C, it is desired to request retransmission of the commit logs 4 to 9, but cannot be retransmitted because it exceeds the window size. Therefore, a recovery route started from the initial state is entered for recovery. That is, the replica 50 has the master 40 retransmit the checkpoint file and the commit logs 5 to 9.

  According to the fourth embodiment, the master 40 uses the same window width as the replica 50 to determine how far the latest commit log can be deleted. Since the other commit logs can be deleted independently, it is possible to delete previous commit logs without inquiring the replica 50. Therefore, since there is no communication for inquiry about deletion between the master 40 and each replica 50, it is possible to reduce the load on the network.

(Fifth embodiment)
In the fifth embodiment, as described in the fourth embodiment, the master and replica do not perform the deletion autonomously, but the master determines the deletion possible range and notifies each replica. It is.

FIG. 13 is a configuration diagram of a log file management system according to the fifth embodiment. The parts having the same functions as those in the fourth embodiment are denoted by the same reference numerals, and the description thereof is not repeated.
After the update execution unit 24 of the master 60 transmits a replication request to each replica 70, each replica 70 transmits a response (ACK) indicating that the replica is complete to the master 60. Based on this response, the update execution unit 24 of the master 60 creates and holds a commit log application status table indicating to which commit log ID the master 60 itself and each replica 70 have been persisted, When an ACK is received from each replica 70, this commit log application status table is updated.

  FIG. 14 is a diagram illustrating application of the commit log according to the fifth embodiment. In FIG. 14, communication interruption occurs between the replica 4 and the master when a replication request after the commit log ID = 9 occurs, and the replica 4 has only completed the application of the commit log ID = 8. After that, after the replication with the commit log ID = 10 shown in FIG. 14, the communication interruption between the replica 4 and the master is resolved. As shown in FIG. 14, in response to the transmission of the replication request with the commit log ID = 10, the replica 70 (here, replicas 1 to 3) transmits ACK to the master, but the replica 4 is committed. Since the perpetuation is completed only up to log ID = 8, ACK cannot be transmitted. The master update execution unit 24 receives the ACKs from the replicas 1 to 3 to update the commit log application status table based on the ACKs so as to grasp the application status of the latest commit log ID for each replica. The commit log ID that can be deleted can be notified to each replica. Further, when the master makes a replication request with the commit log ID = 11, the replica 4 transmits a retransmission request with the commit log ID = 9, 10 to the master.

  The commit log can be deleted at any timing on the master 60 side. For example, the commit log is deleted at a timing at which a checkpoint is acquired in the master 60, a predetermined time, an update amount of the commit log is predetermined, or an arbitrary timing determined by the user. In this case, the update execution unit 24 of the master 60 selects the oldest (smallest) commit log ID (or time) among the commit log IDs of each replica 70 with reference to the stored commit log application status table. It is determined that the previous commit log can be deleted. After the determination, the master 60 notifies each replica 70 of the commit log ID, and the replica 70 deletes the commit log ID based on the notification.

  Next, the flow of processing in the log file management system 300 will be described with reference to the flowchart shown in FIG. First, the update request unit 2 receives a record data update request from the client device 1 (S31). Next, based on this update request, the commit log generation unit 3 generates a commit log, and the update execution unit 24 instructs the commit log storage unit 7 to make the commit log persistent to the commit log control unit 15 and makes it permanent. (S32).

  Next, the replication control unit 16 transmits a replication request to each replica 70. The update execution unit 24 of the master 60 receives notification from the replicas 70 that the commit log related to record data has been made permanent (S33). It is assumed that this notification includes the commit log ID, the commit log itself, or the time when the master 60 outputs the commit log so that the master 60 can know which commit log the notification is sent to. The update execution unit 24 of the master 60 creates or updates a commit log application status table based on this notification (S34). The update process execution unit 24 of the master 60 selects the oldest commit log generated from the commit logs of each replica 70 (S35). The checkpoint acquisition unit 6 acquires a checkpoint corresponding to the oldest generated commit log (S36). The update execution unit 24 of the master 60 deletes the commit log before the oldest generated commit log and notifies the replica 70, and the replica 70 deletes the corresponding commit log (S37). This completes the deletion of the commit log that is no longer needed.

  According to the fifth embodiment, the master 60 manages the application status of the commit log that the master 60 and the replica 70 have, and the master 60 notifies the replica 70 of the application status. This eliminates the need for inquiries about the application status, and makes it possible to reduce the load on the network.

DESCRIPTION OF SYMBOLS 1 Client apparatus 2 Request reception part 3 Commit log production | generation part 4 Update execution part 5 Commit log control part 6 Checkpoint acquisition part 7 Record data storage part 8 Commit log storage part 9 Checkpoint storage part 14 Update execution part 15 Commit log control part DESCRIPTION OF SYMBOLS 16 Replication control part 17 Communication control part 24 Update execution part 40 Master 50 Replica 60 Master 70 Replica 100 Log file management apparatus 200 Log file management system 300 Log file management system

Claims (10)

Log file management for creating and managing a shared file in which a commit log indicating the update contents of each record data is generated and stored for a record set that is a set of record data based on the update operation of the record data A device,
A commit log storage unit for storing the shared file;
A request receiving unit that receives an update request for the record data from a client device;
A commit log generating unit that generates the commit log based on the update request and makes the commit log permanent to the shared file;
Record data stored in the record data storage unit is acquired as a checkpoint at an arbitrary timing and made permanent, and for each record set, the time when the checkpoint is acquired is compared, and the oldest checkpoint is acquired. An update execution unit that instructs the previous commit log to be deleted as an unnecessary commit log;
A checkpoint acquisition unit that acquires the record data of the arbitrary timing as a checkpoint from the record data storage unit and stores the checkpoint in a checkpoint storage unit;
A log file management apparatus comprising: a commit log control unit that deletes a commit log before the checkpoint acquisition time from a commit log accumulated in the shared file based on an instruction from the update execution unit. The commit log generation unit accumulates the shared logs in the order in which the commit logs are generated, and when the shared file includes a plurality of files,
When deleting the commit log, the commit log control unit deletes, in file units, a shared file in which the latest commit log of the shared file is older than the acquisition point of the oldest checkpoint. The log file management apparatus according to claim 1. For a record set that is a set of record data shared between nodes, based on the update operation of the record data, create a shared file that generates and accumulates a commit log indicating the update contents of each record data, A log file management system to manage,
The master log file management device that receives the record data update request from the client device and performs the update processing of the record data, and the same record corresponding to the update of the record data of the master log file management device A replica log file management device that replicates and updates data,
Each of the log file management devices is
A commit log storage unit for storing the shared file;
A request receiving unit that receives an update request for the record data from a client device;
A commit log generating unit that generates the commit log based on the update request and makes the commit log permanent to the shared file;
It is determined whether or not the commit log accumulated in the shared file for each record set is within a predetermined range. When the commit log exceeds the predetermined range, the commit log exceeding the predetermined range is An update execution unit that instructs deletion as an unnecessary commit log,
And checkpoint acquisition unit that stores the checkpoint storage unit obtains the record data stored in the record data storage unit as a checkpoint,
A commit log control unit that deletes unnecessary commit logs from the commit log accumulated in the shared file based on an instruction from the update execution unit;
A communication control unit for communicating with other log file management devices,
When its own log file management device is the master log file management device, the log file management device further includes:
In response to the update of record data in the master log file management device, a replication request is issued to each replica log file management device to request that the same record data be copied and updated. From each log file management device, comprising a replication control unit for receiving a response to the effect that persistence of commit log related to record data update based on the replication request is completed,
The log file management system, wherein the master and the replica log file management devices use the same value as the value indicating the predetermined range. For a record set that is a set of record data shared between nodes, based on the update operation of the record data, create a shared file that generates and accumulates a commit log indicating the update contents of each record data, A log file management system to manage,
The master log file management device that receives the record data update request from the client device and performs the update processing of the record data, and the same record corresponding to the update of the record data of the master log file management device A replica log file management device that replicates and updates data,
The log file management device
A commit log storage unit for storing the shared file;
A request receiving unit that receives an update request for the record data from the client device;
A commit log generating unit that generates the commit log based on the update request and makes the commit log permanent to the shared file;
An update execution unit for instructing to delete unnecessary commit logs among the commit logs accumulated in the shared file for each record set;
And checkpoint acquisition unit that stores the checkpoint storage unit obtains the record data stored in the record data storage unit as a checkpoint,
Based on an instruction from the update execution unit, a commit log control unit that deletes a commit log before the oldest checkpoint acquisition time from the commit log accumulated in the shared file;
A communication control unit for communicating with other log file management devices,
When its own log file management device is the master log file management device, the log file management device further includes:
In response to the update of record data in the master log file management device, a replication request is issued to each replica log file management device to request that the same record data be copied and updated. From each log file management device, comprising a replication control unit for receiving a response to the effect that persistence of commit log related to record data update based on the replication request is completed,
The update execution unit of the master log file management device,
Among the commit logs of the replica log file management apparatus, the oldest commit log is selected, and the replica log file management apparatus is notified that the commit log generated before the commit log is deleted as an unnecessary commit log. remove the commit log has its own log file management apparatus with the notification to,
The log file management system, wherein the update execution unit of the replica log file management device deletes the commit log based on the notification.   The log file management device used in the log file management system according to claim 3.   The log file management device used in the log file management system according to claim 4. Log file management for creating and managing a shared file in which a commit log indicating the update contents of each record data is generated and stored for a record set that is a set of record data based on the update operation of the record data A log file management method used for a device,
The log file management device
Receiving a request for updating the record data from a client device by a request receiving unit;
Generating the commit log based on the update request and perpetuating the commit log in a commit log storage unit having the shared file;
The record data stored in the record data storage unit is acquired as a checkpoint at an arbitrary timing and made permanent, and stored in the checkpoint storage unit;
A log comparing the time when the checkpoint was acquired for each record set, and deleting the commit log before the time when the oldest checkpoint was acquired as an unnecessary commit log. File management method. For a record set that is a set of record data shared between nodes, based on the update operation of the record data, create a shared file that generates and accumulates a commit log indicating the update contents of each record data The master log file management device that receives the update request of the record data from the client device and performs the update processing of the record data, and corresponding to the update of the record data of the log file management device of the master, A log file management method for use in a log file management device of a log file management system comprising a replica log file management device that replicates and updates the same record data,
The log file management device
Receiving an update request for the record data from the client device;
Generating the commit log based on the update request and persisting the commit log to the shared file;
Determining whether the commit log accumulated in the shared file for each record set is within a predetermined range;
Deleting the commit log exceeding the predetermined range as an unnecessary commit log when the commit log exceeds the predetermined range;
And storing the checkpoint storage unit obtains the record data stored in the record data storage unit as a checkpoint,
When its own log file management device is the master log file management device, the log file management device further includes:
In response to the update of record data in the master log file management device, a replication request is issued to each replica log file management device to request that the same record data be copied and updated. Receiving from the log file management device a response that the persistence of the commit log related to the record data update based on the replication request is completed,
The log file management method, wherein the master and replica log file management devices use the same value as the value indicating the predetermined range. For a record set that is a set of record data shared between nodes, based on the update operation of the record data, create a shared file that generates and accumulates a commit log indicating the update contents of each record data The master log file management device that receives the update request of the record data from the client device and performs the update processing of the record data, and corresponding to the update of the record data of the log file management device of the master, A log file management method for use in a log file management device of a log file management system comprising a replica log file management device that replicates and updates the same record data,
The log file management device
Receiving an update request for the record data from a client device;
Generating the commit log based on the update request and persisting the commit log to the shared file;
Deleting unnecessary commit logs from among the commit logs accumulated in the shared file for each record set;
And storing the checkpoint storage unit obtains the record data stored in the record data storage unit as a checkpoint,
In response to the update of record data in the master log file management device, a replication request is issued to each replica log file management device to request that the same record data be copied and updated. Receiving from the log file management device a response that the persistence of the commit log related to the record data update based on the replication request is completed;
When its own log file management device is the master log file management device, the log file management device further includes:
Among the commit logs of the replica log file management apparatus, the oldest commit log is selected, and the replica log file management apparatus is notified that the commit log generated before the commit log is deleted as an unnecessary commit log. a step of deleting the commit log has its own log file management apparatus with the notification to,
The commit log control unit of the replica log file management apparatus includes a step of deleting the commit log based on the notification.   A program for causing a computer to execute the log file management method according to any one of claims 7 to 9.

Images