JP6181247B1 - Database management apparatus, database management method, and database management program - Google Patents

Abstract

To provide a database management apparatus, a database management method, and a database management program capable of reducing processing load related to deletion of invalid records. According to one embodiment, a database management apparatus includes a data processing unit, a representative expiration date determination unit, and a deletion unit. The data processing unit writes or reads data to or from the storage device. The representative expiration date determination unit determines a representative expiration date of a data group in which the data is grouped based on the expiration dates assigned to the data handled by the data processing unit. The deletion unit extracts the data group to be deleted based on the representative expiration date determined by the representative expiration date determination unit, and determines the expiration date among the data included in the extracted data group to be deleted. Delete the missing data. [Selection] Figure 1

Description

  Embodiments described herein relate generally to a database management apparatus, a database management method, and a database management program.

  In a system that manages a database, when a record is registered in the database, an expiration date in which the record can be used is set, and a record that has passed the expiration date (invalid record) may be deleted at a predetermined timing. As a technique for deleting invalid records, for example, there is a technique (garbage collection) in which all records in a database are periodically scanned and records that have expired are deleted. However, in the conventional technique, in order to scan all records, the processing load for deleting invalid records may increase.

JP 2010-267028 A

  The problem to be solved by the present invention is to provide a database management device, a database management method, and a database management program that can reduce the processing load related to deletion of invalid records.

  The database management apparatus according to the embodiment includes a data processing unit, a representative expiration date determination unit, and a deletion unit. The data processing unit writes or reads data to or from the storage device. The representative expiration date determination unit determines a representative expiration date of a data group in which the data is grouped based on the expiration dates assigned to the data handled by the data processing unit. The deletion unit extracts the data group to be deleted based on the representative expiration date determined by the representative expiration date determination unit, and determines the expiration date among the data included in the extracted data group to be deleted. Delete the missing data.

The figure which shows the function structural example of the database management system 1 provided with the database management apparatus 100 of this embodiment. The figure which shows an example of the content of the record management information 150A. The figure which shows an example of the content of the 1st index 150B. The figure which shows an example of the content of the 2nd index 150C. The figure for demonstrating the operation example at the time of the record access in the database management apparatus 100. FIG. The figure for demonstrating the operation example of the garbage collection process in the database management apparatus. The figure which shows the 1st Example which sets a representative TTL value. The figure which shows the 2nd Example which sets a representative TTL value. The figure which shows the 3rd Example which sets a representative TTL value. The figure for demonstrating the example of extraction of the deletion object page. The figure which shows the state of each page after deletion process execution, and the update example of an index. The figure which shows the production | generation example of an index when a record is added to the page. 5 is a flowchart showing an example of a record writing process (new registration) in the database management apparatus 100. The flowchart which shows an example of an index production | generation process. 5 is a flowchart showing an example of record writing processing (update) in the database management apparatus 100. 5 is a flowchart showing an example of a record reading process in the database management apparatus 100. 5 is a flowchart showing an example of a record deletion process in the database management apparatus 100. 10 is a flowchart showing an example of invalid record deletion processing in the database management apparatus 100.

  Hereinafter, a database management device, a database management method, and a database management program according to embodiments will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a functional configuration example of a database management system 1 including a database management apparatus 100 according to the present embodiment. The database management system 1 illustrated in FIG. 1 includes a database management device 100, a storage device 200, and a client 300. Each of the database management device 100, the storage device 200, and the client 300 communicates via a communication network including the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), and the like.

  In the database management system 1 shown in FIG. 1, one or both of the storage device 200 and the client 300 may be integrated with the database management device 100. In the example of FIG. 1, one client 300 is shown as an apparatus that uses the database management apparatus 100, but a plurality of clients 300 may use one database management apparatus 100.

  The database management apparatus 100 accepts input of setting information from the client 300 used by the user and an instruction for executing processing, and issues an I / O command to the storage device 200 based on the accepted content. . Further, the database management apparatus 100 outputs information such as processing results to the client 300.

  The database management apparatus 100 includes a record management unit (data processing unit) 110, an index management unit (representative expiration date determination unit) 120, an invalid record deletion unit (deletion unit) 130, a communication unit 140, and a storage unit 150. Is provided. In addition, among these constituent elements, except for the storage unit 150, for example, a processor such as a CPU (Central Processing Unit) in a computer executes a program (software) stored in a program memory such as the storage unit 150. It is realized with. Some or all of these functional units may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). It may be realized by cooperation with hardware. The storage unit 150 is realized by a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), a flash memory, or the like. The storage unit 150 stores information such as the record management information 150A, the first index 150B, and the second index 150C.

  The record management unit 110 performs processing for writing data instructed to be written from the client 300 to the storage device 200, reading data instructed to be read from the storage device 200, and deleting data instructed to be deleted from the storage device 200. . In the present embodiment, one unit of data that the client 300 instructs to write, read, or delete is referred to as a record. The record includes, for example, identification information (record ID) for identifying the record and a data main body portion associated therewith. The record may be fixed length or variable length.

  The record management unit 110 manages records in a logical hierarchical structure represented by, for example, a page including one or more records and a table including one or more pages. This hierarchical structure is grasped by the record management information 150A. Here, the page indicates a virtual storage area having a fixed-length storage area to which one or a plurality of records can be allocated. The table indicates a virtual storage area having a fixed-length storage area to which one or a plurality of pages can be allocated. A table or page is an example of a data group obtained by grouping records. In the following description, it is assumed that the page is a data group.

  FIG. 2 is a diagram illustrating an example of the contents of the record management information 150A. The record management information 150A is, for example, information in which “page ID”, “record ID”, and “TTL (Time To Live) value” are associated with each other for each table. “Page ID” is identification information for identifying a page. When writing a record to the storage device 200, the record management unit 110 sets a TTL value for each record and adds it to the record management information 150A. The “TTL value” is an example of information that can recognize the expiration date of the record. The TTL value is time information indicating, for example, how long the record is valid. The TTL value may be specified by the client 300 or may be set in the database management apparatus 100.

  As the TTL value shown in FIG. 2, a time type value (for example, date information) is set. In addition, for the TTL value, for example, a valid period from the access time to the record (for example, information such as 24 hours from the last access time) may be set.

  Note that the record management unit 110 may include a conversion unit that converts a logical address corresponding to the record ID stored in the record management information into a physical address of the storage device 200. In the present embodiment, a record may be stored in the storage device 200 in a state where the TTL value is not set. In this case, the record management unit 110 may register the TTL value of the record management information 150A with a blank or the like, and may not register information regarding the record in the record management information 150A.

  The index management unit 120 generates an index (index information) for managing each record when the record management unit 110 writes a record to the storage device 200 or deletes the record. Further, the index management unit 120 determines the representative expiration date (representative TTL value) of the data group in which the data is grouped based on the TTL value assigned to each record, and stores the determined information in the index. . The index management unit 120 also updates the index when the invalid record deletion unit 130 deletes an invalid record in the storage device 200. In the present embodiment, the index includes a first index 150B and a second index 150C.

  FIG. 3 is a diagram illustrating an example of the contents of the first index 150B. The first index 150B is information in which “representative TTL value” and “page ID” are associated with each other, for example. The “representative TTL value” is a representative expiration date for each page managed by the record management information 150A. The representative expiration date is expiration date information determined based on a TTL value assigned to a record stored in each page, and is used for a primary search when an invalid record is deleted. A method for setting the representative TTL value will be described later. The data “representative TTL value” is expressed, for example, in the same data format as the TTL value for each record.

  The first index 150B is configured such that, for example, a page that has passed the representative expiration date can be searched using the representative TTL value as a search condition (key). The first index 150B is an index that is used, for example, by the invalid record deletion unit 130 to realize garbage collection processing (also referred to as compaction, vacuum, or page reorganization) in units of pages.

  FIG. 4 is a diagram illustrating an example of the contents of the second index 150C. The second index 150C is information in which “page ID”, “representative TTL value”, and “additional information” are associated with each other, for example. “Additional information” means, for example, past representative TTL values, maximum, minimum, average, number of records, total record length of each record in all records in the page, rules for obtaining representative TTL values, etc. Information including at least one of them. The “additional information” is information used to recalculate the representative TTL value and update the first index 150B in the index management unit 120, for example.

  The index management unit 120 may manage the first index 150B and the second index 150C described above as one index, or may generate an index including other information.

  The invalid record deletion unit 130 extracts pages to be deleted based on the first index 150B, and deletes records (invalid records) that have passed the expiration date from among the records included in the extracted pages to be deleted. The deletion of the invalid record is, for example, deleting the invalid record stored in the data storage area 210 of the storage device 200 and deleting the management information of the corresponding invalid record from the record management information 150A.

  The invalid record deletion unit 130 may periodically delete the invalid record as a garbage collection process, for example, when the record management information 150A or the data amount of the storage device 200 exceeds a threshold value. Alternatively, it may be performed according to an instruction from the client 300.

  The communication unit 140 is a communication interface that transmits / receives data to / from at least one of the other database management device 100, the storage device 200, and the client 300 via a communication network.

  In addition to the information shown in FIG. 1, the storage unit 150 stores various setting information in the database management apparatus 100, information obtained from the execution progress or execution results of various processes related to database management, log information, error information, and the like.

  The storage device 200 is a storage device managed by the database management device 100, for example. The storage device 200 includes, for example, a hard disk (disk device), a flash memory, and a control circuit. The storage device 200 includes a data storage area 210. The data storage area 210 holds records in a nonvolatile manner.

  The client 300 transmits an instruction to the database management apparatus 100 and acquires a response. For example, the client 300 outputs a command for causing the database management apparatus 100 to read / write / delete data. The client 300 is an information processing apparatus such as a PC (Personal Computer), a tablet terminal, or a mobile phone. The client 300 may be an information processing apparatus used by a user of the database management apparatus 100, and is an apparatus that transmits various commands to the database management apparatus 100 based on commands received from another apparatus. Also good. Further, the client 300 may be a device that generates various commands based on the result of information processing inside and transmits the commands to the database management device 100.

  The client 300 transmits to the database management apparatus 100 a write command for instructing data writing, a read command for instructing data reading, a deletion command for instructing data deletion, and the like. In addition, the client 300 acquires a processing result for the content instructed from the database management apparatus 100.

  Hereinafter, various operations of database management in the database management apparatus 100 will be described. FIG. 5 is a diagram for explaining an operation example at the time of record access in the database management apparatus 100. In this embodiment, record access refers to record write processing. The record writing process includes both new registration and update of records.

  In FIG. 5, the client 300 outputs a command (write command) corresponding to the record write instruction to the database management apparatus 100 (1). The record management unit 110 registers information related to the record to be written in the record management information 150A (2). In addition, the record management unit 110 outputs a record write command instructed to write to the data storage area 210 of the storage device 200 (3). Note that the order of the above (2) and (3) may be reversed or simultaneous.

  Also, the record management unit 110 outputs information related to the written record and an index registration instruction (or update instruction) to the index management unit 120 (4). The information related to the record is a record ID associated with the written record, a page ID to which the written record belongs, and a TTL value. For example, the written record belongs to the management information stored in the record management information 150A. The management information of all records associated with the page ID may be used.

  The index management unit 120 generates (newly creates or updates) the first index 150B and the second index 150C based on the information obtained by the record management unit 110 (5) and (6).

  Further, when the generation of the first index 150B and the second index 150C for the written record is completed, the index management unit 120 outputs information indicating that the data writing process is completed to the client 300 (7).

  Next, an operation at the time of executing the garbage collection process in the database management apparatus 100 will be described with reference to the drawings. FIG. 6 is a diagram for explaining an operation example of the garbage collection processing in the database management apparatus 100.

  The invalid record deletion unit 130 periodically performs invalid record deletion processing by garbage collection processing. The invalid record deletion unit 130 refers to the first index 150B at the timing of executing the deletion process (8), and compares the representative TTL value set for each page ID with the current time. As a result of the comparison, when the current time exceeds the representative TTL value, the invalid record deletion unit 130 extracts the page ID having the representative TTL value as a target page from which the invalid record is deleted.

  Note that the invalid record deletion unit 130 may limit the maximum number of pages to be deleted in one deletion process in order to adjust the deletion process time. In this case, the invalid record deletion unit 130 compares the current time with the representative TTL value, and extracts a number of page IDs that do not exceed the maximum number in the order of pages in which the elapsed time from the representative TTL value is large.

  Next, the invalid record deletion unit 130 refers to the record management information 150A, compares the TTL value of each record stored in the page to be deleted with the current time, and sets a TTL value before the current time. The management information of the recorded record is deleted (9). Further, the invalid record deletion unit 130 performs the above-described deletion processing on the record management information 150A and deletes the target record stored in the storage device 200 (10). In (9) and (10) above, when a record to be deleted is extracted with reference to the record management information 150A, the record of the record management information 150A is deleted after deleting the target record stored in the storage device 200. The management information may be deleted and vice versa.

  Next, the invalid record deletion unit 130 outputs an instruction to update the first index 150B and the second index 150C to the index management unit 120 (11). The invalid record deletion unit 130 outputs, to the index management unit 120, information indicating which record of which page ID has been deleted, as well as instructions for updating the first index 150B and the second index 150C.

  The index management unit 120 refers to the information stored in the second index 150C, and updates the first index 150B using the representative TTL value and additional information before deletion for the page ID to be deleted (12). Further, the index management unit 120 updates the additional information based on the information used when generating the first index 150B, and generates the second index 150C including the updated additional information (13).

  By deleting the invalid record by the garbage collection process by the above-described process, it is not necessary to scan all the records of the record management information 150A, but roughly determines the necessity of deletion in units of pages, and the invalid record is in the page. If not so many, the physical deletion of invalid records can be delayed. As a result, the processing load related to deletion of invalid records can be reduced.

  Further, by managing the index information for each page using the first index 150B and the second index 150C, the data amount of the index can be reduced.

Next, the setting of the representative TTL value described above will be specifically described.
(First embodiment for setting a representative TTL value)
FIG. 7 is a diagram showing a first embodiment for setting a representative TTL value. In the first embodiment, a representative TTL value is set for each page based on a value obtained by performing statistical processing on some or all of the TTL values assigned to each data in the page. For example, in the first embodiment, the representative TTL value T (P) is set on the basis of the ratio to the TTL value assigned to each record belonging to the same page. In FIG. 7, the horizontal axis represents time, Tmin indicates the minimum value of the TTL value of each record included in the same page, and Tmax indicates the maximum value of the TTL value of each record included in the same page.

  In the first embodiment, the threshold value a for the ratio is set to 0 ≦ a ≦ 1. The record management unit 110 sets the representative TTL value T (P) so that when the ratio of invalid records exceeds a among all records in each page, it is extracted as a page to be deleted. This threshold value may be specified by the client 300 or may be set in the index management unit 120 in advance.

  For example, when the above-described threshold value a = 0.3, as shown in FIG. 7, the invalid record deletion process is performed for a page in which the number of invalid records included in the page does not exceed 30% of the total number of records. For pages that are not executed and the number of invalid records exceeds 30% of the total number of records, the representative TTL value T (P) is set so that invalid record deletion processing is executed. In this case, for example, the index management unit 120 sorts all the records in the page by the TTL value, and counts the TTL of the first record that exceeds 30% of the total number of records when counting from the smaller TTL value (earlier time). The value is set to the representative TTL value T (P).

  Further, the index management unit 120 calculates the difference between Tmax and Tmin in the page, and the ratio of the difference value of 30% and the number of records C1 less than the representative TTL value and the number of records C2 greater than or equal to the representative TTL value. A representative TTL value T (P) may be calculated by adding a value obtained by multiplying to Tmin (T (P) = 0.3 × (C2 / C1) × (Tmax−Tmin) + Tmin). .

  When the threshold value a = 0, the representative TTL value T (P) matches Tmin shown in FIG. Therefore, the cycle for executing the deletion of invalid records by the garbage collection process becomes fast and the garbage collection process becomes strict, but the processing load in the deletion process increases.

  When the threshold value a = 1, the representative TTL value T (P) matches Tmax shown in FIG. In this case, since the deletion operation can be delayed until all the records in the page become invalid records, the load due to the garbage collection process can be reduced to the maximum, but the invalid records remain in the page until then.

  Here, an example of setting the representative TTL value T (P) when up to three records are written for each record in the first embodiment will be described. For example, in the first embodiment, when the first record R1 is written in a certain page, the index management unit 120 sets Tmin = 0 and Tmax is the TTL value (T (R1)) of the record R1. Further, the index management unit 120 sets the representative TTL value T (P) = T (R1).

  Next, when the second record R2 is written on the same page, the index management unit 120 sets Tmin to the earliest of the TTL values of the records R1 and R2 (Tmin = min (T (R1), In addition, the index management unit 120 sets Tmax to the latest of the TTL values of the records R1 and R2 (Tmax = max (T (R1), T (R2))).

  Further, the index management unit 120 sets the number of records C1 less than the representative TTL value to 1, and sets the number of records C2 greater than or equal to the representative TTL value to 1, and calculates the above-described formula “T (P) = a × (C2 / C1) × The representative TTL value T (P) is calculated from (Tmax−Tmin) + Tmin ”. Further, the index management unit 120 stores the information used at the time of writing the record R2 described above in the second index 150C as additional information. The additional information at this time is Tmin, Tmax, C1, and C2. As a result, when updating the representative TTL value T (P) in the page, the index management unit 120 quickly acquires Tmin, Tmax, C1, C2, etc. without scanning all the records in the target page. can do.

  Furthermore, when writing the third record R3 on the same page, the index management unit 120 compares the TTL value T (R3) of the record R3 with the Tmin and Tmax acquired from the second index 150C, and creates a new Tmin. And Tmax are determined, and one of C1 and C2 is updated. Further, the index management unit 120 recalculates the representative TTL value T (P) of the page in which the record R3 is registered using the above-described calculation formula. Thereafter, the same processing is performed every time a record is written.

  Thus, in the first embodiment, the representative TTL value T (P) can be set based on the ratio (threshold value a) of the TTL values of all records in the page. Therefore, for example, by dynamically changing the threshold value a in accordance with the free capacity of the storage device 200 and the like, the garbage collection process can be performed at an appropriate timing, so that the processing load can be reduced.

(Second embodiment for setting a representative TTL value)
Next, a second embodiment for setting the representative TTL value will be described. FIG. 8 is a diagram showing a second embodiment for setting a representative TTL value. In the second embodiment, the maximum value and the minimum value are acquired from the TTL values assigned to the respective records belonging to the same page, and based on the ratio to one or both of the acquired maximum value and minimum value, the representative value is obtained. Set the TTL value. For example, as shown in FIG. 8, the index management unit 120 sets the threshold value a indicating the ratio to 0 ≦ a ≦ 1, and internally divides the interval from the minimum value of the TTL value to the maximum value with “a: 1−a”. The set time is set as the representative TTL value T (P).

  Here, an example of setting the representative TTL value T (P) when up to three records are written for each record in the second embodiment will be described. For example, in the first embodiment, when the first record R1 is written in a certain page, the index management unit 120 sets Tmin = 0 and Tmax is the TTL value (T (R1)) of the record R1. Further, the index management unit 120 sets the representative TTL value T (P) = T (R1).

  Next, when the second record R2 is written on the same page, the index management unit 120 sets Tmin to the earliest of the TTL values of the records R1 and R2 (Tmin = min (T (R1), In addition, the index management unit 120 sets Tmax to the latest of the TTL values of the records R1 and R2 (Tmax = max (T (R1), T (R2))).

  In the second embodiment, the index management unit 120 calculates the difference between Tmax and Tmin in the page, and adds a value obtained by multiplying the difference value by the threshold value a to the Tmin, thereby representing the representative TTL value T (P) is calculated (T (P) = a × (Tmax−Tmin) + Tmin). Further, the index management unit 120 stores the information used at the time of writing the record R2 as additional information D in the second index 150C. The additional information at this time is Tmin and Tmax.

  Furthermore, when writing the third record R3 on the same page, the index management unit 120 compares the TTL value T (R3) of the record R3 with the Tmin and Tmax acquired from the second index 150C, and creates a new Tmin. And Tmax are determined. Further, the index management unit 120 recalculates the representative TTL value T (P) of the page in which the record R3 is registered using the above-described calculation formula. Thereafter, the same processing is performed every time a record is written. Note that the index management unit 120 may arbitrarily set one of the above-described minimum or maximum TTL values in the calculation of the representative TTL value T (P).

  As described above, in the second embodiment, the representative TTL value T (P) is set on the basis of one or both of the maximum value and the minimum value of the TTL values of all the records in the page. Invalid records can be deleted at an appropriate time. Therefore, since the garbage collection process can be performed at an appropriate timing, the processing load can be reduced.

(Third embodiment for setting a representative TTL value)
Next, a third embodiment for setting the representative TTL value will be described. FIG. 9 is a diagram showing a third embodiment for setting a representative TTL value. In the third embodiment, the representative TTL value T (P) is set based on the size (record length) of each record belonging to the same page. For example, the index management unit 120 calculates the sum S (P) of the product of the record length of each record in the page and the TTL value, and the sum L (P) of the record length of each record, and represents the representative TTL value T ( P) is set as S (P) / L (P). Further, the additional information D (P) in the third embodiment is L (P).

  For example, in the third embodiment, when adding a record having a record length L (R) and a TTL value T (R), the index management unit 120 sets a new page representative expiration date T (P ′) to “T ( P ′) = S (P ′) / L (P ′) = (T (P) × D (P) + T (R) × L (R)) / (D (P) + L (R)) ” To do. Since the additional information D (P) here is equivalent to L (P), it holds the total record length of all the records in the page. In the above calculation formula, when the first record is added, it is calculated as D (P) = 0.

  Here, an example of setting the representative TTL value T (P) when up to three records are written for each record in the third embodiment will be described. In the first embodiment, when the first record R1 (record length L (R1)) is written on a page, the index management unit 120 sets the representative TTL value T (P) to the TTL value T (R1) of the record R1. ). Further, the index management unit 120 stores the information used at the time of writing the record R1 described above in the second index 150C as additional information D (P). The additional information D (P) at this time is L (R1).

  Next, when the second record R2 (record length L (R2) (for example, L (R2)> L (R1))) is written on the same page, the index management unit 120 displays the representative TTL value T ( P), using the total value D (P) of the record lengths added so far, the TTL value T (R2) of the record R2, and the record length L (R2), the calculated value T (P ′) represents the representative TTL value T (P) is reset (T (P ′) = (T (P) × D (P) + T (R2) × L (R2)) / (D (P) + L (R2))). Note that the additional information D (P) is acquired from, for example, the second index 150C. Further, the additional information D (P) after writing the record R2 is set as “D (P) = D (P) + L (R2)”. That is, as shown in FIG. 9, the reset page expiration date T (P ′) is set as a value that internally divides T (P) and T (R2) into L (R2): L (P). The

  Furthermore, when a new record R3 (record length L (R3)) is written on the same page, the index management unit 120 displays the representative TTL value T (P), the total sum D (P) of the record lengths added so far, Using the TTL value T (R3) of R3 and the record length L (R3), T (P ′) is calculated in the same manner by the above-described formula, and the representative TTL value T (P) is reset with the calculated value. To do. Thereafter, the same processing is performed every time a record is written.

  In the third embodiment, for example, calculation may be performed using a coefficient (weight) that can be specified by the user or the like. For example, the index management unit 120 may set the coefficient β, add the set count β to the TTL value T (R) of the record, and calculate the representative TTL value T (P) (β × T (R) )). In this case, the index management unit 120 sets the new page representative expiration date T (P ′) to “T (P ′) = (T (P) × D (P) + β × T (R) × L (R)) / Calculate as (D (P) + L (R)) ".

  Thereby, the index management part 120 can give priority to either value among a record size and an expiration date. For example, in the above-described example, when the coefficient β is β> 1, the expiration date can be prioritized, and when β <1, the record length can be prioritized.

  As described above, in the third embodiment, by setting the representative TTL value T (P) based on the record size of each record in the page, for example, a page with many records having a large record size is given priority. A garbage collection process can be performed.

  In addition, the setting example of the representative TTL value in the present embodiment is not limited to the first to third examples described above, and for example, all or a part of each example is combined with another example. May be. The index management unit 120 may set the TTL value of a randomly selected record among the records belonging to the same page as the representative TTL value. Further, the index management unit 120 may dynamically change each embodiment according to the instruction from the client 300, the type of data, the usage, and the like. In each of the above-described embodiments, the index management unit 120 sets the representative TTL value, but the record management unit 110 may perform this instead.

  Next, a specific example of deleting invalid records in the garbage collection process will be described with reference to the drawings. FIG. 10 is a diagram for explaining an example of extracting a deletion target page. FIG. 10A shows an example of a record stored in the storage device 200 managed by the record management information 150A. In the example of FIG. 10A, records stored in each page (ID (P)) are shown. FIG. 10B shows the first index 150B, and FIG. 10C shows the second index 150C.

  In the following example, for convenience, a record example included in each of page IDs (P) = 1 to 3 among a plurality of pages in the record management information 150A is shown, but the number of records is limited to this. Is not to be done. For setting the representative TTL value, the setting method in the first embodiment described above is used, and the threshold a for the ratio of invalid records is set to 0.3. Also, the current time will be described as “1/20 22:30”.

  In the example of FIG. 10A, Record 1-1 to Record 1-6 are shown with page ID (P) = 1, but the number of records is not limited to this. Here, it is assumed that Records 1-1 to Records 1-4 are invalid records, and Records 1-5 and Records 1-6 are records that have not expired (valid records).

  Assume that the index management unit 120 sets the TTL value set in Record 1-5 to the representative TTL value T (P) by the setting method in the first embodiment. In this case, the index management unit 120 outputs the TTL value set in Record 1-5 to the first index 150B in association with the page ID (ID (P)). Also, the index management unit 120 associates the above-described representative TTL value T (P), page ID (ID (P)), and additional information D (P) including information used when setting the representative TTL value. To the second index 150C.

  Further, in the page ID (P) = 2, it is assumed that Records 2-1 to 2-3 are invalid records, and Records 2-4 to 2-6 are valid records. Further, it is assumed that the index management unit 120 sets the TTL value set in Record 2-3 to the representative TTL value T (P) by the setting method in the first embodiment. In this case, the index management unit 120 outputs the TTL value set in Record 2-3 to the first index 150B in association with the page ID (ID (P)). Also, the index management unit 120 associates the above-described representative TTL value T (P), page ID (ID (P)), and additional information D (P) including information used when setting the representative TTL value. To the second index 150C.

  In addition, in the page ID (P) = 3, it is assumed that Records 3-1 to 3-5 are invalid records and Record 3-6 is a valid record. Further, it is assumed that the index management unit 120 sets the TTL value set in Record 3-5 to the representative TTL value T (P) by the setting method in the first embodiment. In this case, the index management unit 120 outputs the TTL value set in Record 3-5 to the first index 150B in association with the page ID (ID (P)). Also, the index management unit 120 associates the above-described representative TTL value T (P), page ID (ID (P)), and additional information D (P) including information used when setting the representative TTL value. To the second index 150C.

  In the first index 150B, the representative TTL value (T (P)) representing each page described above and the page ID (ID (P)) are stored in association with each other. The first index 150B is sorted in ascending order, for example, for each representative TTL value.

  The second index 150C stores the representative TTL value T (P) and the additional information D (P) in association with each other on the basis of the page ID (ID (P)). The contents of the additional information A1 to A3 are, for example, Tmin, Tmax, C1, and C2 described above, but are not limited thereto. The second index 150C is sorted in ascending order, for example, for each page ID (P).

  The invalid record deletion unit 130 extracts a page to be deleted using the first index 150B described above, and deletes a record that has expired and exists in the extracted page. For example, in the case of the example of FIG. 10B, page IDs having a representative TTL value before the current time “1/20 22:30” are ID (P) = 2 and 3. Therefore, the invalid record deletion unit 130 deletes invalid records from the records included in ID (P) = 2 and 3.

  FIG. 11 is a diagram illustrating a state of each page after execution of the deletion process and an index update example. In the example of FIG. 11, in the example of FIG. 10 described above, the state of each page in which the invalid record is deleted for ID (P) = 2 and 3, and the first index 150B and the second index after the invalid record is deleted. An example in which the index 150C is regenerated is shown.

  In FIG. 11A, Record2-4 to Record2-6 exist as valid records at ID (P) = 2. In this case, the index management unit 120 sets the TTL value set in Record 2-5 as the representative TTL value T (P) by the setting method shown in the first embodiment described above, and the page ID (ID (P) ) And the first index 150B. Further, the index management unit 120 outputs the above-described representative TTL value T (P), page ID (ID (P)), and additional information D (P) to the second index 150C.

  In addition, Record 3-6 exists as a valid record at ID (P) = 3. In this case, since there is only one record in the page, the index management unit 120 uses the TTL value set in Record 3-6 as the representative TTL value T (P) and the page ID (ID (P)). 1 index is output to 150B. Also, the index management unit 120 outputs the above-described representative TTL value, page ID, and additional information D (P) to the second index 150C.

  The invalid record deletion unit 130 extracts a page from which invalid records are deleted at a predetermined timing based on the newly generated first index 150B and second index 150C, and records an expired record with respect to the extracted page. Perform deletion.

  FIG. 12 is a diagram illustrating an example of index generation when a record is added to a page. FIG. 12 shows an example in which one record (Record 1-7) is newly added to page ID (P) = 1 among the pages shown in FIG.

  In this case, the index management unit 120 performs recalculation of the representative TTL value for the page ID (P) = 1 to which the record is added, and based on the calculated representative TTL value, the first index 150B and the second index TTL value are calculated. The index 150C is updated. Since ID (P) = 2 and 3 have the same contents as in FIG. 10, the target data included in the first index 150B and the second index 150C is not updated, and ID (P) = 1. Updates are made to the corresponding data.

  For example, it is assumed that the index management unit 120 sets the TTL value of the newly added record (Record 1-7) as the representative TTL value by the setting method of the first embodiment described above. In this case, the index management unit 120 updates the management information for ID (P) = 1 in the first index 150B and the second index 150C. Accordingly, the invalid record deletion unit 130 can extract a page from which the invalid record is deleted using the updated first index 150B, and can delete the invalid record included in the extracted page. In the example of FIG. 12, if the current time is “1/20 22:30”, the pages to be deleted are ID (P) = 1-3.

  Next, various processing contents of the database management apparatus 100 will be specifically described with reference to flowcharts. FIG. 13 is a flowchart illustrating an example of a record writing process (new registration) in the database management apparatus 100. In the example of FIG. 13, the record management unit 110 receives a record write instruction from the client 300 (step S100).

  Next, the record management unit 110 writes the management information regarding the record in the record management information 150A stored in the storage unit 150 (step S102). In the process of step S102, the record management unit 110 writes the table and page information storing the record in the record management information 150A. As described above, the management information includes a page ID, a record ID, and a TTL value as necessary.

  Next, the record management unit 110 writes a record in the storage device 200 (step S104). In step S104, the record management unit 110 writes a record in the area of the data storage area 210 allocated corresponding to the page ID and record ID of the record management information 150A. Note that the record management unit 110 issues a command for writing a record to the storage device 200. The processes in steps S102 and S104 described above may be performed in the reverse order or simultaneously.

  Next, the record management unit 110 determines whether the record has been successfully written (step S106). When the record is successfully written, the record management unit 110 determines whether a TTL value is set for the written record (step S108). When the TTL value is set, the record management unit 110 notifies the index management unit 120 of the page ID assigned to the written record, the record ID, and the TTL value together with the index generation instruction. The index generation process is executed (step S110). Details of the index generation process will be described later.

  When the TTL value is not set in the process of step S108, or after the process of step S110, the record management unit 110 outputs information indicating that the record has been successfully written to the client 300 (step S112). If the record writing is not successful in the process of step S106 described above, the record management unit 110 outputs error information corresponding to the cause of the writing failure to the client 300 (step S114). As a cause of the writing failure, for example, there is a case where the free capacity of the storage device 200 is insufficient, but it is not limited to this.

  FIG. 14 is a flowchart illustrating an example of the index generation process. In the example of FIG. 14, the index management unit 120 accepts an index generation instruction (step S200). The index management unit 120 determines whether or not the page ID received together with the index generation instruction exists in the first index 150B stored in the storage unit 150 (step S202).

  When the page ID exists in the first index 150B, the index corresponding to the target page of the first index 150B is updated (step S204). In the process of step S204, the index may be updated using the additional information of the second index 150C stored in the storage unit 150. Further, the second index 150C may be updated after the first index 150B is updated.

  If the page ID does not exist in the first index 150B, an index for the corresponding page ID is newly created in the first index 150B (step S206). The created first index 150B is stored in the storage unit 150. In the process of step S206, the second index 150C may be newly created after the first index 150B is newly created. In this case, the created second index 150C is also stored in the storage unit 150.

  Next, an example of record write processing (update) in the database management apparatus 100 will be described using a flowchart. FIG. 15 is a flowchart illustrating an example of record write processing (update) in the database management apparatus 100. In the following description, a write instruction related to update will be described as an update instruction in order to distinguish from the above-described write process (new registration).

  In the example of FIG. 15, when the record management unit 110 receives a record update instruction (step S300), the record management unit 110 extracts the management information of the target record from the record management information 150A stored in the storage unit 150, and extracts the extracted management information. Is updated (step S302). In the process of step S302, for example, when the TTL value is updated, the TTL value in the record is updated. Further, the record management unit 110 updates the record stored in the storage device 200. (Step S304). Note that the processes in steps S302 and S304 described above may be performed in the reverse order, or may be performed simultaneously.

  Next, the record management unit 110 determines whether or not the record update is successful (step S306). When the record update is successful, the record management unit 110 determines whether a TTL value has been set for the updated record (step S308). When the TTL value is set, the record management unit 110 outputs the page ID, record ID, and TTL value to which the record belongs to the index management unit 120, and performs the above-described index generation process (step S310).

  In the process of step S308, when the TTL value is not set in the updated record or after the process of step S310, the record management unit 110 outputs information indicating that the record update is successful to the client 300 (step S308). S312). If the deletion is not successful in the process of step S306 described above, the record management unit 110 outputs error information corresponding to the cause of the deletion failure to the client 300 (step S314). As a cause of the update failure, for example, the storage device 200 may be inaccessible and cannot be accessed, but is not limited thereto.

  Next, a record reading process in the database management apparatus 100 will be described using a flowchart. FIG. 16 is a flowchart illustrating an example of a record reading process in the database management apparatus 100.

  In the example of FIG. 16, when the record management unit 110 receives a record read instruction from the client 300 (step S400), the record management unit 110 refers to the record management information 150A stored in the storage unit 150 and acquires the management information of the target record. (Step S402).

  Next, the record management unit 110 determines whether the management information of the target record has been successfully acquired (step S404). When the management information is successfully acquired, the record management unit 110 acquires the TTL value of the target record (step S406), and determines whether the current time exceeds the TTL value (step S408).

  If the current time exceeds the TTL value, the record has passed the expiration date, and the record management unit 110 outputs error information indicating that no record exists to the client 300 (step S410). If the current time does not exceed the TTL value, the record management unit 110 acquires the target record from the storage device 200 and outputs the acquired target record to the client 300 (step S412).

  If the management information is not successfully acquired in the process of step S404, error information based on the cause of the failure is output to the client 300 (step S414). For example, the management information of the target record may not exist in the record management information 150A as a cause of the failure in acquiring the management information, but is not limited thereto.

  Next, an example of record deletion processing in the database management apparatus 100 will be described using a flowchart. FIG. 17 is a flowchart illustrating an example of a record deletion process in the database management apparatus 100. Note that the example of FIG. 17 is a process of deleting a corresponding record based on a deletion instruction from the client 300 instead of a periodic record deletion in the garbage collection process, for example. In this processing, the record to be deleted may be a valid record or an invalid record.

  In the example of FIG. 17, when receiving a record deletion instruction (step S500), the record management unit 110 refers to the record management information 150A stored in the storage unit 150 and deletes the management information of the target record (step S502). ). In addition, the record management unit 110 deletes the target record from the storage device 200 (step S504). Note that the above-described processing of step S502 and step S504 may be performed in the reverse order, or may be performed simultaneously.

  Next, the record management unit 110 determines whether or not the record has been successfully deleted (step S506). If the record is successfully deleted, the record management unit 110 determines whether a TTL value has been set for the deleted record (step S508). If the TTL value has been set, the page ID to which the record belongs, the record ID, and the TTL value are output to the index management unit 120, and the above-described index generation processing is performed (step S510).

  If the TTL value is not set in the process of step S508, or after the process of step S510, the record management unit 110 outputs information indicating that the record has been successfully deleted to the client 300 (step S512). If the deletion is not successful in the process of step S506 described above, the record management unit 110 outputs error information corresponding to the cause of the deletion failure to the client 300 (step S514). The cause of the deletion failure may be, for example, that the target record has already been deleted, but is not limited to this.

  Next, the invalid record deletion process in the database management apparatus 100 will be described using a flowchart. FIG. 18 is a flowchart illustrating an example of invalid record deletion processing in the database management apparatus 100. FIG. 18 shows an example of invalid record deletion processing that is performed periodically, such as garbage collection processing.

  In the example of FIG. 18, the invalid record deletion unit 130 determines whether an entry exists in the first index 150B (step S600). The entry is information including a set of “representative TTL value” and “page ID” included in the first index 150B. If there is an entry in the first index 150B, the invalid record deletion unit 130 acquires one entry from the first index 150B (step S602), and acquires a representative TTL value of the acquired entry (step S604). .

  Next, the invalid record deletion unit 130 determines whether or not the current time exceeds the representative TTL value (step S606). When the current time exceeds the representative TTL value, the invalid record deletion unit 130 refers to the record management information 150A, scans all records belonging to the page ID corresponding to the entry, and deletes the invalid record (step S608). ). Further, the invalid record deletion unit 130 deletes the target record stored in the storage device 200 (step S610). Note that the above-described processing of step S608 and step S610 may be performed in the reverse order, or may be performed simultaneously.

  Next, the invalid record deletion unit 130 outputs the page ID to which the deleted record belongs, the record ID of the record belonging to the page ID, and the TTL value to the index management unit 120. Is updated (step S612). In the process of step S612, for example, the first index 150B and the second index 150C are updated as in the index generation process described above.

  In the process of step S606, if the current time does not exceed the representative TTL value, or after the process of step S612, the invalid record deletion unit 130 has processed all entries in the first index 150B. Is determined (step S614).

  If all the entries have not been processed, the invalid record deleting unit 130 returns to the process of step S602, acquires one other entry that has not been processed, and performs the subsequent processes. Further, the invalid record deletion unit 130, when there is no entry in the first index 150B, or when processing is performed for all entries in the first index 150B in the process of step S614, The process ends. Also, in step S600, the process of this flowchart is also terminated when there is no entry in the first index 150B. In the example of FIG. 18, when the maximum number of records that can be deleted by one deletion process is set, the deletion process of invalid records may be terminated when the maximum number of records has been deleted. Good. As a result, the invalid record deletion unit 130 can adjust the time required for the invalid record deletion process.

  According to at least one embodiment described above, based on the record management unit 110 that writes or reads records to / from the storage device 200, and the expiration dates assigned to the records handled by the record management unit 110, respectively. The index management unit 120 that determines the representative expiration date of the data group in which the records are grouped, and the deletion-target data group is extracted based on the representative expiration date determined by the index management unit 120, and the deletion target By having the invalid record deletion unit 130 that deletes the data that has passed the expiration date among the data included in the data group, the processing load related to the deletion of the invalid record can be reduced.

  Specifically, according to at least one embodiment, when there are not many invalid records in the data group (when the number is equal to or less than a threshold value), physical deletion of invalid records can be delayed. Also, according to at least one embodiment, index entries are managed in units of data groups such as pages, so that the amount of index data can be reduced. Further, according to at least one embodiment, even when records are not arranged in the order of expiration date, pages to be scanned can be limited, and invalid records can be efficiently deleted.

  The above-described database management system 1 can perform the same processes as the above-described various processes even when configured by a plurality of distributed database management apparatuses 100, thereby obtaining the same effects. it can.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 ... Database management system, 100 ... Database management apparatus, 110 ... Record management part (data processing part), 120 ... Index management part (representative expiration date determination part), 130 ... Invalid record deletion part (deletion part), 140 ... Communication 150, storage unit, 200 ... storage device, 210 ... data storage area, 300 ... client

Claims (9)

A data processing unit for writing or reading data to or from the storage device;
A representative expiration date determination unit that determines a representative expiration date of a data group in which the data is grouped based on an expiration date assigned to each of the data handled by the data processing unit;
Based on the representative expiration date determined by the representative expiration date determination unit, the data group to be deleted is extracted, and the data that has passed the expiration date among the data included in the extracted data group to be deleted Delete part to delete,
A database management device comprising: The representative expiration date determination unit generates index information for managing the data group and the representative expiration date of the data group in association with each other,
The deletion unit extracts a data group that has passed the representative expiration date based on the index information.
The database management device according to claim 1. The index information further includes additional information for regenerating the representative expiration date,
The representative expiration date determining unit regenerates the representative expiration date based on the additional information when data in the data group is updated.
The database management device according to claim 2. The representative expiration date determining unit updates the index information when the data processing unit writes or deletes data in the storage device.
The database management apparatus according to claim 2 or 3. The representative expiration date determining unit determines the representative expiration date of the data group based on a value obtained by performing statistical processing on a part or all of the expiration dates assigned to the respective data belonging to the same data group. Set,
The database management device according to any one of claims 1 to 4. The representative expiration date determining unit acquires the maximum value and minimum value of the expiration date assigned to each data belonging to the same data group, and based on one or both of the acquired maximum value and minimum value, Set a representative expiry date for the data group,
The database management device according to any one of claims 1 to 5. The representative expiration date determination unit sets the representative expiration date of the data group based on the expiration date assigned to each data belonging to the same data group and the size of each data in the data group;
The database management apparatus according to any one of claims 1 to 6. The computer of the database management device
Write or read data to the storage device,
Based on the expiration date assigned to each of the data to be written or read, determine a representative expiration date of a data group in which the data is grouped,
Extracting the data group to be deleted based on the determined representative expiration date, and deleting the data that has passed the expiration date from the data included in the extracted data group to be deleted,
Database management method. In the computer of the database management device,
Write or read data to the storage device,
Based on the validity period assigned to each of the data to be written or read, the representative validity period of the data group in which the data is grouped is determined,
Based on the determined representative expiration date, the data group to be deleted is extracted, and data that has passed the expiration date is deleted from the data included in the extracted data group to be deleted.
Database management program.

Images