RU2647648C1 - Method of organizing storage of historical deltas of records - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention relates to computer technology, in particular to methods of storing data, and can be used in a database management system (DBMS). Method for storing historical deltas of records in which at least one database record is created that contains data and a header; save the above-formed at least one entry in the page-organized file on the storage device, in which each data page contains a header and a data area for storing records; modifying at least one record on the page on the storage device; determine the delta of the record sufficient to restore the recording state to a modification; add the delta entry defined in the previous step to an ordered set of deltas, which is stored as part of the record.

EFFECT: technical result is to increase the performance of the DBMS by storing the deltas of the record together with the record.

7 cl, 6 dwg, 3 tbl

Description

FIELD OF TECHNOLOGY

[1] This technical solution relates to the field of computer technology, in particular to data storage methods, and can be used in a database management system (DBMS).

BACKGROUND

[2] The patent application US 20150039559 A1, Compressing a multi-version database, patent holder of International Business Machines Corp, proposes to place historical change deltas (DELTA CHANGES) on a separate solid-state drive (SSD). In this case, deltas of records accumulate in the so-called delta block (DELTA BLOCKS) and only after filling it are converted and saved to the data page. Deltas are not part of the original recording.

[3] Also known is the patent application US 20150046413 A1, “Delta store giving row-level versioning semantics to a non-row-level versioning underlying store”, patentee SAP SE, which proposes to provide competitive access to data by creating a delta storage (delta store) in RAM, and the main store (main store) does not allow parallel transactions on stored data and does not store historical deltas.

SUMMARY OF THE INVENTION

[4] This technical solution is aimed at eliminating the disadvantages inherent in existing data storage solutions.

[5] The technical task or problem posed in this solution is the organization of storage of historical record deltas during record modification.

[6] The technical result when solving the above problem is to increase the performance of the database management system by reducing the memory capacity and storing the deltas of the record along with the record.

[7] Also an additional technical result is an increase in the speed of operations associated with the search, reading and modification of historical record deltas stored with the record.

[8] Additionally, in solving the above problem, RAM is used rationally in a technical solution. Also, storing an array of historical deltas as part of a record can reduce the computational overhead associated with addressing and searching for a historic delta compared to competing solutions.

[9] The above technical result is achieved by implementing a method for organizing the storage of historical deltas of records in which at least one database record is formed that contains data and a header; storing the at least one record formed above into a page-organized file on a storage device in which each data page contains a title and a data area for storing records; modifying at least one entry on a page on a storage device; determining a recording delta sufficient to restore the recording state prior to modification; add the record delta defined in the previous step to the ordered set of record deltas, which is stored as part of the record.

[10] In some embodiments of the technical solution, the selection of the data page in which the record will be placed takes into account the correspondence of the total sizes of already placed records and / or their parts to threshold values.

[11] In some embodiments of the technical solution, the record is modified at the request of the user and / or DBMS service operation, and / or when performing stored procedures, and / or when triggers are triggered, and / or performing user-defined functions (UDF).

[12] In some embodiments of the technical solution, when the record is modified, the compliance of the total sizes of the records and / or their parts on the page with threshold values is controlled.

[13] In some embodiments of the technical solution, the recording delta contains an attribute or field of the recording and its old value.

[14] In some embodiments of the technical solution, the set of record deltas is ordered in chronological order.

[15] In some embodiments of the technical solution, the storage device is RAM, and / or ROM, and / or magnetic memory, and / or flash memory, and / or a magnetic disk and / or optical disk.

BRIEF DESCRIPTION OF THE DRAWINGS

[16] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

[17] In FIG. 1 schematically shows the recording structure, where 101 is the header, 102 is the body of the record, 103 is an array of historical deltas 104-106. 104 is the last historical delta added, and 106 is the first.

[18] In FIG. Figure 2 shows a data page (201) with a header (202) and a data area (203); thresholds PDS (206), PDH (207) and records (204, 205) are presented.

[19] In FIG. Figure 3 shows the recording states (301, 301 ', 301' ') at time instants t0 (312), t1 (313), t2 (314) into each of which a new historical delta was added (307, 308, 309, respectively). At time t0 and t1, the record (301, 301 ') is entirely on the data page (315). We assume that adding the next historical delta (309) would lead to exceeding the PHL threshold, therefore, previously added historical deltas (308 ', 307' ') were transferred to the historical data segment (316). If at time t3> t2 there are no transactions interested in saving the historical deltas 308 'and 307' ', the version data segment (316) can be freed.

[20] In FIG. 4, a block diagram has been translated illustrating an embodiment of placing a new record on a page.

[21] In FIG. 5 is a flowchart illustrating an embodiment of updating an existing record.

[22] In FIG. 6 is a flowchart illustrating an embodiment of deleting an existing record.

DETAILED DESCRIPTION OF THE INVENTION

[23] Below will be described the concepts and definitions necessary for the detailed disclosure of the invention.

[24] Record - a set of values of various types, intended for storage in memory.

[25] Database, database - a set of independent materials (articles, calculations, normative acts, court decisions and other similar materials) presented in an objective form, systematized so that these materials can be found and processed using an electronic computer (computer) )

[26] Database management system , Database Management System, DBMS, DBMS - a set of software and linguistic tools for general or special purposes, providing management of the creation and use of databases.

[27] Competitive access control using multi-versioning, MultiVersion Concurrency Control, MVCC - a mechanism for ensuring simultaneous competitive access to a database, which consists in providing each user with a so-called a snapshot of a database that has the property that user-made changes to the database are not visible to other users until the transaction is committed.

[28] A transaction is a group of sequential operations with a database, which is a logical unit of work with data. A transaction can be executed either entirely and successfully, observing data integrity and regardless of other transactions running in parallel, or not executed at all and then it should not have any effect.

[29] Historical record delta or record delta (English delta record) - a description of a record change that is sufficient to restore its original contents.

[30] An outdated historical record delta is a historical record delta designed to restore the contents of a record to a state that, according to the selected database isolation model, can no longer be claimed by any unfinished transaction.

[31] According to FIG. 1, a method of organizing the storage of historical deltas of records is as follows.

[32] Step 101 : forming at least one database record that contains data and a header;

[33] The newly created record (Fig. 1) consists of data (102 ') and a header (101') and has no elements in the array of historical deltas.

[34] Step 102 : save the at least one record generated above into a page-organized file on a storage device in which each data page contains a header and a data area for storing records;

[35] Records are stored in a page-organized file in which each data page consists of a data area and a header. The data area is used to store records, and part of the data area is reserved for the possible extension of existing records during updating and for storage of historical record deltas.

[36] The selection of the data page into which the record will be placed takes into account (Fig. 4) the correspondence of the total sizes of already placed records and / or their parts to the following threshold values:

PDH (page data hard limit) - the maximum allowable total size of all records on the page; upon reaching this value, it is forbidden (406) to insert new records on the page;

PDS (page data soft limit) - the maximum permissible total size of all records on a page without taking into account their historical deltas, and upon reaching a value of PDS it is forbidden (406) to insert new records on a page.

[37] Adding a new record (Fig. 4) is carried out only under the condition that neither the threshold PDH (402) nor the threshold PDS (403) is exceeded.

[38] At the same time, unlike the modification, for pages that do not fulfill the threshold conditions, no space optimization, search or removal of outdated records is performed. This approach is used to minimize the insertion time of a new record.

[39] Step 103 : modifying at least one entry on a page on a storage device;

[40] The reason for the modification of a record may be: user requests, DBMS service operations, execution of stored procedures, triggering of triggers, execution of user-defined functions (UDF), etc.

[41] Suppose that at time moment T = 1 the database has the following state:

[42] Transaction T0 performed the operations Object1 = “Foo” and Object2 = “Bar”, after which the transaction T1 performed the operation Object1 = “Hello”. Object1 will keep its old value “Foo” until all TX transactions created in the interval 0 <X <1 are completed, because they only see the value “Foo”.

[43] If a record is modified to ensure concurrent access control using MVCC, it is necessary to ensure storage of both the new state and value of the record, as well as information about its previous [record] state and value. The current value of the record is stored explicitly in the body of the record (102), and deltas from the array of historical deltas (103) are used to restore any of the previous states of the record. Each new delta is added when a record is modified and contains information sufficient to completely restore the previous record state after applying to the current record value. The storage format of the deltas is not a limiting factor and can be any.

[44] When modifying the record in this technical solution, a check is made on the correspondence of the total sizes of the records and / or their parts on the page (Fig. 2, Fig. 5) to the threshold values PDH (see description above) and PDL.

[45] PHL (page history limit) - the maximum permissible total size of all historical deltas of all records on a page; upon reaching this threshold, a page searches for obsolete historical deltas (508) and removes them (510) and / or transfer (509) parts historical deltas in a separate area of the memory - a segment of versioned data.

[46] If the above thresholds are exceeded when the record is modified, the use of free space on the page is optimized. To do this, if the threshold PDH (502) is exceeded, in all arrays of historical deltas of all records on the page, they search for deltas that are designed to restore states that are not “needed” by any unfinished transaction. Such deltas are called obsolete and can be removed (510), which will free up some space on the page.

[47] If even after deleting all obsolete deltas on the page, the threshold value of PDH remains exceeded, some historical deltas (the “oldest” ones that remain) are transferred (509) to a special area on the storage device — the version data segment.

[48] The versioned data segment (Fig. 3) is organized as one or more linked pages in a page-organized file that are designed to store only historical deltas, each versioned data segment is associated with no more than one data page. A versioned data segment for a page is created when an operation on records on a data page results in the PHL threshold being exceeded; a segment exists as long as it contains at least one historical obsolete delta. The threshold values are set when creating the table.

[49] In the case of deleting a record (Fig. 6), directly the entire record with data and outdated deltas can be deleted from the page only if there are no transactions left that started before the deletion, and all deltas are obsolete. Since the likelihood of such a situation is quite low when there are a large number of simultaneous transactions in the DBMS, instead of deleting, the record is marked (603) as deleted. Physical deletion of a record will occur when a record in a page is modified, and / or one of the thresholds is exceeded. In this case, when searching for outdated data on the page, the record will be found and deleted from the page.

[50] Step 104 : determine a recording delta sufficient to restore the recording state prior to modification;

[51] The storage format of the recording delta can be implemented in various ways. In the simplest case, the deltas store the indication of the attribute (field) of the record and its old value. Suppose, for example, an object with the attributes ID and STR changed during the DBMS operation as follows:

then the record can be arbitrarily represented as follows:

{id: 2, str: "Hello3"} [str: "Hello" → "Hello3"}] [Id: {1-> 2}] [str: {"Foo"} → "Hello"}] [id: {-> 1}, id {-> "Foo"}], where {id: 2, str: "Hello3"} is the last (current) value of the object, and historical deltas are given in brackets. The very first delta describes the creation of the record, so only the added value is indicated, since the previous one did not exist at that time.

[52] Step 105 : add the recording delta defined in the previous step to the ordered set of recording deltas, which is stored as part of the recording.

[53] Orderly storage is necessary to simplify the release (deletion) of obsolete records. Let the record

[54] {id: 2, str: "Hello3"} [str: {"Hello" → "Hello3"}] [Id: {1-> 2}] [str: {"Foo" → "Hello"}] [id: {-> 1}, id {-> "Foo"}]

[55] Describes changes in sequential (or serialized) transactions T0, T1, T2, T3 committed at time points 0, 1, 2, 3, respectively. Then, if at the time of searching and deleting obsolete record deltas, it is known that the system does not have incomplete transactions created earlier than time point 3, then record deltas describing earlier states can no longer be requested by anyone and are outdated and will be deleted without any additional checks, after which the entry will take the form:

{id: 2, str: "Hello3"} [str: {"Hello" → "Hello3"}] [] [] []

[56] The free space in the “tail” of the record will be reused to shift the deltas when a new record delta is added:

{id: 2, str: "NEW"} [str: {"Hello3" → "NEW"}] [str: {"Hello" → "Hello3"}] [] []

[57] In addition, as indicated above, the oldest record deltas, under certain conditions, can be transferred to another page in the versioned data segment. Access to the additional page carries certain overhead costs, but, as practice shows, the “older” the delta, the lower the likelihood that someone will need it, which means it is advisable to store more “popular” deltas on the same page as record, and older - transfer if necessary. This is another argument to keep deltas in order of time.

[58] Each record that is competitively accessed, in addition to directly the data and headers (Fig. 1, positions 101 and 102) at this step additionally contains an array of historical record deltas - a set of historical record deltas (Fig. 1, positions 104, 105 and 106), ordered in reverse chronological order: the first element contains the delta received last, and the last element received the first.

Let Object1 change during the DBMS operation as follows:

then, according to an embodiment of the invention, the record can be arbitrarily represented as “Hello3” [“2” → “3”] [“o” → “o2”] [“Foo” → “Hello”] [→ “Foo”],

those. the last value is stored (“Hello3”) and the chain of changes in reverse chronological order from T3 to T0.

[59] Let the following condition be satisfied for the record from the above example, that there is no transaction Tn for which the condition Т0 <Tn <Т3 is satisfied, then the record can be simplified to the following: “Hello3” [→ “Foo”].

Claims (12)

1. A method of organizing the storage of historical deltas of records, including the following steps: - form at least one database record that contains data and a header; - save the above-formed at least one record in a page-organized file on a storage device in which each data page contains a header and a data area for storing records; - carry out the modification of at least one entry on the page on the storage device; - determine the recording delta sufficient to restore the recording state before modification; - add the record delta defined in the previous step to the ordered set of record deltas, which is stored as part of the record. 2. The method according to p. 1, characterized in that the selection of the data page in which the record will be placed, taking into account the compliance of the total dimensions of the already posted records and / or their parts with threshold values. 3. The method according to claim 1, characterized in that the record is modified at the request of the user and / or DBMS service operation, and / or when performing stored procedures, and / or when triggers are triggered, and / or performing user-defined functions (UDF) . 4. The method according to p. 1, characterized in that during the modification of the record control the compliance of the total size of the records and / or their parts on the page with threshold values. 5. The method according to claim 1, characterized in that the recording delta contains an attribute or field of the record and its old value. 6. The method according to p. 1, characterized in that the set of record deltas is ordered in chronological order. 7. The method according to p. 1, characterized in that the storage device is RAM, and / or ROM, and / or magnetic memory, and / or flash memory, and / or magnetic disk, and / or optical disk.

Images