WO2021073571A1

WO2021073571A1 - Fragment-free recovery-based database multi-version concurrency control system

Info

Publication number: WO2021073571A1
Application number: PCT/CN2020/121149
Authority: WO
Inventors: 陈元熹; 许建辉; 王涛
Original assignee: 深圳巨杉数据库软件有限公司
Priority date: 2019-10-16
Filing date: 2020-10-15
Publication date: 2021-04-22
Also published as: CN110825752B; CN110825752A; CA3130011A1

Abstract

A fragment-free recovery-based database multi-version concurrency control system, comprising a transaction information recording module, a transaction data management module, a data visibility optimization module, a memory optimization speed-up module, and a multi-version recovery module. In the system, rollback segments are designed by using file groups of fixed-length files or fixed-length logical segments, a set of MVCC implementation mechanism and rollback segment recovery and cleaning methods are provided, and memory access optimization is combined to improve access response time, thereby avoiding significant system overhead caused when the rollback segments are recovered, and avoiding the fragmentation of a disk. Meanwhile, in the situation in which there are enough disks, the configurable rollback segments may ensure that there are no limitations on the number of concurrently running transactions and the length of transactions. In addition, the design of the combination of a memory and disks may avoid or reduce disk access in most cases, thereby ensuring the access performance of the system.

Description

Database multi-version concurrency control system based on non-fragment recovery

Technical field

The invention relates to the technical field of system optimization, in particular to a database multi-version concurrency control system based on non-fragment recovery.

Background technique

When traditional databases support concurrent transaction access, they usually provide different data access isolation levels to meet the application's use of data. At present, there are mainly two implementation methods: the mechanism based on transaction lock and the mechanism based on multi-version concurrency control. The biggest drawback of the former is that there will be mutual exclusion of reads and writes, which reduces the concurrency of the system. So the latter has gradually become the current mainstream implementation. The principle is to add a new data version to the back of the data structure when the data is changed or deleted, and the data version of the original version is retained in the system. At the same time, an incremental version number identification should be recorded each time data is generated, so that during the query, it can be determined which version of the data to use according to the identification at the beginning of the query transaction.

The biggest problem based on the multi-version concurrency control mechanism is the need to consume additional storage space. All databases implemented based on the multi-version concurrency control mechanism need to implement a space recovery mechanism to release the versions that are no longer needed. In practice, these recycling processes will bring additional CPU overhead, disk read and write operations, and memory operations. During the triggering process of the recycling operation, the system processing capacity is greatly affected, and the stability is greatly reduced. After recycling, it will also cause disk fragmentation, which reduces the efficiency of disk access.

Summary of the invention

The technical problem to be solved by the embodiments of the present invention is to provide a database multi-version concurrency control system based on non-fragmentation recovery, which can adopt a simple and effective storage recovery mechanism that will not cause too much impact on the system, thereby It greatly improves the database performance of multi-version concurrency control.

In order to solve the above technical problems, an embodiment of the present invention provides a database multi-version concurrency control system based on non-fragmentation recovery, including:

The transaction information recording module is used to assign a unique identifier to each transaction in the database, and to record the oldest uncommitted transaction in the system and the transaction information initiated after the oldest uncommitted transaction;

The transaction data management module is used to use the rollback segment space to perform operation management and access optimization on transactions and recorded data; wherein, the rollback segment is composed of a file or a set of fixed-length files;

The data visibility optimization module is used to use the record exclusive lock to perform write and write mutual exclusion constraints on the access operation of the transaction; according to the read lock result of the transaction on the record lock, the record version information is optimized for access; according to the initial log sequence of the transaction The comparison result of the log number and the original log sequence number of the record, restricts the visibility of the records of the old version;

The memory optimization speed-up module is used to cache the latest old version record in the system into a non-disk memory structure; wherein, the non-disk memory structure includes a hash bucket;

The multi-version recovery module is used to distinguish multiplexed files and non-multiplexed files in the rollback segment space, clear the non-multiplexed files, and recover the multiplexed files.

Further, the transaction information includes a transaction ID, submission status information, and a time stamp at the time of submission.

Further, the transaction data management module includes:

The transaction access management unit is used to use the space of the rollback segment to perform operation management on the transaction; among them, the operation of the transaction on the record includes the creation operation, the modification operation and the deletion operation;

The rollback segment space management unit is used to store and access the record version in the rollback segment.

Further, the rollback segment space is composed of a hash table and a disk file; wherein the number of the disk file is one or a group; when the rollback segment space is composed of a group of disk files, the disk Each file of the file is a fixed-length file.

Further, the data visibility optimization module includes:

The transaction access operation constraint unit is used to use the record exclusive lock to perform write and write mutual exclusion constraints on transaction access operations;

The record version information access optimization unit is configured to perform access optimization processing on the recorded version information according to the read lock result of the record lock by the transaction; wherein the read lock result includes a successful read lock application and a failed read lock application;

The old version record access restriction unit is used to restrict the access visibility of the old version record according to the comparison result of the transaction initial log sequence number and the record initial log sequence number.

Further, the multi-version recovery module includes:

The rollback segment file distinguishing unit is used to compare the rollback segment file with the oldest uncommitted transaction based on the initial log sequence number of the transaction, and distinguish between reused files and non-reusable files in the rollback segment space according to the comparison result file;

The rollback segment file clearing unit is used to clear non-reusable files in the rollback segment space;

The rollback segment file recovery unit is used to recover the reused files in the rollback segment space as a recycling file.

Further, the method for triggering the clearing and reclaiming operation of the rollback segment space by the multi-version reclaiming module includes:

Trigger according to the preset time interval; or,

Triggered according to the preset threshold of the number of times the rollback segment file is used.

Compared with the prior art, the present invention has the following beneficial effects:

1. The recovery of the rollback segment is simple, there is no significant system overhead, no system performance jitter, and no disk fragmentation;

2. The configurable rollback segment can guarantee that there are no restrictions on the number of concurrently running transactions and the length of the transaction when there are enough disks;

3. The design of the combination of memory and disk can meet most situations (short transactions) to avoid or reduce disk access, thereby ensuring the access performance of the system.

Description of the drawings

FIG. 1 is a schematic structural diagram of a database multi-version concurrency control system based on non-fragmentation recovery provided by an embodiment of the present invention;

Fig. 2 is a schematic diagram of an application example of rollback segment access provided by an embodiment of the present invention.

Detailed ways

The following describes the technical solutions in the embodiments of the present invention clearly and completely with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Compared with the prior art, the present invention uses a fixed-length data segment to store multi-version data records in a chain structure, and retains the last updated data version in this segment in the metadata of the data segment. When the old data is reclaimed, it is determined whether the multi-version records of the segment are all older than the active transaction range of the current database system according to the metadata in the data segment to determine whether the entire data needs to be reclaimed. This large-block data recovery mechanism can ensure the recovery efficiency, will not cause disk fragmentation, and effectively reduce the overhead on the system.

The following is an explanation of terms in the embodiments of the present invention:

Transaction (Transaction): A program execution logic unit composed of a series of operations to access and update data in the system. Transactions have Atomicity, Consistency, Isolation, and Duration, referred to as ACID.

Transaction Log Record (LR, Log Record): An additional record in the database system used to record any changes to the database to ensure that the data can be restored to the state before the update when the transaction is rolled back.

Isolation level: In the standard SQL specification, 4 transaction isolation levels are defined, and different isolation levels handle transactions differently. The 4 isolation levels are: read unmentioned (READ_UNCOMMITTED), read submitted (READ_COMMITTED), repeatable read (REPEATABLE_READ), sequential read (SERIALIZABLE).

Multi-version concurrency control (MVCC): A concurrency control mechanism in the database to solve the problem of phantom reading of data, so as to achieve repeatable reading without blocking the reading.

The oldest uncommitted transaction (lowTran): the earliest started and uncommitted transaction in the system.

Log Sequence Number (LSN, Log Sequence Number): The increasing sequence number in the corresponding log when the event occurs.

Transaction ID (TID, Transaction ID): the unique identification number of the transaction.

Transaction Begin LSN (TBLSN, Transaction Begin LSN): The LSN at the beginning of the transaction. Due to the uniqueness of LSN, TBLSN can be used to identify a transaction. In the present invention, TBLSN can preferably be used instead of TID.

Record ID (RID): A unique identification of a record, usually containing information for locating a record, such as the logical or physical location on the disk.

Rollback Segment: Store the old version of the data segment, which can be in memory or on disk.

Referring to Fig. 1, an embodiment of the present invention provides a database multi-version concurrency control system based on non-fragmentation recovery, including:

The transaction information recording module is used to assign a unique identifier to each transaction in the database, and to record the oldest uncommitted transaction in the system and the transaction information initiated after the oldest uncommitted transaction; further, the The transaction information includes transaction ID, submission status information, and time stamp at the time of submission.

In the specific embodiment, it should be noted that each transaction in the database has its unique identifier. This identification is logically incremented according to the order in which the transactions enter the system. In the implementation of various databases, IDs that contain a timestamp or a log sequence number (LSN) are usually used for identification. The database management system usually records the lowtran in the current system and the status of all transactions starting after the lowtran, including at least the transaction ID, whether it is committed, and the time stamp when it is committed. For each record, the system also implicitly marks the transaction ID that created the record version for visibility judgment.

Further, the transaction data management module includes:

In the embodiment of the present invention, preferably, the rollback segment space is composed of a hash table and a disk file; wherein the number of the disk files is one or a group; when the rollback segment space is composed of a group of disk files When the files are composed, each file of the disk file is a fixed-length file.

In the embodiment of the present invention, the modification process of the transaction to the data is as follows:

1. Obtain TBLSN from the system when all transactions are initiated and write a transaction start log.

2. When a transaction is created (insert), modified (update) or deleted (delete) a record, the record lock is acquired first, and the original version record (only for modification and deletion operations) is put into the rollback segment after success. The position is directly changed or inserted into the record, and the header of the record contains the TBLSN of the transaction, which is the transaction TBLSN that generated this version of the data. The delete operation can directly delete the record.

3. The rollback segment contains a hash table in memory and one or a group of disk files. The file consists of multiple fixed-length data segments. The records are in a chain structure in the file according to the version. Refer to Figure 2. The record will be hashed into different hash buckets according to the RID, and each bucket stores the disk address of the last old version of the record. Each old version record on the disk will contain a pointer to an older version to form a chain. But when adding a new record version through the hash bucket, the system only needs to append that record at the end of the rollback segment, point its pointer to the address originally saved in the hash bucket, and then change the address in the hash bucket to point to the newly added record. Just record. In this way, each hash bucket corresponds to the old record version chain.

Of course, due to the limited number of hash buckets, there will still be conflicts on each bucket, that is, multiple records may appear on a chain, and the system can find matching records by comparing the RID.

4. The rollback segment is composed of a file or a group of fixed-length files;

When it is composed of a group of files, each file has a fixed length size. In the metadata of the file, it can be placed in the file header or elsewhere to save the maximum TBLSN (maxTBLSN) of all recorded transactions in the file. This TBLSN is different from the TBLSN on the record header, it is the TBLSN of the transaction that wants to modify the version record. When a file is full, or the free space is not enough to save the next record, the rollback segment automatically switches to the next file. In a specific embodiment, the size of a single file and the total number of files can be artificially limited, or there is no limitation. When the limit is not set, it may take up a lot of disk space, but when the limit is set, the number of uncommitted transactions allowed in the database system and the maximum allowable execution time of uncommitted transactions are also limited.

When the rollback segment is composed of a file, the file can be composed of multiple fixed-length logical segments. Each logical segment stores the maxTBLSN in this segment. The logic segment switching logic is similar to the file switching logic. There will be metadata in the system to track the currently used file or logical segment information.

The data visibility optimization module is used to use the record exclusive lock to perform write and write mutual exclusion constraints on the access operation of the transaction; according to the read lock result of the transaction on the record lock, the record version information is optimized for access; according to the initial log sequence of the transaction The result of the comparison between the number and the sequence number of the record initial log restricts the visibility of the old version record; further, the data visibility optimization module includes:

The record version information access optimization unit is configured to perform access optimization processing on the recorded version information according to the read lock result of the record lock of the transaction; wherein the read lock result includes a successful read lock application and a failed read lock application;

In the embodiment of the present invention, there are multiple judgment methods for data visibility and are independent of the design of the above rollback segment. The following specific examples describe an optimized implementation method combining transaction TBLSN and transaction lock:

1. Insert/modify/delete operations need to hold a record exclusive lock to ensure that writes are mutually exclusive;

2. The read operation will first try to obtain the shared lock of the record. According to the success/failure of obtaining the record lock, the version information of the record can be handled in the following three situations:

(1) The read lock application is successful. If the read transaction TBLSN is greater than the TBLSN of the current disk record, it means that the read transaction is initiated after the latest version is generated, and the read transaction can directly use the current disk version record;

(2) The read lock application is successful and the read transaction TBLSN is less than the TBLSN recorded on the disk, and the read transaction needs to use the old version in the rollback segment. In this scenario, the record lock can be released immediately, that is, there is no need to hold the record lock in the process of reading the old version.

(3) The read lock application fails, the read transaction needs to use the old version, and the specific operation is similar to case 2. Also do not need to hold the record lock in the process of reading the old version

3. When using the old version, first find the corresponding hash bucket through RID, and read the address (file plus offset) recorded in the rollback segment. Read the record information. If the RID matches, compare the transaction TBLSN with the record TBLSN. If the record TBLSN is less than the transaction TBLSN, the record is visible to the transaction, otherwise an older version is found according to the pointer until the visible version is found.

The memory optimization speed-up module is used to cache the most recent old version record in the system into a non-disk memory structure; wherein, the non-disk memory structure includes a hash bucket; in the embodiment of the present invention, it should be noted that because The disk access speed is too slow, which may affect the response time of the transaction. In order to improve the response speed, the system can cache the most recent old version in a hash bucket or other memory structure. This ensures that in a short transaction scenario, this old record version in the memory can meet most of the access requirements, thereby avoiding disk access.

The multi-version recovery module is used to distinguish multiplexed files and non-multiplexed files in the rollback segment space, clear the non-multiplexed files, and recover the multiplexed files. Further, the multi-version recovery module includes:

In the embodiment of the present invention, further, the method for triggering the clearing and reclaiming operation of the rollback segment space by the multi-version reclaiming module includes:

Trigger according to the preset time interval; or,

It should be noted that the system can set a background task to be responsible for reclaiming the space of the rollback segment. Among them, when using a set of rollback segment files, if maxTBLSN>lowtran of a rollback segment file, the file can be reused or deleted. When using a rollback segment file, the logical segment can be reused when maxTBLSN>lowtran of a logical segment. It is understandable that the principle is that all transactions that may access this version have ended (commit or rollback), any running or new transactions only need to access the latest data items on the current disk or relatively new in the rollback segment The old version is recorded, and the entire database system no longer needs the record in this file or logical segment.

Such a background task is very simple, only need to scan the metadata of the file or logical segment to clear the unneeded large fixed-length space. Generally, you can configure the fixed-length size and the total number of logical blocks/files according to the actual load of the database, and choose whether to reuse the space. You can select the mode of one file and multiple logical blocks when deciding to reuse, and select the mode of multiple files when not reusing space. In the multi-file (not multiplexed) mode, clearing means deleting unused files. In the reuse mode, the system only needs to remember which file or logical block has been recovered, and then recycle it. In a specific embodiment, the background task may be set to be triggered periodically, or triggered when the use of the rollback segment reaches a certain preset threshold.

In summary, the embodiment of the present invention uses fixed-length file file groups or fixed-length logical segments to design rollback segments, provides a set of MVCC implementation mechanism and rollback segment recovery and cleaning methods, and combines memory to improve access response time. Compared with the existing MVCC implementation, it has the following advantages:

It should be noted that for the above method or process embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not affected by the described sequence of actions. Limitation, because according to the embodiment of the present invention, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are optional embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

The above are the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications are also considered This is the protection scope of the present invention.

Claims

A database multi-version concurrency control system based on non-fragmentation recovery, which is characterized in that it includes:

The transaction information recording module is used to assign a unique identifier to each transaction in the database, and to record the oldest uncommitted transaction in the system and the transaction information initiated after the oldest uncommitted transaction;

The transaction data management module is used to use the rollback segment space to perform operation management and access optimization on transactions and recorded data; wherein, the rollback segment is composed of a file or a set of fixed-length files;

The data visibility optimization module is used to use the record exclusive lock to perform write and write mutual exclusion constraints on the access operation of the transaction; according to the read lock result of the transaction on the record lock, the record version information is optimized for access; according to the initial log sequence of the transaction The comparison result of the log number and the original log sequence number of the record, restricts the visibility of the records of the old version;

The memory optimization speed-up module is used to cache the latest old version record in the system into a non-disk memory structure; wherein, the non-disk memory structure includes a hash bucket;

The multi-version recovery module is used to distinguish multiplexed files and non-multiplexed files in the rollback segment space, clear the non-multiplexed files, and recover the multiplexed files.
The database multi-version concurrency control system based on non-fragmentation recovery according to claim 1, wherein the transaction information includes a transaction ID, submission status information, and a timestamp at the time of submission.
The database multi-version concurrency control system based on non-fragmentation recovery according to claim 1, wherein the transaction data management module comprises:

The transaction access management unit is used to use the space of the rollback segment to perform operation management on the transaction; among them, the operation of the transaction on the record includes the creation operation, the modification operation and the deletion operation;

The rollback segment space management unit is used to store and access the record version in the rollback segment.
The database multi-version concurrency control system based on non-fragmentation recovery according to claim 3, wherein the rollback segment space is composed of a hash table and a disk file; wherein the number of the disk file is one or one Group; when the rollback segment space is composed of a group of disk files, each file of the disk file is a fixed-length file.
The database multi-version concurrency control system based on non-fragmentation recovery according to claim 1, wherein the data visibility optimization module comprises:

The transaction access operation constraint unit is used to use the record exclusive lock to perform write and write mutual exclusion constraints on transaction access operations;

The record version information access optimization unit is configured to perform access optimization processing on the recorded version information according to the read lock result of the record lock by the transaction; wherein the read lock result includes a successful read lock application and a failed read lock application;

The old version record access restriction unit is used to restrict the access visibility of the old version record according to the comparison result of the transaction initial log sequence number and the record initial log sequence number.
The database multi-version concurrency control system based on non-fragmentation recovery according to claim 1, wherein the multi-version recovery module comprises:

The rollback segment file distinguishing unit is used to compare the rollback segment file with the oldest uncommitted transaction based on the initial log sequence number of the transaction, and distinguish between reused files and non-reusable files in the rollback segment space according to the comparison result file;

The rollback segment file clearing unit is used to clear non-reusable files in the rollback segment space;

The rollback segment file recovery unit is used to recover the reused files in the rollback segment space as a recycling file.
The database multi-version concurrency control system based on non-fragmentation recovery according to claim 1, wherein the multi-version recovery module triggers the clearance and recovery operation of the rollback segment space including:

Trigger according to the preset time interval; or,

Triggered according to the preset threshold of the number of times the rollback segment file is used.