CN115729748A

CN115729748A - High-throughput log-free online transaction processing method for nonvolatile memory

Info

Publication number: CN115729748A
Application number: CN202211520856.2A
Authority: CN
Inventors: 姚士宇; 林菲; 张聪
Original assignee: Hangzhou Dianzi University
Current assignee: Hangzhou Dianzi University
Priority date: 2022-11-29
Filing date: 2022-11-29
Publication date: 2023-03-03

Abstract

The invention discloses a high-throughput log-free online transaction processing method for a nonvolatile memory; the method comprises the following steps: 1. constructing an engine architecture; 2. and the engine architecture constructed by the step one is used for carrying out a no-log transaction processing process when a transaction request is received. If the database is crashed, executing a transaction crash recovery process; the invention sets the last persistent bit in the NVM tuple, thereby still realizing crash recovery under the condition of not recording logs. The engine architecture provided by the invention adopts three new technologies of tuple caching (Met-Cache) enhanced by metadata, log-free persistent transactions and lightweight NVM space management to solve the problems of frequent NVM write operation, NVM write redundancy and NVM persistent overhead.

Description

High-throughput log-free online transaction processing method for nonvolatile memory

Technical Field

The invention belongs to the technical field of databases, and particularly relates to a high-throughput log-free online transaction processing method for a nonvolatile memory.

Background

The emerging non-volatile memory (NVM) has the characteristics of byte-addressable, almost as fast as DRAM, larger capacity than DRAM, etc., and brings huge performance potential for OLTP database, but the characteristics of having write bandwidth lower than read bandwidth, cache line refresh and memory barrier command, etc. higher overhead than ordinary write operation, limited number of write operations, etc. require that the NVMOLTP system must be designed to satisfy 3 design principles: placing frequently accessed data structures in DRAM minimizes NVM write operations and minimizes endurance operations.

MMDB systems with NVM As with existing MMDB designs, the system stores tuples and indices in volatile memory, puts WAL (write-ahead) and checkpoints in the NVM. After a system crash, the tuples and indices in volatile memory are considered lost and recovery is based on the log and checkpoints in NVM. This design writes the modified tuple to the WAL and checkpoint, resulting in two additional NVM write operations on the tuple. Recent research has proposed several NVM-based log improvement schemes, including WBL, which significantly reduces log size, does not maintain checkpoints, significantly reduces the number of NVM writes, and FOEDUS, which completely handles transactions in DRAM, thereby avoiding writing metadata in NVM, among others. However, there are three important issues with these designs: firstly, in MMDB and WBL, tuple metadata is stored with tuples in NVM, tuple metadata modification may result in frequent NVM write operations, secondly, in MMDB and FOEDUS, modified tuples are written into tuple heap, log, checkpoint of NVM, there is NVM write redundancy problem, and finally coarse-grained NVM space management may generate large NVM persistence overhead.

Disclosure of Invention

The invention aims to provide a high-throughput log-free online transaction processing method for a nonvolatile memory.

The invention relates to a high-throughput log-free online transaction processing method for a nonvolatile memory, which comprises the following steps of:

step one, constructing an engine architecture; the engine architecture includes NVM metadata, an index, and a plurality of base tables. Each base table includes a tuple heap stored in NVM, met-Cache stored in DRAM, and NVM tuple manager for each thread in the base table.

The NVM tuple is composed of a tuple header and tuple data. The tuple header includes the last persistence bit LP, a transaction commit timestamp, a tuple ID, and a delete bit. The tuple ID and transaction commit timestamp are used to uniquely identify the different tuple versions. The delete bit is used to show whether the logical tuple has been deleted; the last persistence bit LP is used to show whether the logical tuple is the last tuple to persist in the commit transaction.

The Met-Cache consists of a plurality of entries; one entry contains metadata data and 7 metadata fields: the first metadata field is a pointer to the NVM tuple, the second metadata field is a tuple ID, the third metadata field is a dirty bit, the fourth metadata field is an active bit, the five metadata fields are clock bits, the sixth metadata field is a duplicate bit, and the seventh metadata field is a CC-Meta field.

And step two, performing a no-log transaction processing process when the transaction request is received through the engine architecture constructed in the step one.

2-1 execution phase, performing execution transaction processing for transaction request in DRAM

2-2. The persistence stage comprises the following specific processes:

2-2-1, persisting all tuples written in the transaction execution process except the last tuple into the NVM;

2-2-2, persisting all data of the last tuple written in the transaction execution process except the tuple header to the NVM;

2-2-3, execute the sfence write barrier instruction to ensure that all tuples are persisted before the last persisted bit LP is persisted.

2-2-4. Mark the last persistent bit LP of the last tuple written during execution of the transaction as persistent and persist its tuple header to NVM using an NVM atomic write operation.

In step two, if the database crashes, a transaction crash recovery process is executed, which is specifically as follows:

(1) The engine architecture runs multiple threads, each of which is responsible for scanning one NVM tuple heap area, initializing the timestamp ts-commit and the list pending-list. ts-commit represents the maximum transaction commit timestamp that has occurred so far. pending-list represents a set of tuples that have not been determined to have committed.

(2) Performing first scanning, wherein the area scanned for the first time is the whole NVM tuple stack, and for each NVM tuple, if the last persistence bit LP is set, updating the timestamp ts-commit; if the delete bit or transaction commit timestamp is equal to 0, then the NVM tuple is placed in the garbage queue.

If the transaction submission timestamp of the NVM tuple is less than or equal to the timestamp ts-commit, updating the main index by using the NVM tuple, otherwise, putting the NVM tuple into a pending list pending-list;

(3) Performing second scanning, wherein the area of the second scanning is each NVM tuple in the pending list pending-list; at this point, the timestamp ts-commit is the maximum transaction commit timestamp for the scanned region in the first scan; for each NVM tuple, if the transaction commit timestamp of one NVM tuple is less than or equal to the timestamp ts-commit, updating the main index by using the NVM tuple; otherwise, indicating that the transaction associated with the NVM tuple crashes during persistence, and marking the NVM tuple as discarded by marking a tuple header corresponding to the NVM tuple as empty and persisting the NVM tuple to the NVM through NVM atomic write operation; at the same time, the NVM tuple is placed in a garbage queue.

(4) The sfence write barrier instruction is executed to ensure that the NVM tuples of all crash transactions are marked as discarded.

Preferably, the specific process of the execution stage in the step two is as follows:

2-1-1, taking the time when the transaction request of the client is received as the transaction starting time, and obtaining the time stamp corresponding to the transaction starting time.

2-1-2, searching whether a request tuple exists in the main index; if not, entering the step 2-1-3; otherwise, directly entering the step 2-1-4;

2-1-3. Reading the NVM tuple and enhancing the NVM tuple using the CC-Meta field of the concurrency control method to construct an entry for Met-Cache, and updating the index using the location of the Met-Cache entry based on the request tuple.

2-1-4, the engine architecture executes the affairs according to the Met-Cache entry of the request tuple in a concurrent manner; if no conflict exists and the transaction is successfully submitted, directly entering the persistence stage of the step 2-2; if the transaction aborts, go to step 2-1-5;

2-1-5. Check if the dirty bit of all Met-Cache entries accessed by the transaction has been set. For Met-Cache entries whose dirty bits have been set, the entries are respectively restored from the NVM tuples pointed to by the corresponding NVM tuple pointers.

Preferably, in step two, the maintenance phase is entered after the persistence phase. The specific process of the maintenance stage is as follows: when a thread finds that a new version exists, the NVM tuples of the old version are put into a garbage queue.

Preferably, for the garbage queue, the engine architecture periodically extracts the transaction commit timestamp of the last committed transaction of all threads, and the minimum value of the obtained transaction commit timestamps is used as the global minimum timestamp. NVM tuples in the garbage queue having transaction commit timestamps less than the global minimum timestamp are moved from the garbage queue to an allocator free list.

Preferably, the engine architecture further comprises transaction private data. The transaction private data and the index are stored in the DRAM.

Preferably, the NVM metadata stores a table schema and a coarse-grained allocation structure;

preferably, the tuple heap consists of fixed-size pages. Each page contains a fixed number of NVM tuple slots;

preferably, the NVM tuple heap contains multiple tuple versions of a logical tuple;

preferably, the Met-Cache manages tuple caching for the corresponding NVM tuple heap in the DRAM.

Preferably, a main index in the indexes comprises a key and a value; the key is the primary key of the tuple, and the value points to the Cache entry in the Met-Cache or the latest version in the NVM tuple heap.

Preferably, the engine architecture supports multi-threaded concurrent transaction processing, and each thread has a private space in the DRAM

The invention has the beneficial effects that:

1. the engine architecture provided by the invention adopts three new technologies of tuple caching (Met-Cache) enhanced by metadata, log-free persistent transactions and lightweight NVM space management to solve the problems of frequent NVM write operation, NVM write redundancy and NVM persistent overhead.

2. The invention sets the last persistent bit in the NVM tuple, thereby still realizing crash recovery under the condition of not recording logs.

3. The invention provides an improved crash recovery scanning algorithm, wherein the maximum transaction commit timestamp (ts-commit) occurring so far is used for identifying the committed tuples as much as possible, the whole NVM tuple stack does not need to be scanned in the second scanning, and only each NVM tuple in a pending list (pending-list) needs to be scanned, so that the crash recovery efficiency is improved.

Drawings

FIG. 1 is an architecture diagram of an engine architecture constructed in accordance with the present invention.

Detailed Description

The invention is further described below with reference to the accompanying drawings.

As shown in fig. 1, a high-throughput log-free online transaction processing method for a non-volatile memory includes the following steps:

step one, constructing an engine architecture; the engine architecture includes NVM metadata, an index, transaction private data, and a plurality of base tables (i.e., htables). Each base table includes a tuple heap stored in NVM (non-volatile memory), met-Cache (metadata-enhanced tuple Cache) stored in DRAM (volatile memory), and NVM tuple manager for each thread in the base table; a thread can access the base tables through the corresponding NVM tuple manager in the base tables.

The NVM metadata stores a table mode and a coarse-grained distribution structure; the transaction private data and the index are stored in the DRAM.

The engine architecture stores all tuples in the base table as NVM tuples in an NVM tuple heap; the tuple pile consists of fixed-size (2 MB in this embodiment) pages. Each page contains a fixed number of NVM tuple slots; the NVM tuple consists of a tuple header and tuple data. The NVM tuple pile contains multiple tuple versions of a logical tuple; the tuple header includes a last persistence bit (LP), a transaction commit timestamp (Tx-CTS), a tuple ID, and a delete bit. The tuple ID and transaction commit timestamp (Tx-CTS) are used to uniquely identify the different tuple versions. The delete bit is used to show whether the logical tuple has been deleted; the last persistence bit (LP) is used to show whether the logical tuple is the last tuple to persist in the commit transaction.

The Met-Cache manages tuple caching in DRAM for the corresponding NVM tuple heap. The engine architecture performs concurrency control entirely in the DRAM through the Met-Cache.

The Met-Cache consists of a plurality of entries; one entry contains meta-data and 7 meta-data fields: the first metadata field points to a pointer to the NVM tuple, the second metadata field is the tuple ID, the third metadata field is a dirty bit, the fourth metadata field is an active bit (indicating whether the entry is used by an active transaction), the fifth metadata field is a clock bit (supporting the cache replacement algorithm), the sixth metadata field is a copy bit (indicating whether the entry has been copied), and the seventh metadata field is the CC-Meta field (metadata related to the concurrency control method being used).

The engine architecture maintains an index in DRAM for each base table, rebuilding the index upon crash recovery. The primary index is required and the secondary index is optional. The primary index contains a key and a value; the key is the primary key of the tuple, and the value points to the Cache entry in the Met-Cache or the latest version in the NVM tuple heap. The engine architecture supports multi-thread concurrent transaction processing, and each thread has a private space in the DRAM for recording transaction private data such as read, write and insert activities of the transaction.

And step two, performing a no-log transaction processing process when the engine architecture constructed in the step one receives a transaction request of the client, wherein the no-log transaction processing process specifically comprises three stages of execution, persistence and maintenance.

2-1, performing: transactions are performed in the DRAM.

2-1-1, taking the time when the transaction request of the client is received as the transaction starting time, and obtaining a time stamp (Tx-CTS) corresponding to the transaction starting time.

2-1-2, searching whether a request tuple exists in the main index; if not, entering the step 2-1-3, otherwise, directly entering the step 2-1-4;

2-1-3, reading an NVM tuple and enhancing the NVM tuple by using a CC-Meta field unique to a currently used concurrency control method to construct an entry of Met-Cache, and updating an index by using the position of the Met-Cache entry based on the request tuple;

2-1-4, the engine architecture completely and concurrently executes the transaction in the DRAM according to the Met-Cache entry of the request tuple; if no conflict exists and the transaction can be successfully submitted, directly entering the persistence phase processing of the step 2-2; if the transaction aborts, proceed to step 2-1-5;

2-2. Persistence: the newly written tuple is persisted to NVM.

2-2-3. Execute the sfence write fence instruction, guarantee to persist all tuples before persisting the last persisted bit (LP).

2-2-4. Mark the last persistent bit (LP) of the last tuple written during execution of the transaction as persistent and persist its tuple header to NVM using an NVM atomic write operation. In this way, the last persistent bit (LP) can achieve the same effect as committing log records; during crash recovery, if the last persistent bit (LP) exists, the transaction has committed; otherwise, indicating a crash during the persisted transaction, any NVM tuples written by the transaction must be discarded.

2-3, maintenance: an expired tuple is collected.

To reduce contention, each thread has a private NVM heap allocator and garbage queue. When a thread finds that a newer version exists, the NVM tuple for the old version is placed into the garbage queue. Entries in the garbage queue cannot be released directly because the relevant NVM tuple version may still be used by other transactions. The engine architecture periodically extracts the transaction commit timestamp (Tx-CTS) of the last committed transaction of all threads, and the minimum value of the acquired transaction commit timestamps (Tx-CTS) is used as the global minimum timestamp Tx-CTS. The NVM tuples in the garbage queue having transaction commit timestamps less than the global minimum timestamp Tx-CTS are moved from the garbage queue into the allocator free list. This approach can avoid the erroneous removal of the NVM tuple version being accessed by the current transaction to the free list.

(1) The engine architecture runs multiple threads, each of which is responsible for scanning one NVM tuple heap area, initializing the timestamp ts-commit and the list pending-list. ts-commit represents the largest transaction commit timestamp that has occurred so far, and pending-list represents the set of tuples that have not been determined to have committed.

(2) Performing a first scanning, wherein the area of the first scanning is the whole NVM tuple heap responsible for the thread, and if the last persistent bit (LP) is set, updating the timestamp ts-commit for each NVM tuple; if the delete bit or transaction commit timestamp is equal to 0, then the NVM tuple is placed in the garbage queue.

(3) Performing second scanning, wherein the area of the second scanning is each NVM tuple in the pending list pending-list; at this point, the timestamp ts-commit is the maximum transaction commit timestamp for the scanned region in the first scan; for each NVM tuple, if the transaction commit timestamp of one NVM tuple is less than or equal to the timestamp ts-commit, updating the main index with the NVM tuple; otherwise, indicating that the transaction associated with the NVM tuple crashes during the persistence, and making the NVM tuple marked as discarded by marking the header of the tuple corresponding to the NVM tuple as empty (Tx-CTS = 0) and persisting the NVM tuple into the NVM through an NVM atomic write operation; at the same time, the NVM tuple is placed in a garbage queue.

Claims

1. A high throughput log-free online transaction processing method for non-volatile memory, characterized by: the method comprises the following steps:

step one, constructing an engine architecture; the engine architecture comprises NVM metadata, an index and a plurality of base tables; each base table comprises a tuple heap stored in the NVM, a Met-Cache stored in the DRAM, and an NVM tuple manager of each thread in the base table;

the NVM tuple consists of a tuple header and tuple data; the tuple header includes a last persistence bit LP, a transaction commit timestamp, a tuple ID, and a delete bit; the tuple ID and the transaction submission timestamp are used for uniquely identifying different tuple versions; the delete bit is used to show whether the logical tuple has been deleted; the last persistence bit LP is used to show whether the logical tuple is the last tuple to persist in the commit transaction;

the Met-Cache consists of a plurality of entries; one entry contains meta-data and 7 meta-data fields: the first metadata field is a pointer pointing to an NVM tuple, the second metadata field is a tuple ID, the third metadata field is a dirty bit, the fourth metadata field is an active bit, the five metadata fields are clock bits, the sixth metadata field is a copy bit, and the seventh metadata field is a CC-Meta field;

2-1, executing transaction processing in DRAM according to the transaction request;

2-2. The persistence stage comprises the following specific processes:

2-2-3, executing the sfence barrier writing instruction to ensure all tuples to be persisted before the last persisted bit LP is persisted;

2-2-4, marking the last persistence bit LP of the last tuple written in the transaction execution process as persistence, and using NVM atomic write operation to persist the tuple header to NVM;

in step two, if the database crashes, a transaction crash recovery process is executed, which specifically includes:

(1) The engine architecture runs a plurality of threads, each thread is responsible for scanning an NVM tuple heap area, and a timestamp ts-commit and a list pending-list are initialized; ts-commit represents the maximum transaction commit timestamp that has occurred so far; pending-list represents a set of tuples that have not been determined to have been committed;

(2) Performing first scanning, wherein the area of the first scanning is the whole NVM tuple stack, and for each NVM tuple, if the last persistent bit LP is set, updating the timestamp ts-commit; if the delete bit or the transaction commit timestamp is equal to 0, then the NVM tuple is placed in a garbage queue;

(3) Performing second scanning, wherein the area of the second scanning is each NVM tuple in the pending list pending-list; at this point, the timestamp ts-commit is the maximum transaction commit timestamp for the scanned region in the first scan; for each NVM tuple, if the transaction commit timestamp of one NVM tuple is less than or equal to the timestamp ts-commit, updating the main index with the NVM tuple; otherwise, indicating that the transaction associated with the NVM tuple crashes during persistence, and marking the NVM tuple as discarded by marking a tuple header corresponding to the NVM tuple as empty and persisting the NVM tuple to the NVM through NVM atomic write operation; simultaneously, putting the NVM tuple into a garbage queue;

2. The high throughput log-less online transaction processing method for non-volatile memory as claimed in claim 1, wherein: the specific process of the execution stage in the step two is as follows:

2-1-1, taking the time of receiving the transaction request of the client as the transaction starting time, and obtaining a timestamp corresponding to the transaction starting time;

2-1-3, reading an NVM tuple, using a CC-Meta field of a concurrency control method to enhance the NVM tuple to construct an entry of Met-Cache, and using a position update index of the Met-Cache entry based on the request tuple;

2-1-4, the engine architecture executes the transaction according to the Met-Cache entry of the request tuple in a concurrent manner; if no conflict exists and the transaction is successfully submitted, directly entering the persistence stage of the step 2-2; if the transaction aborts, proceed to step 2-1-5;

2-1-5, checking whether dirty bits of all Met-Cache entries accessed by the transaction are set; for Met-Cache entries whose dirty bits have been set, the entries are respectively restored from the NVM tuples pointed to by the corresponding NVM tuple pointers.

3. A high throughput log-free online transaction processing method for non-volatile memory according to claim 1 or 2, characterized in that: step two, entering a maintenance phase after the persistence phase; the specific process of the maintenance stage is as follows: when a thread finds that a new version exists, the NVM tuples of the old version are put into a garbage queue.

4. The high throughput log-less online transaction processing method for non-volatile memory according to claim 3, wherein: for the garbage queue, periodically extracting the transaction submission timestamp of the last submitted transaction of all threads by an engine architecture, and taking the minimum value of the obtained transaction submission timestamps as a global minimum timestamp; the NVM tuples in the garbage queue having transaction commit timestamps less than the global minimum timestamp are moved from the garbage queue to the allocator free list.

5. The high throughput log-less online transaction processing method for non-volatile memory as claimed in claim 1, wherein: the engine architecture further comprises transaction private data; the transaction private data and the index are stored in the DRAM.

6. The high throughput log-less online transaction processing method for non-volatile memory as claimed in claim 1, wherein: the NVM metadata is stored with a table pattern and a coarse-grained allocation structure.

7. The high throughput log-less online transaction processing method for non-volatile memory as claimed in claim 1, wherein: the NVM tuple heap contains multiple tuple versions of a logical tuple.

8. The high throughput log-less online transaction processing method for non-volatile memory as claimed in claim 1, wherein: and the Met-Cache manages tuple caching for the corresponding NVM tuple heap in the DRAM.

9. The high throughput log-less online transaction processing method for non-volatile memory as claimed in claim 1, wherein: the engine architecture supports multi-threaded concurrent transactions, and each thread has a private space in the DRAM.

10. A high throughput log-free online transaction processing engine architecture for non-volatile memory comprising NVM metadata, an index, and a plurality of base tables; the method is characterized in that: each base table comprises a tuple heap stored in the NVM, a Met-Cache stored in the DRAM, and an NVM tuple manager of each thread in the base table;

the Met-Cache consists of a plurality of entries; one entry contains meta-data and 7 meta-data fields: the first metadata field is a pointer to the NVM tuple, the second metadata field is a tuple ID, the third metadata field is a dirty bit, the fourth metadata field is an active bit, the five metadata fields are clock bits, the sixth metadata field is a duplicate bit, and the seventh metadata field is a CC-Meta field.