CN112214171B

CN112214171B - SQLite database-oriented non-volatile memory buffer area design method

Info

Publication number: CN112214171B
Application number: CN202011083545.5A
Authority: CN
Inventors: 胡卉芪; 胡耀艺
Original assignee: East China Normal University
Current assignee: East China Normal University
Priority date: 2020-10-12
Filing date: 2020-10-12
Publication date: 2022-08-05
Anticipated expiration: 2040-10-12
Also published as: CN112214171A

Abstract

The invention discloses a design method of a non-volatile memory buffer area facing an SQLite database, which is characterized in that a copy SQLite buffer area method based on a non-volatile memory is adopted, two pieces of same data are maintained in one buffer area, a transaction manager CC-manager for distributing a global time stamp is introduced, and a modified page index Updated-pages index is added, so that the ordering, atomicity and persistence of transaction execution are improved and sent, and the consistency between the non-volatile memory and a database file in a disk is ensured by modifying an index file of a page through written attributes. Compared with the prior art, the invention has the concurrency performance equivalent to the SQLite-WAL mode, improves the transaction throughput by three times, reduces the read-write delay by 40 percent, effectively solves the problem of write amplification in a disk, and provides a buffer area management scheme with low delay and high throughput for the SQLite.

Description

SQLite database-oriented non-volatile memory buffer area design method

Technical Field

The invention relates to the technical field of database systems, in particular to a design method of a non-volatile memory buffer area facing an SQLite database based on application copy in a non-volatile memory.

Background

A non-volatile memory (NVM, or referred to as persistent memory) is a new type of storage device, which has a similar byte addressing capability as a normal memory, far surpasses the read/write performance of a solid-state disk and the data persistence capability that a normal memory does not have, and is considered as a potential next generation storage device. The appearance and development of the novel hardware strongly impact storage systems such as a file system, a cache system and the like, and also have new influence on the architecture and design of the current database system.

Chen S, Gibbons P B, Nath S, and others firstly discuss parameters (cache misses, cache line write backs, words modified) and analysis indexes (total wear, energy, total PCM access latency) on the non-volatile memory to replace DRAM and the related design of the relational database on the non-volatile memory; other work considers the effect of non-volatile memory on relational database storage. van Renen a, Leis V, Kemper a, etc. take a nonvolatile memory as an intermediate layer from the perspective of nonvolatile memory management in a database, and maintain a buffer area in both DRAM and nonvolatile memory, during which DRAM loads data from NVM with cache-line granularity for transfer, and propose a Mini Page data structure that reduces memory occupancy to better utilize the fast random access characteristic of NVM and reduce space occupancy of DRAM.

The invention is based on the research of real nonvolatile memory (equipment is Intel AEP) and combines the design of a buffer module of a database system under the nonvolatile memory. The lightweight relational database SQLite is selected as a prototype system, has the characteristics of small size and easiness in use, and is very wide in practical application due to development efficiency, reliable transaction support and lightweight codes. However, at the same time, compromises in lightweight code libraries have forced it to employ less complex but more costly schemes to provide transactional support, such as serialized transactional execution, physical logging at page granularity, redundant logging, and forced commit policies. These designs impose excessive I/O overhead on the transaction execution. Meanwhile, two designs of default cache and shared cache are used in a SQLite buffer area management module, the former creates a buffer area for each connection, which causes the problems of space waste and time-consuming coordination between caches, and the latter shares a buffer area by a plurality of connections, thereby reducing space occupation and reducing concurrency.

Disclosure of Invention

The invention aims to design a method for designing a SQLite database-oriented non-volatile memory buffer area, which aims at the defects of the prior art, adopts a method for ensuring the atomicity and the persistence of a transaction by using a copy buffer area based on a non-volatile memory, constructs a novel SQLite buffer area SQLite-CC (copy cache) based on the non-volatile memory, ensures the orderliness of transaction execution by using a transaction manager CCmanager for distributing a global timestamp, ensures the atomicity and the persistence of the transaction by using the copy buffer area based on the non-volatile memory, promotes concurrency to a certain extent, and ensures the consistency between the data base files in the non-volatile memory and the data base files in a disk by using an index file of a written attribute modification page. The method makes full use of the hardware characteristic of the nonvolatile memory, maintains two identical data in a buffer area, simultaneously introduces CC-manager for maintaining transaction atomicity, and adds a modified page index Updated-pages index to ensure the consistency between a database file and a cache, so that the concurrency performance equivalent to an SQLite-WAL mode is achieved, the mode transaction throughput is improved by three times compared with a rollback log, the read-write delay is reduced by 40%, the problem of write amplification in a disk is effectively solved, and a low-delay and high-throughput buffer area management scheme is provided for the SQLite.

The purpose of the invention is realized as follows: a design method for a non-volatile memory buffer area facing an SQLite database is characterized in that a copy SQLite buffer area method based on a non-volatile memory is adopted, two pieces of same data are maintained in one buffer area, a transaction manager CC-manager for distributing a global time stamp is introduced, and a modified page index Updated-pages index is added to ensure the consistency between a database file and a cache, so that the ordering, atomicity and persistence of transaction execution are improved and concurrent, and the consistency between the non-volatile memory and the database file in a disk is ensured by modifying an index file of a page through written attributes.

The implementation of the transaction manager CC-manager that assigns global timestamps is as follows:

a1: a global timestamp generation function based on a physical clock distributes a timestamp for each transaction when the transaction starts, and is used for coordinating the precedence relationship between read-write transactions.

A2: and managing and detecting the consistency state between the main buffer area and the copy buffer area, and judging the consistency state of the two buffer areas to ensure the ACID characteristic of the transaction. And provides a function of judging a recovery point.

The copy SQLite buffer based on the nonvolatile memory is implemented as follows:

b1: a copied buffer is additionally added into a buffer maintained by each process and is stored in two hash tables of the same Pcache handle, the data of the two hash tables are the same in a general state, and when a write transaction is carried out, a main buffer takes a write task and an inconsistent state occurs. Management in conjunction with CCmanager guarantees that transactions proceed in order.

B2: the states of the two buffers are recorded in the PCache handle, the copied secondary buffer is used for ensuring the concurrence of the read transaction when the primary buffer is modified, the secondary buffer still keeps the original data and plays the role of an undo log to ensure the atomicity of the transaction, and when the transaction abort occurs or the transaction modification is not completed, the value of the secondary buffer is an accurate value and can be used for restoring the modification in the primary buffer. When the transaction modification is completed but not submitted, namely the primary buffer area is modified and the secondary Cache is not synchronized, the current state is judged through the state check of the CC-manager, the primary buffer area can play the role of the redo log, the modified value of the redo log is synchronized to the secondary buffer area, and the transaction is submitted so as to guarantee the durability of the transaction.

The index of the modified page is implemented as follows:

c1: and one modified page index records the number, offset and other matedata of the modified page in the buffer area, and marks the page out of synchronization with the database file in the disk. Then, the updating is synchronized to the disk by a general delay synchronization mode and the time when the buffer is changed out, so that the consistency of the buffer area in the nonvolatile memory and the database file in the disk is ensured.

Compared with the prior art, the invention has the following beneficial technical effects and remarkable progress:

1) the characteristics of the nonvolatile memory are exerted, the memory is used as the storage of the SQLite cache part, the advantages of two buffer areas are integrated, the interaction between the memory and a disk is reduced, the read-write performance of the database is optimized, the transaction throughput is improved by 3 times compared with a rollback mode, and the transaction delay is reduced by 40%.

2) By means of the persistence characteristic of the nonvolatile memory relative to the memory, the buffer area is used as a carrier, functions of an undo log and a redo log are achieved in a phase-changing manner by copying a double-version mode, log records in a disk are optimized, space overhead and write amplification of the disk are reduced, and database recovery is accelerated.

3) Read-read concurrency and write-once-read concurrency comparable to the SQLite-WAL mode are maintained, and the problem of tail delay existing in the WAL mode is solved, which is one effective attempt of applying the NVM in the database direction.

Drawings

FIG. 1 is a schematic diagram of an SQLite architecture of the present invention;

FIG. 2 is a flow chart of a write process;

FIG. 3 is a schematic diagram of a write process;

FIG. 4 is a flowchart of a reading process;

FIG. 5 is a schematic view of a read process;

FIG. 6 is a bar graph of transaction throughput under different read-write proportional loads;

FIG. 7 is a graph of transaction execution times for different thread counts;

FIG. 8 is a graph of read latency in the execution of a transaction;

FIG. 9 is a diagram of write latency in the execution of a transaction.

Detailed description of the invention

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in detail with reference to the accompanying drawings and specific embodiments. The procedures, conditions, experimental methods and the like for carrying out the present invention are general knowledge and common general knowledge in the art except for the contents specifically mentioned below, and the present invention is not particularly limited.

Example 1

Referring to fig. 1, the SQLite system oriented to the non-volatile memory buffer based on the original SQLite database architecture is divided into three parts, namely a B-tree module and a CC-manager module in the memory, a Pager module and a modified page index module in the non-volatile memory, and database file storage in the disk. The CC-manager module and the B-tree module cooperate to manage the transaction, the B-tree module performs read-write operation and other operations on the data page of the buffer area by accessing the Pager module in the nonvolatile memory, and the modified page index is used for synchronizing the disk and the nonvolatile memory.

Referring to fig. 2, a specific flow of write operations is: when writing operation is normally performed, the SQLite sends a request to the Pager module through the B-tree module, and then modifies the data through the buffer. In SQLite-CC, firstly, the current two buffers are ensured to be in a synchronous state by the judgment of sqliteCCheckState () of a CC-manager, namely, the write authority can be obtained, and the write timestamp is distributed to the two buffers through sqliteCCGTTimestamp ().

Referring to fig. 3, during the modification process, two asynchronous buffer periods are formed in the buffer, the two buffers are uniformly managed by one Pcache handle, the hash table corresponding to the main buffer is in a modified state, and the modified data page is also regarded as a redo log data page. The hash table corresponding to the copy buffer keeps the original value unchanged, so the data page corresponding to the modified part in the main buffer can be regarded as a rollback log data page. Namely, the normal modification process is a process from synchronization to asynchronization and then to synchronization, and the atomicity is ensured through a transaction object type interface of an Intel nonvolatile memory management library in the process. Due to the persistence of the nonvolatile memory, data loss does not need to be worried about, the modified page index can be recorded after the transaction sqlite CCommit () is synchronized and submitted to ensure the consistency with the database file after the pages of the buffer are swapped out, the modified value is not required to be swiftly printed to the database file in the disk, and the disk I/O is reduced. When a buffer swap-in and swap-out occurs, it is detected whether the swap-out page is marked in the modified page index, if so, the contents in the disk are modified, and if not, the contents are directly discarded.

Referring to fig. 4, the specific flow of the read operation is as follows: in SQLite-CC, two different situations are faced, one is that when two buffers are in synchronous state, the reading process is the same as the original SQLite; alternatively, the current buffer status is not synchronized, i.e., there is a write operation in progress. When the transaction starts, the current state is judged through the CC-manager, a timestamp is distributed, and then the two conditions are divided into two conditions according to the sequence of reading and writing the transaction: when a read is after a write transaction, it will be blocked waiting because it needs to read through the new contents in the main buffer, reading the modified value; otherwise, reading through the copied buffer area, reading the old value, and keeping the same flow.

Referring to FIG. 5, if there is a write operation at time T2, when the write transaction begins, the new and old read transactions are divided, there is no effect on the thread that has previously performed the completed read operation, and an ongoing operation such as read transaction 3, read transaction 6 will be treated as an old read transaction, reading the value from the copy buffer, and then when the old transaction threads are all completed, the two buffers are synchronized, completing and committing the operation. Read transaction 2, which begins executing after time T2, read transaction 4 will read the latest value from the main buffer after waiting for the write transaction in the process to commit to completion.

And (3) recovering the specific flow of operation: when the crash occurs, the original SQLite has its own method to perform fault detection during the restart, and when it runs in the "delete log" mode, if there is a hot log, it indicates that the SQLite is in a crash state before the restart, where the hot log (i.e. the log that has been recorded before the crash but still exists because the transaction was not committed) can be used to restore the database consistency. When the original SQLite runs in the WAL log mode, the part which does not submit the record in the WAL log represents the transaction which is in progress during crash, and the part is deleted for modification. The SQLite-CC fault detection mode is different from the former two modes, the atomicity and the persistence of the transaction are ensured through the buffer, and due to the persistence of the nonvolatile memory, the fault recovery under any condition can acquire the state of the buffer before the downtime. If data needing to be recovered exists, two buffers in the SQLite-CC are certainly not synchronous, and meanwhile, because at most one write transaction runs on the same database at any time, one of the two write transactions is accurate, sqliteCCrecharger () is executed, firstly, the starting state sqliteCCheckstate () of the secondary buffer is detected, the state point needing to be recovered can be judged, and then corresponding rollback or redo operation is carried out, so that the database reaches a consistency state. In addition, if the downtime occurs when the page swapped out of the buffer is synchronized to the database file, whether the page in the buffer needs to be copied to the database file again can be judged by detecting whether the page is recorded in the modified page index and comparing the values of the swapped out page and the corresponding page in the database file, so that the consistency of the data is ensured.

Referring to FIG. 6, comparing the transaction throughputs of the three modes (Delete, WAL, SQLite-CC) under the load a read-write proportion of 50%/50%, the load b read-write proportion of 95%/5% and the load c read-write proportion of 100%/0%, it can be seen that the throughputs of the three modes all show an upward trend along with the decrease of the specific gravity of the write operation, and under the same load, the performance of the SQLite-CC is slightly better than that of the SQLite-WAL, and the SQLite-Journal shows the worst performance.

Referring to fig. 7, the running time test of the three modes by using single thread, double thread, four threads and eight threads is shown, and data modification is performed in a single-write manner to ensure the thread safety by SQLite. Multithreading is not a good way to promote concurrency for SQLite. Meanwhile, it can be seen that in the load with a large write operation, the Journal mode has poor performance, while the WAL mode adopts the redo log, and only one disk refreshing for recording a new value is needed, so that the scene support is good, and the SQLite-CC is equivalent to the execution time of the SQLite-CC. In addition, it is obvious that the higher the reading ratio is, the smaller the influence of the SQLite single writing is, and the difference between the SQLite single writing and the SQLite single writing is gradually reduced.

Referring to fig. 8, the read delay time percentage points of the three modes are shown, the trends of the three modes are approximately the same, and the SQLite-CC performs best, which is improved by 40% on average.

Referring to fig. 9, percentage points of write delay time of three modes are shown, and the trends of the three modes are basically consistent, wherein when a large number of write operations are performed in the Journal mode, performance degradation is obvious due to more IO operations, and the SQLite-WAL and the SQLite-CC perform better.

The present invention is not limited to the above embodiments, and variations and advantages that can be realized by those skilled in the art are included in the present invention without departing from the spirit and scope of the inventive concept, and the scope of the present invention is defined by the appended claims.

Claims

1. A design method for a non-volatile memory buffer area facing an SQLite database is characterized in that two identical data are maintained in a copy SQLite-CC buffer area of a non-volatile memory, a transaction manager CC-manager for distributing a global time stamp is introduced, a modified page index Updated-pages index is added, the sequence, the atomicity and the persistence of transaction execution are promoted and sent, the consistency between the non-volatile memory and a database file in a disk is ensured by modifying an index file of a page through written attributes, and the transaction manager CC-manager realizes the distribution of the global time stamp according to the following steps:

a1: based on the global timestamp generation function of the physical clock, when each transaction starts, a timestamp is distributed to each transaction and is used for coordinating the precedence relationship between the read-write transactions;

a2: managing and detecting the consistency state between the main buffer area and the copy buffer area, ensuring the ACID characteristic of the transaction by judging the consistency state of the two buffer areas, and providing the judgment function of a recovery point;

the copy SQLite-CC buffer area based on the nonvolatile memory is realized by the following steps:

b1: additionally adding a copied buffer area in a buffer area maintained by each process, storing the copied buffer area in two hash tables of the same Pcache handle, wherein the data of the two hash tables are the same in a general state, and when a write transaction is carried out, the main buffer area bears a write task; when an inconsistent state occurs, the ordered operation of the transaction is ensured by combining the management of the CC-management;

b2: recording the states of two buffers in a PCache handle, ensuring that read transactions are concurrent when a copied secondary buffer modifies a main buffer, and simultaneously playing the role of an undo log to ensure the atomicity of the transactions because the secondary buffer still keeps original data; when the transaction modification is completed but not submitted, namely the main buffer area is modified and the auxiliary buffer area is not synchronized, the current state is judged by the state check of the CC-manager, the main buffer area can play the role of a redo log, the modified value of the redo log is synchronized to the auxiliary buffer area, and the transaction is submitted so as to ensure the durability of the transaction;

the index of the modified page is realized according to the following steps:

c1: recording metadata such as the number and offset of modified pages in the buffer area, marking out pages which are not synchronous with the database file in the disk, synchronizing the update to the disk in a general delay synchronization mode and at the moment of buffer exchange, and thus ensuring the consistency of the buffer area in the nonvolatile memory and the database file in the disk.