KR102287774B1 - Method for Processing Data in Database Based on Log Structured Merge Tree Using Non-Volatile Memory - Google Patents

Abstract

According to the present embodiments, data exceeding a predetermined capacity of the volatile memory is stored in the non-volatile memory, and the flush operation and the compaction operation are performed through the list structure and the persistence buffer of the non-volatile memory, so that the write delay while maintaining data persistence and provides a database that can minimize read delay.

Description

{Method for Processing Data in Database Based on Log Structured Merge Tree Using Non-Volatile Memory}

The technical field to which the present invention pertains relates to a log structure merge tree-based database using a non-volatile memory and a data processing method thereof.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

Key-value based databases are useful for handling unstructured data such as sensor data and social network data. A key-value based database mainly uses a log structured merge tree.

Log Structured Merge Tree (LSM-Tree) is designed for workloads that perform continuous write operations. The LSM-Tree structure consists of one in-memory data structure and an append-type data structure for storage in multiple blocks (eg, disk, etc.).

LSM-Tree efficiently performs inserts and modifications that occur frequently in key-value databases. It is a write friendly structure in which data is first stored in a log format, and merging such as sorting data in the log and processing of correction operations is postponed. However, merge operations that occur later cause write amplification and affect system performance and storage lifespan.

LSM-Tree writes data sequentially without writing data in random order. When searching for data, it is impossible to know where the given data is in the tree, so to find the data, it must be searched sequentially from the upper level. All files at all levels must be read, even if there is no data on the disk.

US Patent Publication US 2017-0344619 (2017.11.30.) Korean Patent Publication No. KR 10-2016-0121819 (2016.10.21.) US Patent Publication US 2018-0121121 (2018.05.03.)

In embodiments of the present invention, a database including a volatile memory and a non-volatile memory stores data exceeding a predetermined capacity of the volatile memory in the non-volatile memory, and performs a flush operation and A primary object of the invention is to minimize write delay and read delay while maintaining data persistence by performing a compaction operation.

Other objects not specified in the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

According to one aspect of this embodiment, in a data processing method of a database, storing data in a volatile memory of the database; generating a list structure in which a plurality of nodes are connected to a nonvolatile memory of the database; There is provided a data processing method of a database including performing a flush operation in a manner of storing data in the list structure.

The database stores data in a key-value format, and the list structure may be a skip list with multiple next pointers.

The method may include performing a compaction operation in such a way that the database does not write data, but creates a new list structure and points the new list structure to a key-value assigned to a node of the existing list structure. .

In the performing the flush operation, a key-value is sequentially copied to a persistent buffer of the non-volatile memory, and the persistence buffer prevents random access of the key-value assigned to the node, and the The list structure may point to an offset of the persistence buffer corresponding to the list structure.

The performing of the compaction operation may include creating a new list structure in the non-volatile memory and pointing the new list structure to an offset of a persistence buffer corresponding to the previous list structure.

When the data stored in the nonvolatile memory of the database exceeds a preset capacity range, the data may be ejected to the block drive of the database according to a tiering policy.

The staging policy includes (i) a first tiering policy that selects data at a specific level and stores it in the block drive, (ii) a second tiering policy that selects data with old data access and stores it in the block drive policy, (iii) a third staging policy that stores all data in the non-volatile memory, or a combination thereof.

When a system error occurs in the database, the method may include sequentially recovering data through a result of inquiring about data pointed to by the list structure stored in a non-volatile memory of the database.

According to another aspect of the present embodiment, in a database including a processor, a volatile memory, and a non-volatile memory, data is stored in the volatile memory, a list structure in which a plurality of nodes are connected is created in the non-volatile memory, and the There is provided a database characterized in that the flush operation is performed by storing data in the list structure.

According to another aspect of this embodiment, as a computer program for data processing recorded in a non-transitory computer readable medium including computer program instructions executable by a processor, the computer program instructions are stored in a database. When executed by at least one processor, storing data in a volatile memory of the database, and generating a list structure in which a plurality of nodes are connected to a non-volatile memory of the database and storing the data in the list structure A computer program for performing operations including performing a flush operation in a store manner is provided.

As described above, according to the embodiments of the present invention, a database including a volatile memory and a non-volatile memory stores data exceeding a predetermined capacity of the volatile memory in the non-volatile memory, and a list structure of the non-volatile memory And by performing a flush operation and a compaction operation through the persistence buffer, there is an effect of minimizing write delay and read delay while maintaining data persistence.

Even if it is an effect not explicitly mentioned herein, the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as if they were described in the specification of the present invention.

1 is a diagram illustrating an existing log structure merge tree-based database.
2 is a diagram exemplifying that a data compaction operation is performed by an existing log structure merge tree-based database.
3 is a block diagram illustrating a database according to an embodiment of the present invention.
4 is a diagram illustrating an internal data structure of a database according to an embodiment of the present invention.
5 is a flowchart illustrating a data processing method of a database according to another embodiment of the present invention.
6 is a diagram illustrating a skip list generated by a database in a nonvolatile memory according to another embodiment of the present invention.
7 is a diagram illustrating that the database performs compaction on a skip list according to another embodiment of the present invention.
8 is a diagram illustrating sequential copying of a database through a persistence buffer according to another embodiment of the present invention.
9 is a diagram illustrating that a database performs byte addressing compaction through a persistence buffer according to another embodiment of the present invention.
10 to 12 show simulation results performed according to embodiments of the present invention.

Hereinafter, in the description of the present invention, if it is determined that the subject matter of the present invention may be unnecessarily obscured as it is obvious to those skilled in the art with respect to related known functions, the detailed description thereof will be omitted, and some embodiments of the present invention will be described. It will be described in detail with reference to exemplary drawings.

1 is a diagram illustrating an existing log structure merge tree (LSM-Tree)-based database, and FIG. 2 is a diagram illustrating a data compaction operation of an existing log structure merge tree-based database. .

Representative databases using LSM-Tree include LevelDB and RocksDB.

LSM-Tree stores data in the memory area first when an insert operation is performed. When data is accumulated up to a certain amount of memory, the contents of the memory are flushed to the disk. The data to be flushed is recorded by merge-sorting with the existing data stored on the disk. When each level of disk area crosses a threshold, a merge sort is executed to create a lower level.

LSM-Tree-based database stores data in key-value format. When a data insertion operation request comes in to the LSM-Tree-based database, the log is first written to the log file before data is written to the memory. After writing the log, the data is stored in the memtable in the memory area. If the write request continues and data is written to the memtable up to a certain capacity, the memtable is changed to an immutable memtable (read-only memtable) that cannot be changed. When the immutable memtable becomes full, a flush occurs to the block (disk) area.

When a flush operation is performed, the memtable files are sorted according to the key order and changed to a SST (Stored String Table) file. An SST file has a plurality of blocks. Examples of the block include a data block for storing data, an index block for indexing the position of the data block, and a footer block for processing the position of the index block.

The SST file is updated through compaction in the disk area. Once created, the SST file may not disappear. As the SST file residing at the lower level, data older than the SST file at the upper level may be located.

If a problem such as a system error or power cut occurs during transaction execution, data that has not yet been reflected in the disk and remaining in the buffer is lost. When the database performs recovery after the system is rebooted, a log is used to record what update operation the transaction performed. As a logging method, there is a write-ahead-logging (WAL) rule. WAL is a rule to write related logs to log files before data changed due to a transaction is written to disk.

The LSM-Tree-based database executes two commands. One is a flush command that goes from memory to disk, and the other is a compaction command that adjusts the levels of the disk.

When the flush command is executed, the immutable memtable is changed to a single SST file. When a large amount of data is input at once, the flush rate is intentionally adjusted to balance the flush rate and maintain the capacity limit of the level of the SST file. This intended delay is called 'Write Stall'. Stacked Write Stalls are exemplified in Table 1.

Each level of LSM-Tree is distinguished by its characteristics. Compaction costs and disk writes are concentrated at a specific level. The hierarchical storage structure causes a gradual accumulation of data from a higher level to a lower level. The lifetime of an SST file means the number of times it performs compaction while it exists at that level. If the SST file is not deleted at a certain level during compaction, the lifespan of the corresponding SST file is high. Table 2 exemplifies the lifetime of the SST file, the number of compaction files, the ratio of compaction files, and the amount of writing during compaction.

Looking at the results of each level of performing the compaction command, it indicates that _{SST files of higher levels (eg, L 0} to L ₃ ) are created and deleted during a short time during which data compaction is performed. Looking at the number of compaction files and the ratio of compaction files, it indicates that the size of the compaction file is not small due to the hierarchical structure of LSM-Tree.

The amount of writes at the upper level with spare resources during compaction seems insignificant, but disk I/O is unavoidable. Even though quite a lot of data is input, the number and write amount of compaction files of the _{lower level L 4} are smaller than those _{of L 3 .}

The database according to the present embodiment reduces the disk I/O wasted at the upper level through NVM by focusing on the lifetime of the SST file having a high value at the lower level such as _{L 4 .} NVM uses byte addressing to solve write amplification and enables persistent compaction.

3 is a block diagram illustrating a database according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating an internal data structure of a database according to an embodiment of the present invention.

As shown in FIG. 3 , the database 10 includes a processor 100 , a volatile memory 200 , and a nonvolatile memory 300 . The database 10 may omit some of the various components exemplarily illustrated in FIG. 3 or may additionally include other components. For example, the database 10 may further include a block device 400 according to a staging policy.

The database 10 is a device for processing and storing data. The database 10 can store and read data in a key-value format. The key-value processing command may be defined as (SET K, V), (DEL K, V), and the like.

The processor 100 transmits predefined commands to the volatile memory 200 , the non-volatile memory 300 , and the block device 400 to control the flow of various signals and data.

The volatile memory 200 is a memory that requires power supply to continuously maintain stored information. For example, the volatile memory 300 includes a dynamic random access memory (DRAM).

The non-volatile memory 300 is a memory that continuously maintains stored information even when power is not supplied. The non-volatile memory 300 may include a list structure 310 . The list structure 310 has a head and a rear, and has an address of a key and an address of a value, respectively. List structure 310 may be implemented as a skip list with multiple next pointers. The skip list has a key length, a key, a value length, and a value for each node. The non-volatile memory 300 may include a persistence buffer 320 .

The block device 400 is a storage medium that can be accessed randomly in units of blocks. For example, the block device 400 includes a hard disk drive (HDD), a solid state drive (SSD), and the like.

The database 10 primarily stores data in the volatile memory 200 . When the stored data exceeds a preset capacity, some data is secondaryly stored in the non-volatile memory 300 . The database 10 may generate a list structure 310 in which a plurality of nodes are connected to the non-volatile memory 300 , and may perform a flush operation by storing data in the list structure 310 .

5 is a flowchart illustrating a data processing method of a database according to another embodiment of the present invention.

In step S210, the database stores data in the volatile memory.

In step S220, the database performs a flush operation on the nonvolatile memory. A flush is an operation of copying from a first storage to a second storage. For example, data is copied from volatile memory to non-volatile memory.

In step S230, the database performs a compaction operation. Compaction is an operation of lowering data of a specific level to a lower level when data is filled up to a threshold of a specific level as a merging process.

In step S240, the database performs an operation of emitting data to the block drive according to the policy.

In step S250, the database performs an operation of recovering data when an error occurs.

6 is a diagram illustrating a skip list generated by a database in a nonvolatile memory according to another embodiment of the present invention.

Non-Volatile Memory (NVM) corresponds to an intermediate level of flexibility, and the skip list can replace the existing block file format (ex. SST file).

Non-Volatile Memory (NVM) is capable of byte addressability. Examples of nonvolatile memory capable of byte addressing include Spin-Transfer Torque Magnetic Random Access Memory (STT-MRAM) and Phase-Change Memory (PCM).

In order for NVM to operate consistently in the database system, the database system must use the 'cflush' and 'mfence' commands. The 'cflush' command flushes the cache line to memory and ensures that the data is completely saved to the NVM. The 'mfence' instruction is a memory barrier instruction that protects the processor from reordering of instructions.

The Persistent Memory Development Kit (PMDK) API provides a key-value based database using NVM.

7 is a diagram illustrating that the database performs compaction on a skip list according to another embodiment of the present invention.

The database does not write data, but creates a new list structure and performs the compaction operation in such a way that the new list structure points to the key-value assigned to the node of the existing list structure.

8 is a diagram illustrating sequential copying of a database through a persistence buffer according to another embodiment of the present invention.

Table 3 shows the algorithm related to the flush operation of the database.

The flush operation sequentially copies key-values to the persistence buffer of non-volatile memory, and the persistence buffer prevents random access of key-values assigned to nodes. The list structure points to the offset of the persistence buffer corresponding to the list structure.

9 is a diagram illustrating that a database performs byte addressing compaction through a persistence buffer according to another embodiment of the present invention.

Table 4 shows the algorithm for the byte addressing compaction operation of the database.

The compaction operation creates a new list structure in the non-volatile memory and points the new list structure to the offset of the persistence buffer corresponding to the old list structure. When performing a compaction operation, the persistence buffer does not write to the key length, key, value length, and value. For NVM, use the 'cflush' and 'mfence' commands to ensure persistence. Unlike SST files, persistence buffers do not contain index blocks and footer blocks. Improves data retrieval performance through offsetting of persistence buffers. Before compaction, iterators for the lowest level of the skip list are created, and the iterators are merged in the byte addressing compaction process.

The database performs storage tiering. The database contains block drives. When the data stored in the non-volatile memory exceeds a preset capacity range, the database ejects it to the block drive of the database according to a tiering policy.

The tiering policy consists of (i) a first tiering policy that selects data at a specific level and stores it in a block drive (Leveled Tiering), (ii) a second stage that selects data with old data access and stores it in a block drive It may be set to a policy (LRU Tiering), (iii) a third tiering policy to store all data in non-volatile memory (No Tiering), or a combination thereof. The specific level may be calculated and set in a statistical manner according to an implemented design.

The algorithm for the operation of restoring the database is shown in Table 5.

When a system error occurs in the database, the data is sequentially restored through the result of inquiring the data pointed to by the list structure stored in the non-volatile memory of the database. A persistence pointer is obtained from the NVM and the skip list is restored using the persistence pointer. Find a node in the skip list, obtain metadata (ID), iterator for ID, omit already input data, and consider remaining byte addressing compaction. Recovers the meta data of the skip list and inserts the meta data into the list.

10 to 12 show simulation results performed according to embodiments of the present invention.

As shown in FIG. 10 , it can be seen that the performance of the database according to the present embodiment is improved in terms of write delay and read delay.

11 , it can be seen that the performance is improved in terms of write delay and read delay in the order of the third tiering policy (No Tiering), the second tiering policy (LRU Tiering), and the first tiering policy (Leveled Tiering). there is.

Referring to FIG. 12 , it can be seen that the performance of the database TLSM according to the present embodiment is improved in terms of write delay and compact write amount.

Although the components included in the database are illustrated separately in FIG. 3 , a plurality of components may be combined with each other and implemented as at least one module. The components are connected to a communication path connecting a software module or a hardware module inside the device to operate organically with each other. These components communicate using one or more communication buses or signal lines.

The database may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or special-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a system on chip (SoC) including one or more processors and controllers.

The database may be mounted in the form of software, hardware, or a combination thereof on a computing device provided with hardware elements. A computing device includes all or part of a communication device such as a communication modem for performing communication with various devices or a wired/wireless communication network, a memory for storing data for executing a program, and a microprocessor for executing an operation and command by executing the program. It can mean a device.

Although it is described that each process is sequentially executed in FIG. 5, it is only illustratively described, and those skilled in the art change the order described in FIG. 5 within the range not departing from the essential characteristics of the embodiment of the present invention Alternatively, various modifications and variations may be applied by executing one or more processes in parallel or adding other processes.

The operations according to the present embodiments may be implemented in the form of program instructions that can be performed through various computer means and recorded in a computer-readable medium. Computer-readable media refers to any medium that participates in providing instructions to a processor for execution. Computer-readable media may include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. A computer program may be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment may be easily inferred by programmers in the technical field to which the present embodiment pertains.

The present embodiments are for explaining the technical idea of the present embodiment, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present embodiment.

10: database 100: processor
200: volatile memory 300: non-volatile memory
310: list structure 320: persistence buffer
400: block device

Claims (16)

In the data processing method of the database,
storing data in a volatile memory of the database; and
generating a list structure in which a plurality of nodes are connected to a non-volatile memory of the database and performing a flush operation in such a way that the data is stored in the list structure,
The database stores data in a key-value format,
The data processing method of a database, characterized in that the list structure is a skip list having a plurality of next pointers. delete According to claim 1,
and performing a compaction operation in such a way that the database does not write data, but creates a new list structure and points the new list structure to a key-value assigned to a node of the existing list structure. How to process data in the database. 4. The method of claim 3,
The step of performing the flush operation includes:
sequentially copying key-values to a persistent buffer of the non-volatile memory;
The persistence buffer prevents random access of the key-value assigned to the node,
The data processing method of the database, characterized in that the list structure points to an offset of a persistence buffer corresponding to the list structure. 4. The method of claim 3,
The step of performing the compaction operation is,
Creating a new list structure in the non-volatile memory and pointing the new list structure to an offset of a persistence buffer corresponding to the previous list structure. According to claim 1,
and when the data stored in the non-volatile memory of the database exceeds a preset capacity range, ejecting it to the block drive of the database according to a tiering policy. of data processing methods. 7. The method of claim 6,
The tiering policy is
(i) a first staging policy that selects data at a certain level and stores it in the block drive, (ii) a second tiering policy that selects data with outdated data access and stores it in the block drive, (iii) all A data processing method of a database, characterized in that it is set as a third step policy for storing data in the non-volatile memory, or a combination thereof. According to claim 1,
and, when a system error occurs in the database, sequentially recovering data through a result of inquiring the data pointed to by the list structure stored in the non-volatile memory of the database. processing method. A database comprising a processor, a volatile memory, and a non-volatile memory, comprising:
store data in the volatile memory;
generating a list structure in which a plurality of nodes are connected to the non-volatile memory and performing a flush operation in a manner that stores the data in the list structure;
The database stores data in a key-value format,
wherein the list structure is a skip list having a plurality of next pointers. delete 10. The method of claim 9,
The database is characterized in that the compaction operation is performed in such a way that the database does not write data, but creates a new list structure and points the new list structure to a key-value assigned to a node of the existing list structure. . 12. The method of claim 11,
The flush operation is
sequentially copying key-values to the persistence buffer of the non-volatile memory;
The persistence buffer prevents random access of the key-value assigned to the node,
The list structure points to an offset of a persistence buffer corresponding to the list structure. 12. The method of claim 11,
The compaction operation is
and creating a new list structure in the non-volatile memory and pointing the new list structure to an offset of a persistence buffer corresponding to the previous list structure. 10. The method of claim 9,
The database includes a block drive,
The database, characterized in that when the data stored in the non-volatile memory exceeds a preset capacity range, the database is characterized in that the ejection (Eviction) to the block drive of the database according to a tiering policy (Tiering Policy). 15. The method of claim 14,
The tiering policy is
(i) a first staging policy that selects data at a certain level and stores it in the block drive, (ii) a second tiering policy that selects data with outdated data access and stores it in the block drive, (iii) all A database, characterized in that it is set as a third staging policy for storing data in the non-volatile memory, or a combination thereof. 10. The method of claim 9,
When a system error occurs in the database, data is sequentially restored through a result of inquiring the data pointed to by the list structure stored in the non-volatile memory of the database.

Images