CN113535711B

CN113535711B - Hierarchical index data management method and system based on LSM-Tree

Info

Publication number: CN113535711B
Application number: CN202110316895.XA
Authority: CN
Inventors: 李永坤; 左泽; 刘朕; 许胤龙
Original assignee: University of Science and Technology of China USTC
Current assignee: University of Science and Technology of China USTC
Priority date: 2021-03-23
Filing date: 2021-03-23
Publication date: 2024-05-17
Anticipated expiration: 2041-03-23
Also published as: CN113535711A

Abstract

The invention relates to a hierarchical index data management method and system based on LSM-Tree, wherein the method comprises the following steps: step S1: separating index blocks and data blocks in an LRU linked list block cache in the LSM-Tree, and establishing independent index block data; step S2: according to the index block data, a jump table structure is established between indexes of different layers in the LSM-Tree; step S3: and (3) utilizing a skip list structure to realize hierarchical query on the index of the LSM-Tree. The method provided by the invention improves the reading performance of a key value system of a database adopting an LSM-Tree data structure by a data management mode of layering indexes, optimizes an index inquiry search path and shortens the response time of a reading request.

Description

Hierarchical index data management method and system based on LSM-Tree

Technical Field

The invention relates to the technical field of databases, in particular to a hierarchical index data management method and system based on LSM-Tree.

Background

The LSM-Tree (Log Structured MERGE TREES) data organization mode is applied to various databases, such as a LevelDB, an HBase and the like, and compared with a B+ Tree or other index storage implementation mode, the LSM-Tree provides better writing performance, mainly realizes random reading and writing of massive Key values (Key-Value) into memory random reading and writing, sequential disc brushing and periodical merging, fully utilizes the characteristic that the sequential writing performance of a magnetic disc is higher than that of random writing, and greatly improves the data writing capability.

However, the reading efficiency of the LSM-Tree is not high, when the data of the designated Key (Key) needs to be read, the data is searched from MemTable and Immutable MemTable in the memory, if the data is not found, the data is continuously searched from the Level 0 layer, if the data is not found, the data of the Key is searched from the SSTable file of the higher layer, and if the data is failed to be searched, the data of the Key does not exist in the whole LSM-Tree. If the data of this Key is found anywhere in the middle, then the data found along this path is up to date.

The Key value range of SSTable files of each layer is not repeated, so that only one SSTable file needs to be searched to determine whether the data of the designated Key exists in the layer. The process of searching the Key value in each layer SSTable is relatively independent, so that multiple layers need to be traversed when the designated Key is inquired, multiple memory accesses can be caused, and LSM-Tree reading performance is affected.

Therefore, how to improve the reading performance of the key value system of the database adopting the LSM-Tree data structure, optimize the index inquiry search path and shorten the response time of the reading request becomes a urgent problem to be solved.

Disclosure of Invention

In order to solve the technical problems, the invention provides a hierarchical index data management method and system based on LSM-Tree.

The technical scheme of the invention is as follows: a hierarchical index data management method based on LSM-Tree comprises the following steps:

step S1: separating index blocks and data blocks in an LRU linked list block cache in the LSM-Tree, and establishing independent index block data;

Step S2: according to the index block data, a jump table structure is established among index blocks of different layers in the LSM-Tree;

step S3: and using the jump table structure to realize hierarchical query on the LSM-Tree index.

Compared with the prior art, the invention has the following advantages:

The invention discloses a hierarchical index data management method based on LSM-Tree, which adopts the operation steps of separating and storing index blocks and data blocks, designing a memory index data structure, designing a hierarchical acceleration query strategy and maintaining the memory index structure; when processing the read request, the index structure with fine granularity and the cross-layer acceleration strategy are adopted, compared with the original LSM-Tree index data management mode, the index inquiry search path is optimized on the premise of ensuring the write performance of the key value system, the average response time of the read request is shortened, and the read performance of the key value system is improved.

Drawings

FIG. 1 is a flowchart of a hierarchical index data management method based on LSM-Tree in an embodiment of the invention;

FIG. 2 is a schematic diagram illustrating a data distribution structure of LevelDB according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a flow of LevelDB reading data according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a data structure of a LevelDB conventional data cache according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a modified data cache data structure of LevelDB according to an embodiment of the present invention;

fig. 6 is a block diagram of step S2 in a hierarchical index data management method based on LSM-Tree in an embodiment of the present invention: establishing a flow chart of a skip list structure between index blocks of different layers in the LSM-Tree according to the index block data;

FIG. 7 is a flowchart of updating a global index at the L0 layer according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a process of searching L1-Lk by using each interlayer jump table in an embodiment of the present invention;

FIG. 9 is an example of a layered acceleration strategy in an embodiment of the present invention;

FIG. 10 is a flowchart of another LSM-Tree based hierarchical index data management method in accordance with an embodiment of the present invention;

FIG. 11 is a block diagram of a target tracking system for intelligently ordering candidate boxes in accordance with an embodiment of the present invention;

FIG. 12 is a block diagram of another intelligent ordering candidate block target tracking system in accordance with an embodiment of the invention.

Detailed Description

The invention provides a hierarchical index data management method based on LSM-Tree, which improves the reading performance of a key value system of a database adopting an LSM-Tree data structure by a data management mode of layering indexes, optimizes an index inquiry search path and shortens the response time of a reading request.

The present invention will be further described in detail below with reference to the accompanying drawings by way of specific embodiments in order to make the objects, technical solutions and advantages of the present invention more apparent.

Example 1

As shown in fig. 1, the hierarchical index data management method based on LSM-Tree provided by the embodiment of the present invention includes the following steps:

step S2: according to the index block data, a jump table structure is established between indexes of different layers in the LSM-Tree;

step S3: and (3) utilizing a skip list structure to realize hierarchical query on the index of the LSM-Tree.

Taking LevelDB as an example, the LSM-Tree has three file forms, the first is Memtable in memory, which normally receives a write request. When Memtable exceeds the threshold, it is frozen and becomes the second file, i.e., the non-modifiable Immutable Memtable stored in memory. When Immutable Memtable is written to disk, it becomes the third file, SSTable (Sorted String Table). SSTable is divided into L0-Lk layers, when the sum of the sizes of the L0 layer files exceeds a threshold value, the L0 layer files are selected to be written into the next L1 layer, namely, the L0 layer files and the L1 layer files are combined (Compaction). When the sum of the L1 layer file sizes exceeds a threshold, the L1 layer file is selected to be combined with the L2 layer file of the next layer. And so on until merging into the bottom-most Lk layer. After merging occurs, sstables on each of the L1 through Lk layers are globally ordered on keys, while the L0 layer has overlapping Key value ranges for each SSTable of that layer, since no merging operation occurs. As shown in fig. 2, the Key value of the first SSTable in the L0 layer is from 1 to 38, and the Key value of the second SSTable is from 9 to 67, and the Key value ranges are overlapped. Key value ranges of SSTable in the L1-Lk layers are not overlapped with each other and are globally ordered.

As shown in fig. 3, a read flow of LevelDB is illustrated. When reading data, levelDB first looks at Memtable in memory. If Memtable contains Key and Value corresponding to Key, returning Value; if Key is not read at Memtable, then it is next read to Immutable Memtable, which is also in memory. If Key is not read, the process continues to search in SSTable stored in disk. SSTable numbers are large and are divided into a plurality of layers, each block in the disk represents one SSTable file, wherein the L0 layer has 4 SSTable files at most, and index ranges in the files may have overlay intersections, wherein the Lk (k > 0) layer has a plurality of SSTable files, and the index ranges in the files do not overlay intersections. Firstly, searching from the files belonging to L0, returning a corresponding Value if the Value is found, and searching from the files in L1 if the Value is not found, and repeating the steps until the Value corresponding to the Key is found in a layer of SSTable files, wherein the speed is very slow because the searching process needs to perform multiple disk I/O operations.

Within the LSM-Tree key system is an LRU (LEAST RECENTLY Used ) linked list Block Cache (Block Cache), which is dedicated to caching blocks that are most recently accessed. As shown in fig. 4, the index Block and the data Block are stored in the Block Cache together without distinction. The unified use of global LRU algorithm for management easily causes stateless problems for the index block. Therefore, in order to increase the searching speed, the embodiment of the invention separates the index block and the data block.

In one embodiment, step S1 described above: separating the index block and the data block in the LRU linked list block cache in the LSM-Tree to establish independent index block data, wherein the method specifically comprises the following steps:

And separating the index blocks from the chain table block cache, and establishing independent index block data, wherein each index block data stores the key value of the index block of the layer, the next index block of the layer and the index block of the lower layer.

The embodiment of the invention adopts a group of data structures to separate the index Block from the Block Cache. As shown in FIG. 5, the index block is isolated from the data block, and different doubly linked lists are used to measure the heat of two different data. Each index block data in the index blocks is used as a minimum management unit and is independently managed. Each index block occupies 12 bytes, and stores the Key value Key of the index block of the layer, the next index block of the layer and the index block of the lower layer.

As shown in fig. 6, in one embodiment, step S2 described above: according to the index block data, a jump table structure is established among the index blocks of different layers in the LSM-Tree, and the jump table structure specifically comprises the following steps:

Step S21: the index block in the L0 layer in the LSM-Tree is searched for the heat degree, and an index ordering linked list of the latest heat degree is constructed;

Because the Key value ranges of SSTable in the L0 layer are overlapped, the embodiment of the invention rescans the overlapped Key values and updates the time stamp according to the search heat of the Key values of the index blocks. The process is shown in FIG. 7, wherein the L0 layer consists of A, B and 3 SSTable of C, and the L0 layer is an index block. In order to obtain the latest index block linked list, 3 SSTable files are scanned according to the sequence of the time stamps, and an index sequencing linked list of the latest search heat of the layer is constructed. Wherein the index block C is the coldest in terms of time stamps, its index ranges from 8 to 13, and the L0 global index is first all ordered by the index of the index block C. Next, the global index is updated again according to the index block B. The index range of the index block B is 5 to 10, and since the ranges of 8, 9, and 10 in the Key values of the index block B and the file index block C overlap, the global indexes 8, 9, and 10 are updated with time stamps such that the time stamps of 8, 9, and 10 are updated to the time stamps of the index block B. Finally, the global L0 index is updated according to the latest index block A. Wherein, 5 and 6 in the Key values of the index block a and the index block B overlap, so the time stamps of 5 and 6 in the global index are updated to the time stamp of the index block a. So far, the global index of the L0 layer is completely constructed, and as shown on the right side of fig. 6, the time stamps of Key values 1-13 are constructed according to the searching heat. When a new L0 layer SSTable is generated, the file is recorded until the total number of the L0 layer files reaches a preset threshold T, and then the L0 layer global index needs to be updated integrally.

Step S22: and establishing a jump table structure between each layer of SSTable by establishing connection between each index block on the layer and the index block of the next layer between the index blocks of the L1 layer and the LK layer in the LSM-Tree.

As shown in fig. 8, a basic idea of constructing a skip list for the Lk (k > 0) layer is given. Since the index ranges of the index blocks of the Lk layer do not overlap and are ordered. For this purpose, a multi-level index structure is constructed from the indices of all index blocks of the bottom layer. A multi-layer index structure is built for each index and index links are built between files. As shown in fig. 8, there are 3 layers of sstables, and the index structure of each layer is organized in the form of a skip list, i.e., each index block in each layer is added with a pointer to the index block corresponding to the next layer. Only the index structure of the bottommost layer is shown in the figure, and the bottom layers among all layers are connected by using a jump table.

When the index number of a certain layer reaches a preset threshold S, the index layer number of the layer is considered to be expanded, so that efficient dynamic updating and deleting are realized.

In one embodiment, the step S3: and (3) utilizing a skip list structure to realize hierarchical query on the index of the LSM-Tree.

Indexes among layers in the existing LSM-Tree key value system are relatively independent, and the search result of the upper layer cannot provide history information for the lower layer. In order to solve the problem, the embodiment of the invention designs a new metadata multi-layer jump mechanism of SSTable, index blocks among layers connect the latest Key values by using a jump table organization form, and then the index blocks of the index pointing to the lower layer are dynamically increased according to the number of the indexes of the lower layer of each layer.

As shown in FIG. 9, when a query 61 is to be made, the existing LSM-Tree needs to traverse all the indices of layer 3, thereby yielding a query result without query 61. By the hierarchical skip list structure provided by the embodiment of the invention, 47 in the L _i-1 layer directly skips to 58 on the L _i layer of the next layer, and skips from 58 on the L _i layer to 59 on the L _i+1 layer, when the next index of 59 is found to be 62, the query result of no query 61 can be obtained. Therefore, by using the method provided by the embodiment of the invention, the search result of the upper layer is utilized to provide reference for the search of the lower layer, the search path is reduced, the query result can be obtained only by 3 times of query, and SSTable of each layer is not required to be traversed, so that the query time is greatly shortened.

As shown in fig. 10, in one embodiment, a hierarchical index data management method based on LSM-Tree further includes step S4: updating the global index structure by adopting a lazy updating method; the lazy updating method adjusts the multi-level index according to the query path generated in each query to update the global index structure.

Because the construction of the full index structure when the database is started brings about larger IO and CPU overheads, and the recovery overhead of the database after the occurrence of the crash is increased. Therefore, the global index structure in the embodiment of the invention is not constructed from the beginning, but dynamically adopts a lazy updating method. The method comprises the steps of updating a global index structure by using a query path generated in each query, and adjusting the pointing direction of the multi-layer jump pointer through each query path. When a Key is queried, the pointing direction of a multi-layer jump pointer is regulated, so that the pointer can always keep the latest state, when hot spot data comes, the data index can always keep the optimal state, and when the hot spot data is faced, the lazy updating mechanism can ensure that the index structure always keeps the optimal state because query data is accessed for a plurality of times in a short time, and cold data does not need to be additionally regulated because the cold data is not accessed for a long time, thereby achieving the balance of query performance and maintenance cost.

Meanwhile, due to frequent merging operation of the bottom layer caused by writing of the upper layer, the data structure needs to be properly adjusted to keep the latest index structure, specifically, when the number of SSTable of a certain layer is greater than a preset threshold T, that is, merging operation occurs, the original pointer direction needs to be deleted and updated, which causes a great number of index failure problems. After the merging operation is completed, the SSTable numbers which participate in the merging operation and are deleted are independently stored in a deleted _sst list, the idea of lazy updating is still adopted, when index records meeting the conditions are queried, the corresponding SSTable numbers are queried in deleted _sst, when the SSTable is confirmed to be deleted, all records corresponding to the SSTable are deleted, the upper index of the skip list is returned, and the query is performed again. Such an update mechanism can ensure that a large number of sstables are deleted from the database without causing serious performance fluctuations due to index failure when performing heavy merge operations. Similarly, when a large number of SSTable is written, when the index structure is not synchronously updated to the latest state, by maintaining an add_sst list, when the index query falls into the index structure, the traditional block-based query mode is returned, and after the index structure is waited for updating, the SSTable number is deleted from the add_list, so that the high availability and data consistency of the system are ensured.

The hierarchical index data management method based on the LSM-Tree index provided by the invention adopts the operation steps of separating and storing an index block and a data block, designing a memory index data structure, designing a hierarchical acceleration query strategy and maintaining the memory index structure; when processing the read request, the index structure with fine granularity and the cross-layer acceleration strategy are adopted, compared with the original LSM-Tree index data management mode, the index inquiry search path is optimized on the premise of ensuring the write performance of the key value system, the average response time of the read request is shortened, and the read performance of the key value system is improved.

Example two

As shown in fig. 11, an embodiment of the present invention provides a hierarchical index data management system based on LSM-Tree, including the following modules:

the separating index block and data block module is used for separating the index block and the data block in the LRU chain table block cache in the LSM-Tree and establishing independent index block data;

The jump table structure building module is used for building a jump table structure between index blocks of different layers in the LSM-Tree according to the index block data;

And the query module is used for realizing hierarchical query on the index of the LSM-Tree by utilizing the skip list structure.

As shown in fig. 12, in one embodiment, the embodiment of the present invention provides a hierarchical index data management system based on LSM-Tree, further including the following modules:

The update index structure module is used for updating the global index structure by a lazy update method; the lazy updating method adjusts the multi-level index according to the query path generated in each query to update the global index structure.

The above examples are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalents and modifications that do not depart from the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims

1. The hierarchical memory index data management method based on the LSM-Tree is characterized by comprising the following steps of:

step S1: separating the index block and the data block in the LRU linked list block cache in the LSM-Tree to establish independent index block data, wherein the method specifically comprises the following steps:

Separating the index blocks from the chain table block cache, and establishing independent index block data, wherein each index block data occupies 12 bytes and is used for storing a Key value Key of the index block of the layer, the next index block of the layer and the index block of the lower layer; isolating the index block from the data block, and measuring the heat of two different data by using different doubly linked lists; each index block data in the index blocks is used as a minimum management unit and is managed independently;

step S2: according to the index block data, a jump table structure is established among index blocks of different layers in an LSM-Tree, and the jump table structure specifically comprises the following steps:

Step S21: the index block in the L0 layer in the LSM-Tree is searched for the heat degree, and an index ordering linked list of the latest heat degree is constructed; rescanning the overlapped Key values and updating the time stamps according to the search heat of the Key values of the index blocks;

Step S22: the jump table structure between each layer of SSTable is constructed by establishing the connection between the index blocks from the L1 layer to the LK layer in the LSM-Tree and the index blocks at the lower layer; because the index ranges of the index blocks of the LK layer are not overlapped and are orderly ordered, a multi-level index structure is constructed according to the indexes of all index blocks of the bottom layer; constructing a multi-layer index structure for each index, and establishing index links among files; the index structure of each layer is organized in the form of a jump table, namely, each index block in each layer is added with a pointer pointing to the index block corresponding to the next layer;

2. The LSM-Tree based hierarchical memory index data management method of claim 1, further comprising the steps of:

Step S4: updating the global index structure by adopting a lazy updating method; the lazy updating method adjusts the multi-level index according to the query path generated in each query to update the global index structure.

3. The hierarchical memory index data management system based on the LSM-Tree is characterized by comprising the following modules:

The separating index block and data block module is used for separating the index block and the data block in the LRU linked list block cache in the LSM-Tree, and establishing independent index block data, and specifically comprises the following steps:

The jump table structure building module is used for building a jump table structure between index blocks of different layers in the LSM-Tree according to the index block data, and specifically comprises the following steps:

and the query module is used for realizing hierarchical query on the index of the LSM-Tree by utilizing the jump table structure.

4. The LSM-Tree based hierarchical memory index data management system of claim 3, further comprising the modules of: