WO2019228009A1

WO2019228009A1 - Lsm tree optimization method and device and computer equipment

Info

Publication number: WO2019228009A1
Application number: PCT/CN2019/077404
Authority: WO
Inventors: 阳振坤; 席华锋; 韩富晟; 肖金亮
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2018-05-31
Filing date: 2019-03-08
Publication date: 2019-12-05
Also published as: TW202004521A; CN108804625A; TWI710918B; CN108804625B

Abstract

Disclosed are an LSM tree optimization method and device and a computer equipment, the method comprising: determining a target leaf node for which indicated data has already been deleted in an LSM tree; and adding a deletion tag for the target leaf node in dynamic data of the LSM tree.

Description

Optimization method, device and computer equipment of LSM tree

Technical field

The embodiments of the present specification relate to the field of data processing technologies, and in particular, to an optimization method, device, and computer device for an LSM tree.

Background technique

The LSM tree (Log-Structured Merge Tree) is a hard disk-based data structure that includes dynamic data and static data. Dynamic data and static data store the incremental modification of data. During the process of accessing the LSM tree, its dynamic data and static data are read in turn, and the read results are combined to obtain the final read result.

Because the LSM tree holds the modification increment of the data, when a large number of modification increments are used to indicate that the data has been deleted in the LSM tree, a large amount of useless data needs to be traversed during the data reading process. Valid data is read, which also causes the data read performance to decrease.

Summary of the Invention

In view of the above technical problems, the embodiments of the present specification provide a method, an apparatus, and a computer device for optimizing an LSM tree. The technical solutions are as follows:

According to a first aspect of the embodiments of the present specification, a method for optimizing an LSM tree is provided. The method includes:

Determine the target leaf node in the LSM tree whose data has been deleted;

In the dynamic data of the LSM tree, a delete tag is added to the target leaf node.

According to a second aspect of the embodiments of the present specification, an apparatus for optimizing an LSM tree is provided. The apparatus includes:

A target determining module, configured to determine a target leaf node in the LSM tree whose data has been deleted;

A first adding module is configured to add a delete tag to the target leaf node in the dynamic data of the LSM tree.

According to a third aspect of the embodiments of the present specification, there is provided a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the program when the processor executes the program. An optimization method of the LSM tree provided in the embodiment of the specification.

In the technical solution provided in the embodiment of the present specification, a target leaf node in which data pointed to by the LSM tree has been deleted is determined, and a deletion mark is added to the determined target leaf node in dynamic data of the LSM tree. Because when accessing the LSM tree, first access the dynamic data of the LSM tree, and then access the static data of the LSM tree. Therefore, in the dynamic data, add delete marks to the target leaf nodes whose data has been deleted. When accessing dynamic data, it can directly skip the leaf nodes with delete marks, thereby improving data access efficiency. Furthermore, when accessing static data, there is no need to repeatedly visit leaf nodes with delete marks, thereby improving data access efficiency.

It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and should not limit the embodiments of the present specification.

In addition, any one of the embodiments in this specification does not need to achieve all the effects described above.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly explain the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely These are some of the embodiments described in the embodiments of this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings.

Figure 1 is an example of an LSM tree;

2 is a flowchart of an embodiment of an LSM tree optimization method according to an exemplary embodiment of the present specification;

FIG. 3 is an example of static data of the LSM tree;

FIG. 4 is an example of adding a deletion marker to the dynamic data of the LSM tree;

FIG. 5 is another example of adding a deletion marker to the dynamic data of the LSM tree;

6 is a block diagram of an embodiment of an apparatus for optimizing an LSM tree according to an exemplary embodiment of the present specification;

FIG. 7 shows a more specific schematic diagram of a hardware structure of a computing device provided by an embodiment of the present specification.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present specification, the technical solutions in the embodiments of the present specification will be described in detail below with reference to the drawings in the embodiments of the present specification. Obviously, the described implementations The examples are only a part of the embodiments of this specification, but not all the examples. Based on the embodiments in this specification, all other embodiments obtained by a person having ordinary skill in the art should fall into the protection scope.

The LSM tree is a hard disk-based data structure. Its representative databases include Hbase, NessDB, and LevelDB. The storage engine of the LSM tree is the same as the storage engine of the B + tree. It also supports the addition, deletion, reading, modification, and sequential scanning operations, and it can effectively avoid the problem of random disk writes through batch storage technology. Specifically, the core idea is: Assume that the memory is large enough. Therefore, you do not need to write data to disk every time there is a data update. Instead, you can store the latest data in memory first, wait until the data in the memory reaches the specified size limit, and then use it. Merge and sort the data in memory and append them to the end of the disk queue. Based on this, the data stored in the LSM tree can be divided into static data and dynamic data. Among them, dynamic data refers to data in memory and static data. The data in is the data stored on persistent media, such as disk. Those skilled in the art can understand that what is stored in the LSM tree is the modification increment of the data, that is, the change information of the data. For example, as shown in Figure 1, it is an example of an LSM tree.

In the process of accessing the LSM tree, it is necessary to sequentially read its dynamic data and static data, and combine the read results to obtain the final read result.

At present, there are two problems in accessing LSM:

One: When frequent modification increments are inserted into the dynamic data to indicate that the data has been deleted, that is, when a large number of deletion records are stored in the dynamic data, it is necessary to traverse a large amount of useless data when accessing the dynamic data. As a result, the read performance of the data is reduced. For example, suppose the data is searched by the index key range [1, 50], and as shown in FIG. 1, there is a large amount of deleted data in the index key range [1, 50], for example, the leaves whose index keys are 3 to 8 The data pointed to by the node, the data pointed by the leaf nodes with the index keys of 15 to 31, and the data pointed by the leaf nodes with the index key of 32 and the index key of 49, then in the prior art, it is still necessary to traverse one by one The leaf nodes can finally get the data pointed to by the leaf nodes with index keys of 9 to 14, and the index keys of 33 to 48.

Second: When accessing static data, it is still necessary to traverse a large amount of useless data, resulting in data query performance degradation. For example, when accessing the static data illustrated in FIG. 1, it is still necessary to traverse the data pointed to by the leaf nodes whose index keys are 3 to 99, and it is impossible to skip the leaf nodes whose data has been deleted.

Based on this, the embodiment of the present specification provides a method for optimizing the LSM tree. In this method, when accessing the LSM tree, first access its dynamic data and then its static data, so that in the dynamic data of the LSM tree, Add a delete tag to the leaf node to which the pointed data has been deleted to indicate that the data has been deleted, so that when subsequent access to the LSM tree, if a leaf node with a delete tag is found in the dynamic data, it will jump directly This leaf node is passed, and in the subsequent search for static data, the leaf node with the same index key is no longer searched, thereby improving data query performance.

The following embodiments are described in detail below to describe the optimization method of the LSM tree:

Please refer to FIG. 2, which is a flowchart of an embodiment of an LSM tree optimization method according to an exemplary embodiment of the present specification. The method includes the following steps:

Step 202: Determine the target leaf node in the LSM tree whose data has been deleted.

In the embodiment of the present specification, the leaf nodes to which the data pointed to have been deleted can be determined respectively for the dynamic data and the static data. For the convenience of description, the leaf nodes that have been determined, that is, the data pointed to have been deleted, are determined. Called the target leaf node.

As follows, it is described from two aspects of dynamic data and static data:

First, dynamic data:

Taking the dynamic data shown in Figure 1 as an example, assuming that the data is searched by the index key range [1, 50], during the query process, it can be detected that the data pointed to by multiple leaf nodes with continuous index keys have been deleted. For example, Leaf nodes whose index keys are 3 to 8 and leaf nodes whose index keys are 15 to 31, then these leaf nodes can be determined as target leaf nodes.

Second, static data:

Taking the static data shown in Figure 3 as an example, suppose that the data is searched by the index key range [1, 50]. During the query process, it can be detected that the data pointed to by multiple leaf nodes with consecutive index keys have been deleted. For example, Leaf nodes with index keys of 3 to 15, then these leaf nodes can be determined as target leaf nodes.

Step 204: In the dynamic data of the LSM tree, add a delete tag to the target leaf node.

In this step, it is still explained from two aspects of dynamic data and static data:

First, dynamic data:

Based on the relevant examples in step 202 above, the target leaf nodes determined in the dynamic data include: leaf nodes with index keys of 3 to 8 and leaf nodes with index keys of 15 to 31. Then, in this step, you can Add delete marks to these target leaf nodes. For example, as shown in FIG. 3, in FIG. 3, "deleted" indicates a delete mark. Those skilled in the art can understand that the use of "deleted" as a delete mark is merely an example. In practical applications, the deletion mark may be embodied in other forms, which is not limited in the embodiments of the present specification.

In addition, it is also proposed in the embodiment of the present specification that if in the dynamic data of the LSM tree, all leaf nodes under any branch node are added with a delete tag, a delete tag is added to the branch node. For example, as shown in FIG. 3, under the branch node of “15”, all leaf nodes are added with a delete mark, so it is also possible to add a delete mark to the branch node, as shown in FIG. Through this kind of processing, when accessing the dynamic data of the LSM tree in the future, after accessing all the leaf nodes under the "3" branch node, you can directly skip the "15" branch node and access the "32" branch node To improve data access efficiency.

In addition, in the embodiment of the present specification, in consideration of further improving the efficiency of data access in static data in the future, it can be further judged that among the target leaf nodes to which the deletion mark is added, the leaf node pointed to by the target leaf node with the largest index key Whether the index key of the index is continuous with the maximum index key. If the index key is continuous, then no processing is required. If the index key is not continuous, a virtual leaf node can be inserted after the target leaf node with the largest index key. The index key of the virtual leaf node Add 1 to the largest index key.

For example, take the target leaf node to which the deletion tag is added as a leaf node with an index key of 3 to 8 as an example. Among these target leaf nodes, the maximum index key is 8, as shown in FIG. 4, the index key is 8 The index key of the next leaf node pointed to by the leaf node is 9, which is continuous with 8, so that no processing is required.

For another example, take the target leaf node to which the deletion tag is added as a leaf node having an index key of 15 to 31. Among these target leaf nodes, the maximum index key is 31, as shown in FIG. 4, the index key is The leaf node of 31 no longer points to other leaf nodes. Then, a virtual leaf node can be inserted after the leaf node, and the index key of the virtual leaf node is 32, for example, as shown in FIG. 4. Of course, those skilled in the art can understand that the virtual leaf node does not have a delete mark.

Through this kind of processing, the efficiency of data access in static data can be further improved. For example, when accessing dynamic data, when traversing to a leaf node with an index key of 9 and finding that it does not have a delete mark, you can change [ 3, 8] This index key range is regarded as the index key range of the deleted data. Later, when accessing static data, you can no longer access the leaf nodes whose index keys belong to this range.

Second, static data:

Based on the relevant examples in step 202 above, the target leaf nodes determined in the static data include leaf nodes with index keys of 3 to 15, at this time, it can be first determined whether there is an index key of 3 to 15 in the dynamic data. If the target leaf node does not exist, two target leaf nodes with the largest index key and the smallest index key in the target leaf node can be inserted into the dynamic data of the LSM tree. In the dynamic data of the LSM tree, the inserted Add a delete mark to the target leaf node.

For example, among the leaf nodes with index keys of 3 to 15, the maximum index key is 15 and the minimum index key is 3. According to the foregoing description, target leaf nodes with index keys of 3 and 15 are inserted into the dynamic data of the LSM tree, and In the dynamic data, a delete mark is added to the inserted target leaf node, as shown in FIG. 5.

In addition, it can be further judged whether the index key of the leaf node pointed by the target leaf node with the largest index key among the target leaf nodes to which the deletion mark is added is continuous with the largest index key. If the index key is continuous, then no processing is required. Continuously, a virtual leaf node can be inserted after the index key is the target leaf node of the largest index key, and the index key of the virtual leaf node is the largest index key plus one.

For example, in FIG. 5, the target leaf node to which the deletion mark is added is a leaf node with index keys of 3 and 15, the maximum index key of which is 15, and after the leaf node whose index key is 15, no With other leaf nodes, a virtual leaf node with an index key of 16 can be inserted after the leaf node, as shown in FIG. 5.

Through this kind of follow-up, in the process of accessing dynamic data, when the virtual leaf node with an index key of 16 is traversed and found that it does not have a delete mark, [3, 15] can be regarded as the index key range of the deleted data. Subsequently, when accessing static data, the leaf nodes whose index keys belong to this range can no longer be accessed, thereby improving the efficiency of data access in static data.

Corresponding to the foregoing method embodiments, an embodiment of the present specification further provides an apparatus for optimizing an LSM tree. Please refer to FIG. 6, which is a block diagram of an embodiment of an apparatus for optimizing an LSM tree according to an exemplary embodiment of the present specification. It may include a target determination module 61 and a first adding module 62.

The target determination module 61 may be used to determine a target leaf node in the LSM tree to which the pointed data has been deleted;

The first adding module 62 may be configured to add a delete tag to the target leaf node in the dynamic data of the LSM tree.

In an embodiment, the apparatus may further include (not shown in FIG. 6):

A second adding module is configured to add a delete tag to the branch node if it is detected that all leaf nodes under any branch node are added with a delete tag in the dynamic data of the LSM tree.

In an embodiment, the target determining module 61 may be specifically configured to:

For the dynamic data and static data of the LSM tree, it is determined that the data pointed to has been deleted and the two or more target leaf nodes have consecutive index keys.

In an embodiment, the first adding module 62 may include (not shown in FIG. 6):

An inserting sub-module for inserting two target leaf nodes with the largest index key and the smallest index key of the determined target leaf nodes into the LSM tree if the target leaf nodes are determined for the static data of the LSM tree In dynamic data

The tag adding submodule is used to add a delete tag to the inserted target leaf node in the dynamic data of the LSM tree.

In an embodiment, the apparatus may further include (not shown in FIG. 6):

A judging module, configured to judge whether the index key of the leaf node pointed to by the target leaf node with the largest index key among the target leaf nodes to which the deletion mark is added is continuous with the largest index key;

A virtual node inserting module is configured to insert a virtual leaf node after the target leaf node having the largest index key if it is not continuous, wherein the index key of the virtual leaf node is the largest index key plus one.

It can be understood that the target determination module 61 and the first adding module 62 are two modules with independent functions, which can be configured in the device at the same time as shown in FIG. 6 or can be separately configured in the device, so FIG. 6 shows The structure should not be construed as limiting the scheme of the embodiment of the present specification.

In addition, the implementation process of the functions and functions of each module in the above device is described in detail in the implementation process of the corresponding steps in the above method, and details are not described herein again.

An embodiment of the present specification further provides a computer device including at least a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the foregoing optimization of the LSM tree when executing the program. method. The method at least includes: determining a target leaf node in the LSM tree whose data has been deleted; and adding a deletion mark to the target leaf node in the dynamic data of the LSM tree.

FIG. 7 shows a more specific schematic diagram of a hardware structure of a computing device provided by an embodiment of the present specification. The device may include a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750. Among them, the processor 710, the memory 720, the input / output interface 730, and the communication interface 740 realize a communication connection within the device through the bus 750.

The processor 710 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits, etc., for performing related operations. Program to implement the technical solution provided by the embodiment of the present specification.

The memory 720 may be implemented in the form of ROM (Read Only Memory, read-only memory), RAM (Random Access Memory, random access memory), static storage devices, and dynamic storage devices. The memory 720 may store an operating system and other application programs. When the technical solutions provided in the embodiments of the present specification are implemented by software or firmware, related program codes are stored in the memory 720 and are called and executed by the processor 710.

The input / output interface 730 is used to connect an input / output module to implement information input and output. The input / output / module can be configured in the device as a component (not shown in FIG. 7), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, and an indicator light.

The communication interface 740 is used to connect a communication module (not shown in FIG. 7) to implement communication interaction between the device and other devices. The communication module can implement communication through a wired method (for example, USB, network cable, etc.), and can also implement communication through a wireless method (for example, mobile network, WIFI, Bluetooth, etc.).

The bus 750 includes a path for transmitting information between various components of the device (for example, the processor 710, the memory 720, the input / output interface 730, and the communication interface 740).

It should be noted that although the above device only shows the processor 710, the memory 720, the input / output interface 730, the communication interface 740, and the bus 750, in a specific implementation process, the device may also include necessary for achieving normal operation Other components. In addition, those skilled in the art can understand that the foregoing device may also include only components necessary to implement the solutions of the embodiments of the present specification, and does not necessarily include all the components shown in the drawings.

An embodiment of the present specification also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the foregoing optimization method of the LSM tree is implemented. The method at least includes: determining a target leaf node in the LSM tree whose data has been deleted; and adding a deletion mark to the target leaf node in the dynamic data of the LSM tree.

Computer-readable media includes permanent and non-persistent, removable and non-removable media. Information storage can be accomplished by any method or technology. Information may be computer-readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transmitting medium may be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

It can be known from the description of the foregoing embodiments that those skilled in the art can clearly understand that the embodiments of the present specification can be implemented by means of software plus a necessary universal hardware platform. Based on such an understanding, the technical solution of the embodiments of the present specification may be embodied in the form of a software product that is essentially or contributes to the existing technology. The computer software product may be stored in a storage medium, such as ROM / RAM, Magnetic disks, optical disks, and the like include a number of instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or portions of the embodiments of this specification.

The system, device, module, or unit described in the foregoing embodiments may be specifically implemented by a computer chip or entity, or a product with a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email sending and receiving device, and a game control Desk, tablet computer, wearable device, or a combination of any of these devices.

Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, as for the device embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the relevant part may refer to the description of the method embodiment. The device embodiments described above are only schematic, and the modules described as separate components may or may not be physically separated. When implementing the solutions of the embodiments of this specification, the functions of each module may be the same. Or multiple software and / or hardware. Some or all of the modules may also be selected according to actual needs to achieve the objective of the solution of this embodiment. Those of ordinary skill in the art can understand and implement without creative efforts.

The above are only specific implementations of the embodiments of the present specification. It should be noted that for those of ordinary skill in the art, without departing from the principles of the embodiments of the present specification, several improvements and retouches can be made. These Improvement and retouching should also be regarded as the protection scope of the embodiments of the present specification.

Claims

An optimization method for an LSM tree, the method includes:

Determine the target leaf node in the LSM tree whose data has been deleted;

In the dynamic data of the LSM tree, a delete tag is added to the target leaf node.
The method according to claim 1, further comprising:

In the dynamic data of the LSM tree, if it is detected that all leaf nodes under any branch node are added with a delete tag, a delete tag is added to the branch node.
The method according to claim 1, wherein determining the target leaf node in the LSM tree to which the pointed data has been deleted comprises:

For the dynamic data and static data of the LSM tree, it is determined that the data pointed to has been deleted and the two or more target leaf nodes have consecutive index keys.
The method according to claim 3, if the target leaf node is determined for the static data of the LSM tree, adding a delete tag to the target leaf node in the dynamic data of the LSM tree comprises:

Insert two target leaf nodes with the largest index key and the smallest index key from the determined target leaf nodes into the dynamic data of the LSM tree;

In the dynamic data of the LSM tree, a delete mark is added to the inserted target leaf node.
The method according to claim 1, after adding a delete tag to the target leaf node in the dynamic data of the LSM tree, the method further comprises:

Judging whether the index key of the leaf node pointed by the target leaf node with the largest index key among the target leaf nodes to which the deletion mark is added is continuous with the maximum index key;

If discontinuous, a virtual leaf node is inserted after the target leaf node with the largest index key, where the index key of the virtual leaf node is the largest index key plus one.
An apparatus for optimizing an LSM tree, the apparatus includes:

A target determining module, configured to determine a target leaf node in the LSM tree whose data has been deleted;

A first adding module is configured to add a delete tag to the target leaf node in the dynamic data of the LSM tree.
The apparatus according to claim 6, further comprising:

A second adding module is configured to add a delete tag to the branch node if it is detected that all leaf nodes under any branch node are added with a delete tag in the dynamic data of the LSM tree.
The apparatus according to claim 6, the target determination module is specifically configured to:

For the dynamic data and static data of the LSM tree, it is determined that the data pointed to has been deleted, and more than two target leaf nodes have consecutive index keys.
The apparatus according to claim 8, the first adding module comprises:

An inserting sub-module for inserting two target leaf nodes with the largest index key and the smallest index key of the determined target leaf nodes into the LSM tree if the target leaf nodes are determined for the static data of the LSM tree In dynamic data

The tag adding submodule is used to add a delete tag to the inserted target leaf node in the dynamic data of the LSM tree.
The apparatus according to claim 6, further comprising:

A judging module, configured to judge whether the index key of the leaf node pointed to by the target leaf node with the largest index key among the target leaf nodes to which the deletion mark is added is continuous with the largest index key;

A virtual node inserting module is configured to insert a virtual leaf node after the target leaf node having the largest index key if it is not continuous, wherein the index key of the virtual leaf node is the largest index key plus one.
A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the program according to any one of claims 1 to 5 when executing the program. method.