CN114840134A

CN114840134A - Log merge tree key value storage system, related method and related equipment

Info

Publication number: CN114840134A
Application number: CN202110146051.5A
Authority: CN
Inventors: 胡皓胜; 单海康; 闵鑫豪; 万继光
Original assignee: Huawei Cloud Computing Technologies Co Ltd
Current assignee: Huawei Cloud Computing Technologies Co Ltd
Priority date: 2021-02-02
Filing date: 2021-02-02
Publication date: 2022-08-02

Abstract

The embodiment of the application provides a log merge tree key value storage system, a related method and related equipment, and relates to the technical field of storage. The system is based on a hybrid storage architecture, which comprises a host, and a persistent storage medium and a nonvolatile storage medium connected with the host, wherein the persistent storage medium stores a plurality of ordered character string surface layers, each ordered character string surface layer comprises a plurality of ordered character string table groups, and each ordered character string table group comprises a plurality of ordered character string tables; the log merge tree key value storage system adopts a grouping compression mechanism and comprises two stages of filters corresponding to the ordered character string table groups, wherein the two stages of filters comprise group filters corresponding to the ordered character string table groups and table filters corresponding to the ordered character string table. The embodiment optimizes the read amplification problem and the write amplification problem of the storage system, and improves the read performance and the write performance of the storage system.

Description

Log merge tree key value storage system, related method and related equipment

Technical Field

The embodiment of the application relates to the technical field of storage, in particular to a log merge tree key value storage system based on a hybrid storage architecture, a related method and related equipment.

Background

In the background of the era of explosive information processing, the demand of data intensive applications can be met by key-value storage systems. Generally, disk reading and writing are performance bottlenecks of a key value storage system, and the key value storage system also has the problem of relatively serious reading amplification and writing amplification.

Disclosure of Invention

The embodiment of the application discloses a log merge tree key value storage system optimization method based on a hybrid storage architecture and related equipment, which can optimize the read amplification problem of a storage system and improve the read performance of the storage system.

The application discloses a log merge tree key value storage system based on a hybrid storage architecture, which is applied to electronic equipment, wherein the hybrid storage architecture comprises a host, a persistent storage medium and a nonvolatile storage medium, the host is connected with the nonvolatile storage medium and the persistent storage medium, the persistent storage medium stores a plurality of ordered character string surface layers, each ordered character string surface layer comprises a plurality of ordered character string table groups, and each ordered character string table group comprises a plurality of ordered character string tables; the log merge tree key value storage system adopts a grouping compression mechanism and comprises two stages of filters corresponding to the ordered character string table groups, wherein the two stages of filters comprise group filters corresponding to the ordered character string table groups and table filters corresponding to the ordered character string tables.

The embodiment of the application can solve the problems that due to a grouping compression strategy of the log merging tree, reading operation possibly needs to carry out invalid access on ordered character string tables in a plurality of groups, and extra reading overhead and reading amplification are caused, so that the reading amplification problem and the writing amplification problem of a storage system are optimized, and invalid reading operation is avoided through a two-stage filter. The embodiment of the application can improve the reading performance of the storage system.

In some alternative embodiments, the two-stage filter is stored on a non-volatile storage medium.

The nonvolatile storage medium has the excellent storage characteristics of both the internal memory and the persistent storage medium, and can improve the reading performance and the writing performance of the storage system. Metadata information such as a filter is stored on a nonvolatile storage medium, and write operation is reduced through in-place updating operation, so that write overhead is reduced. The reading performance of the storage system is optimized by reducing the access of invalid persistent storage media by utilizing metadata such as a high-performance and persistent nonvolatile storage media storage filter, and the long tail delay is reduced.

In some alternative embodiments, the group filter comprises a bloom filter or a valley filter and the table filter comprises a bloom filter, a quotient filter, or a valley filter.

Embodiments of the present application enable group filters.

In some alternative embodiments, the hybrid storage architecture further includes a volatile storage medium or memory.

The second aspect of the present application discloses a data search method, based on the above log merge tree key value storage system, the data search method includes: acquiring a target key of data to be searched; and searching the target key from the persistent storage medium according to the two-stage filter.

Embodiments of the present application can avoid invalid lookups and reduce read latency by a two-stage filter.

In some alternative embodiments, the seeking a target key from a persistent storage media according to a two-stage filter comprises: searching a target key from a memory of the electronic equipment; and if the target key is not searched from the memory of the electronic equipment, searching the target key from the persistent storage medium according to the two-stage filter.

The embodiment of the application can search the target key from the memory and the persistent storage medium of the electronic equipment.

In some optional embodiments, after obtaining the target key of the data to be searched, the data searching method further includes: searching a target key from a memory of the electronic equipment; and if the target key is not searched from the memory of the electronic equipment, searching the target key from the persistent storage medium according to the two-stage filter.

In some alternative embodiments, finding the target key from the persistent storage media according to the two-stage filter comprises: according to the layer sequence, for any current ordered character string surface layer in the ordered character string surface layers, judging whether a target ordered character string table group for storing a target key exists in the current ordered character string surface layer or not according to the group key range of a plurality of ordered character string table groups in the current ordered character string surface layer; and if a target ordered character string table group for storing the target key exists in the current ordered character string surface layer, searching the target key from the target ordered character string table group according to the two-stage filter.

According to the method and the device, before the target key is searched from the target ordered character string table group in the current ordered character string surface layer according to the two-stage filter, whether the target ordered character string table group storing the target key exists in the current ordered character string surface layer or not is judged, and the searching efficiency is improved.

In some optional embodiments, if the target ordered character string table group storing the target key does not exist in the current ordered character string surface layer, the target key is searched from the next ordered character string surface layer of the current ordered character string surface layer.

The method and the device for searching the target key in the ordered character string surface layer can search the target key from the next ordered character string surface layer of the current ordered character string surface layer in the order of the layer sequence numbers or the random order.

In some alternative embodiments, finding the target key from the set of target ordered strings according to the two-stage filter comprises: judging whether the target ordered character string table group has the possibility of including a target key or not according to a group filter in the two-stage filter; if the possibility that the target ordered character string table group comprises the target key is determined according to the group filter in the two-stage filter, judging whether the target ordered character string table group has the target ordered character string table for storing the target key or not according to the table key range of a plurality of ordered character string tables in the current ordered character string table group; if the target ordered character string table storing the target key exists in the target ordered character string table group is determined according to the table key range of the plurality of ordered character string tables in the current ordered character string table group, whether the target ordered character string table has the possibility of including the target key is judged according to the table filter in the two-stage filter; and if the possibility that the target ordered character string table has the target key is determined according to the table filter in the two-stage filter, searching the data to be searched in the target ordered character string table according to the target key.

According to the method and the device, the target key can be searched from the target ordered character string table group according to the two-stage filter, the possibility that the target ordered character string table group comprises the target key can be judged firstly, if the target ordered character string table group comprises the possibility of the target key, the target ordered character string table group has higher probability of comprising the target key, then the ordered character string table in the target ordered character string table group is further judged, and finally whether the target key is included in the target ordered character string table group or not is accurately judged. By using the two-stage filter, invalid searching can be avoided, and the target key searching efficiency is improved.

In some optional embodiments, if it is determined that the target ordered string table set does not have the possibility of including the target key according to the set filter in the two-stage filter, or it is determined that the target ordered string table does not have the possibility of including the target key according to the table filter in the two-stage filter, the target key is searched for from the next ordered string surface layer of the current ordered string surface layer.

According to the method and the device, the target key can be searched from the next ordered character string surface layer of the current ordered character string surface layer under the condition that the target key does not exist in the current ordered character string surface layer. And finishing the search of all the ordered character string surface layers.

In some alternative embodiments, finding the target key from the persistent storage media according to the two-stage filter comprises: and searching the target key from a plurality of ordered character string surface layers in the persistent storage medium by using a two-stage filter according to the layer sequence.

According to the method and the device, the target key can be searched from the surface layers of the ordered character strings according to the sequence of the layer sequence numbers. Thereby finding the more recently stored target key.

In some optional embodiments, for any one of the plurality of ordered string surface layers that is currently ordered, finding the target key from the currently ordered string surface layer according to the two-stage filter comprises: judging whether a target key exists in the current ordered character string surface layer or not according to the two-stage filter; if a target key exists in the current ordered character string surface layer according to the two-stage filter, searching data to be searched according to the target key; and if the target key does not exist in the current ordered character string surface layer according to the two-stage filter, searching the target key from the next ordered character string surface layer of the current ordered character string surface layer.

According to the method and the device, the target key can be searched from the current ordered character string surface layer.

In some optional embodiments, determining whether the target key exists in the current ordered character string surface layer according to the two-stage filter includes: judging whether a target ordered character string table group exists in the current ordered character string surface layer or not according to a group filter in the two-stage filter, wherein the target ordered character string table group has the possibility of comprising a target key; if the target ordered character string table group does not exist in the current ordered character string surface layer, determining that a target key does not exist in the current ordered character string surface layer; if a target ordered character string table group exists in the current ordered character string surface layer, judging whether the target ordered character string table group exists in a target ordered character string table according to a table filter in the two-stage filter, wherein the target ordered character string table has the possibility of comprising a target key; if the target ordered character string table does not exist in the target ordered character string table group, determining that a target key does not exist in the surface layer of the current ordered character string; if the target ordered character string table exists in the target ordered character string table group, judging whether a target key exists in the target ordered character string table or not; and if the target key exists in the target ordered character string table, determining that the target key exists in the current ordered character string surface layer.

The method and the device for judging the target key in the ordered character string surface layer can judge whether the target key exists in the ordered character string surface layer based on the ordered character string surface layer in the current ordered character string surface layer according to the two-stage filter.

In some optional embodiments, determining whether a target ordered character string table group exists in the current ordered character string surface layer according to a group filter in the two-stage filter, where the possibility that the target ordered character string table group includes a target key includes: acquiring the group key range of a plurality of ordered character string table groups in the current ordered character string surface layer; determining one or more ordered character string table groups of which the group key range comprises a target key from a plurality of ordered character string table groups in the current ordered character string surface layer to obtain a determined ordered character string table group; judging whether the determined ordered character string table group has the possibility of including the target key or not through a group filter of the determined ordered character string table group; if the determined ordered character string table group has the possibility of comprising the target key, determining that the ordered character string table group with the possibility of comprising the target key exists in the current ordered character string surface layer, and taking the determined ordered character string table group as the target ordered character string table group; and if the determined ordered character string table group does not have the possibility of comprising the target key, determining that the target ordered character string table group which has the possibility of comprising the target key does not exist in the current ordered character string surface layer.

The method and the device for judging the target ordered character string table group can judge whether the target ordered character string table group with the possibility of existence of the target key exists in the current ordered character string surface layer according to the group filter in the two-stage filter based on the group key range.

In some optional embodiments, determining whether a target ordered character string table group exists in the current ordered character string surface layer according to a group filter in the two-stage filter, where the possibility that the target ordered character string table group includes a target key includes: acquiring ordered character string table groups in the current ordered character string surface layer one by one, and judging whether the ordered character string table groups have the possibility of including a target key or not through a group filter of the ordered character string table groups; if any one ordered character string table group in the current ordered character string surface layer has the possibility of comprising the target key, determining that the target ordered character string table group with the possibility of comprising the target key exists in the current ordered character string surface layer; and if all the ordered character string table groups in the current ordered character string surface layer do not have the possibility of including the target key, determining that the target ordered character string table group with the possibility of including the target key does not exist in the current ordered character string surface layer.

According to the method and the device, each ordered character string table group in the current ordered character string surface layer can be obtained one by one, and whether the target ordered character string table group with the possibility of existence of the target key exists in the current ordered character string surface layer is judged according to the group filter in the two-stage filter. Alternatively, the ordered character string table group in the current ordered character string surface layer may be sequentially acquired, or the ordered character string table group in the current ordered character string surface layer may be randomly traversed.

In some optional embodiments, determining whether the target ordered string table exists in the target ordered string table group according to a table filter in the two-stage filter, where the possibility that the target ordered string table includes the target key includes: acquiring the table key ranges of a plurality of ordered character string tables in the target ordered character string table group; determining one or more ordered character string tables of which the table key range comprises a target key from a plurality of ordered character string tables in the target ordered character string table group to obtain a determined ordered character string table; judging whether the determined ordered character string table has the possibility of including the target key or not through a table filter of the determined ordered character string table; if the determined ordered character string table has the possibility of comprising the target key, determining that the ordered character string table with the possibility of the target key exists in the target ordered character string table group, and taking the determined ordered character string table as the target ordered character string table; and if the determined ordered character string table does not have the possibility of comprising the target key, determining that the target ordered character string table which has the possibility of comprising the target key does not exist in the target ordered character string table group.

The embodiment of the application can judge whether the target ordered character string table group has the target ordered character string table with the possibility of including the target key according to the table filter in the two-stage filter based on the table key range.

In some optional embodiments, determining whether the target ordered character string table exists in the target ordered character string table group according to the table filter in the two-stage filter, where the possibility that the target ordered character string table group includes the target key includes: acquiring ordered character string tables in the target ordered character string table group one by one, and judging whether the ordered character string tables have the possibility of including the target key or not through a table filter of the ordered character string tables; if any one ordered character string table in the target ordered character string table group has the possibility of comprising the target key, determining that the target ordered character string table with the possibility of comprising the target key exists in the target ordered character string table group; and if all the ordered character string tables in the target ordered character string table group do not have the possibility of including the target key, determining that the target ordered character string table having the possibility of including the target key does not exist in the target ordered character string table group.

The method and the device for obtaining the ordered character string table group can achieve the purpose that the ordered character string table in the target ordered character string table group is obtained one by one, and whether the target ordered character string table with the possibility of including the target key exists in the target ordered character string table group is judged according to the table filter in the two-stage filter. Alternatively, the newly generated ordered character string table may be obtained from the target ordered character string table group, or the ordered character string tables in the target ordered character string table group may be traversed randomly.

The third aspect of the present application discloses a filter updating method, based on the above log merge tree key value storage system, the filter updating method including: determining any one of the ordered character string surface layers as a current ordered character string surface layer, wherein the ordered character string surface layers are stored according to a preset sequence; obtaining the number of ordered character string tables in the current ordered character string surface layer and the upper limit number of a preset table; if the number of the ordered character string tables in the current ordered character string surface layer is larger than or equal to the upper limit number of the preset tables, selecting an ordered character string table group from the current ordered character string surface layer; generating an ordered character string table to be merged according to the selected ordered character string table group; merging the to-be-merged ordered character string table to a next ordered character string surface layer of the current ordered character string surface layer to obtain a merged ordered character string surface layer; and updating the two-stage filter of the next ordered character string surface layer of the current ordered character string surface layer according to the merged ordered character string surface layer.

In the compression operation process, the storage system needs to update the metadata block and the filter block of the persistent storage medium due to the log merging tree hierarchical structure, so that the problems of extra write overhead and write amplification are caused.

In some alternative embodiments, selecting the ordered set of strings from the current ordered string table layer comprises: and selecting the ordered character string table group from the current ordered character string table layer according to the number and the generation time of the ordered character string tables.

The embodiment of the application can realize the selection of the ordered character string table.

In some optional embodiments, generating the ordered string table to be merged according to the selected ordered string table group includes: merging and sorting the ordered character string tables in the selected ordered character string table group to obtain a first ordered character string table; and dividing the first ordered character string table into at least one second ordered character string table according to the key range of the next ordered character string surface layer of the current ordered character string surface layer to obtain the ordered character string table to be merged, which comprises at least one second ordered character string table.

The embodiment of the application can generate the ordered character string table to be merged according to the selected ordered character string table group.

In some alternative embodiments, dividing the first ordered string table into at least one second ordered string table according to a key range of a next ordered string table of the current ordered string table comprises: if the table key range of the first ordered character string table is contained in any group key range of the next ordered character string surface layer of the current ordered character string surface layer, the first ordered character string table is not divided; if the table key range of the first ordered character string table is included in any plurality of group key ranges of the next ordered character string surface layer of the current ordered character string surface layer, dividing the first ordered character string table according to the any plurality of group key ranges; and if the table key range of the first ordered character string table is not contained in any group key range of the next ordered character string surface layer of the current ordered character string surface layer, uniformly dividing the first ordered character string table.

The embodiment of the application can divide the first ordered character string table into at least one second ordered character string table according to the key range of the next ordered character string surface layer of the current ordered character string surface layer.

In some alternative embodiments, merging the ordered string table to be merged to a next ordered string surface of the current ordered string surface comprises: and in the next ordered character string surface layer of the current ordered character string surface layer, merging each second ordered character string table in the ordered character string tables to be merged into an ordered character string table group with the group key range containing the table key range of the second ordered character string table through multithreading.

The embodiment of the application can realize that the ordered character string table to be merged is merged to the next ordered character string surface layer of the current ordered character string surface layer.

In some optional embodiments, the two-stage filter for updating the next ordered string surface layer of the current ordered string surface layer according to the merged ordered string surface layer comprises: generating corresponding group element data according to the ordered character string table group generated in the combined ordered character string surface layer, and establishing a corresponding group filter in a nonvolatile storage medium; and updating the corresponding group filter in the nonvolatile storage medium according to the modified ordered character string table group in the combined ordered character string table layer.

The embodiment of the application can update the two-stage filter of the next ordered character string surface layer of the current ordered character string surface layer according to the merged ordered character string surface layer. Write operations and write overhead are reduced through in-place updating.

The fourth aspect of the present application discloses a nonvolatile storage medium management method, which is based on the log merge tree key value storage system, and includes: receiving a create request for a table filter or group filter; converting the creation request into a space allocation request for the nonvolatile storage medium; acquiring a lock of an idle linked list of the nonvolatile storage medium, wherein the idle linked list comprises page numbers of idle pages of the nonvolatile storage medium; extracting a plurality of page numbers from the idle linked list; creating a storage management object in a nonvolatile storage medium according to the plurality of page numbers; returning the storage management object; the lock of the free linked list is released.

The embodiment of the application can solve the problem of space allocation of the nonvolatile storage medium. And paging management is performed on the storage space of the nonvolatile storage medium, bottom paging is packaged, and memory allocation of the nonvolatile storage medium is efficiently performed. Memory allocation of non-volatile storage media may be implemented based on multiple threads.

The sixth aspect of the present application discloses a nonvolatile storage medium management method, which is based on the log merge tree key value storage system, and includes: triggering a reclamation request for space stored by the non-volatile storage media in response to a data failure of the table filter or the group filter; adding the recovery request into a recovery queue, and informing the scheduling thread of the recovery request; creating a plurality of recovery threads according to the recovery requests in the recovery queue through the scheduling threads so as to recover the failure pages through the plurality of recovery threads; updating an idle linked list according to the failure page, wherein the idle linked list comprises the page number of the idle page of the nonvolatile storage medium; waiting for the recycle thread to end.

The embodiment of the application can perform page reclamation based on multithreading. And the space management of the lightweight nonvolatile storage medium is realized.

In some optional embodiments, updating the idle-linked list according to the stale page includes: acquiring a header page number of an idle linked list; assigning the header page number to the next page number of the failure page; and re-determining the page number of the failed page as the head page number.

According to the embodiment of the application, the idle linked list can be updated according to the failure page.

A seventh aspect of the present application discloses a computer device comprising a processor and a memory; a memory to store instructions; a processor for invoking instructions in the memory to cause the computer device to perform a method associated with a log merge tree key-value storage system.

An eighth aspect of the present application discloses a computer-readable storage medium storing at least one instruction, which when executed by a processor, implements a related method of a log merge tree key value storage system.

For technical effects brought by the seventh aspect to the eighth aspect, reference may be made to the description related to the methods in the above method part, and details are not described herein again.

Drawings

FIG. 1 is the LSM-Tree tissue structure.

Fig. 2 is the RocksDb structure.

FIG. 3 is a compression mechanism for a key-value storage system.

FIG. 4 is an SLM-DB.

FIG. 5 is an architecture of a hybrid storage architecture-based log merge tree key-value storage system according to an embodiment of the present application.

Fig. 6 is a schematic structural diagram of a two-stage filter provided in an embodiment of the present application.

Fig. 7 is a schematic diagram of access to a two-stage filter provided in an embodiment of the present application.

Fig. 8 is a flowchart of a data lookup method of a log merge tree key-value storage system based on a hybrid storage architecture according to an embodiment of the present application.

Fig. 9 is a schematic data search diagram of a log merge tree key-value storage system based on a hybrid storage architecture according to an embodiment of the present application.

Fig. 10 is a flow chart of filter updating provided by an embodiment of the present application.

Fig. 11 is a schematic diagram of filter updating provided by an embodiment of the present application.

FIG. 12 is a flowchart of a mutually exclusive application method according to an embodiment of the present application.

Fig. 13 is a flowchart of a concurrent recovery method provided in an embodiment of the present application.

Fig. 14 is a data structure logical relationship diagram of a nonvolatile storage medium management method according to an embodiment of the present application.

Fig. 15 is a schematic diagram of filter creation provided in an embodiment of the present application.

Fig. 16 is a schematic diagram of a computer device provided in an embodiment of the present application.

Detailed Description

For ease of understanding, some descriptions of concepts related to the embodiments of the present application are given by way of illustration and reference.

In the embodiments of the present application, "at least one" means one or more, "and" a plurality "means two or more than two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, e.g., A and/or B may represent: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The terms "first," "second," "third," "fourth," and the like in the description and in the claims and drawings of the present application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.

Key Value (KV) storage systems may be implemented based on Log Structured Merge Tree (LSM-Tree). A Dynamic Random Access Memory (DRAM) -based key-value storage system usually has a performance bottleneck of disk reading and writing, and an LSM-Tree has a relatively serious problem of reading amplification and writing amplification, where the reading or writing amplification indicates a ratio of an actual read or write data amount of the storage system to a data amount desired to be written by a user. Read amplification can affect the read performance of a key-value storage system, and write amplification can reduce the lifetime of a key-value storage system due to wear caused by repeated writes. A Non-Volatile Memory (NVM) has excellent Memory characteristics of both internal Memory (Memory) and external Memory. The key value storage system in the embodiment of the application is based on a hybrid storage architecture of a persistent storage medium and an NVM and a log merge tree, and can improve data read-write performance. The persistent storage medium may be a Solid State Drive (SSD), which is also called a Solid State Disk (SSD), and has the characteristics of large capacity and good stability.

For the convenience of understanding, the LSM-Tree, LSM-Tree key-value storage system architectures, key-value storage system compression mechanisms, and nonvolatile storage media are briefly described below.

As shown in fig. 1, is LSM-Tree organization structure. The LSM-Tree converts random writes to persistent storage media, such as a Disk (Disk), to sequential writes, thereby increasing write speed. The index Tree structure in the LSM-Tree comprises a small Tree (C0) and a large Tree (C1, C2... Ck), wherein the small Tree is stored in a memory, the large Tree can be stored in a persistent storage medium, and the two trees jointly maintain an ordered Key space. The received write operation firstly operates the treelet C0 in the memory, and as the treelet in the memory becomes larger, when the size of the treelet reaches a preset condition, a merge operation (MergeSort) with a large tree (C1, C2... Ck) in the persistent storage medium is triggered, and the write operation generated by the merge operation is sequential write.

The LSM-Tree key value storage system architecture may include a RocksDb structure or a LevelDB structure, etc. As shown in fig. 2, is a RocksDb structure. The LSM-Tree may include memory structures (memtable, immutable memtable) and on-disk structures (e.g., a Sorted String Table (SSTable, SSTable or SST)). The role of the EMS memory structure is similar to that of C0 in the LSM-Tree of FIG. 1, and the role of the on-disk structure is similar to that of the large Tree of the LSM-Tree of FIG. 1. The memtable is a structure for organizing and maintaining data in the memory; an immutable memtable is a non-modifiable memtable, also referred to as an immutable for short. Before writing data to the memory buffer, a pre-write log, also known as a Write Ahead Log (WAL), is performed. Flush refers to the process of importing memtable data into SSTable and changing the data into persistent storage.

The RocksDb structure write operation data flow path is: "mtable- > immutable memables- > L0 SStables- > L1 SStables- >. Ln SStables". Therefore, the SSTable with the smaller layer number has the newer data, and the memory has the newest data.

Read operation flow of RocksDb structure: after the memory has been sequentially accessed by read operation, the target SSTable is located by SSTable Info Cache maintained in the memory, and at the same time, SSTable filter corresponding to the target SSTable is used to judge whether the key of target data exists in the target SSTable, if yes, the target data is obtained from the target SSTable. Target data can be acquired from L0, L1 to Ln layers on the magnetic disk in sequence; if the Ln layer of the highest layer is searched, the key of the target data is not hit, and the searched target data does not exist in the RocksDb structure. The search mechanism of the LSM-Tree and the key value storage system can be realized, if a plurality of target data are stored in the RocksDb structure, the key of the target data read first in the reading process is the key of the latest target data in time sequence. Among them, the SSTable Filter (SSTable Filter) is a hash structure for determining whether an object exists in the SSTable, and examples thereof include a Bloom Filter (Bloom Filter), a Quotient Filter (finite Filters), and the like.

As shown in fig. 3, is a compression mechanism for a key-value store system. LSM-Tree based key value storage systems (such as levelDB and RocksDB) need to regularly merge low-level SStables into higher-level SStables, and the organization form and merging mode of the SStables in different levels are called compression mechanisms (compact). The compression mechanism of the key-value storage system may include a Level compression mechanism (Level compact), a Tier compression mechanism (Tier compact), and a Group compression mechanism (Group compact). The hierarchical compression mechanism is a default compression mechanism of LSM-Tree-based key value storage systems such as a LevelDB and a RocksDB, and is a compression mechanism capable of dividing each layer of files according to key values. The layered compression mechanism is a compression mechanism that manages and merges SSTable by file size. The packet compression mechanism is based on a layered compression mechanism, and each layer of SSTable is organized according to a Group (Group).

In a key value storage system based on LSM-Tree, if the data quantity of SSTable of Li layer is equal to or more than the upper limit, one or more SSTable of Li layer is merged with SSTable of Li +1 layer, SSTable in original Li +1 layer is deleted, SSTable generated by merging is inserted into Li +1 layer.

As shown in fig. 3A, is a hierarchical compression mechanism for a key-value store system. In the hierarchical compression mechanism, the SStables of each layer are arranged in sequence, and the table key ranges of every two SStables are not intersected with each other. And if the data quantity of the SSTable of the Li layer is equal to or more than the upper limit, selecting one or more SStables of the Li layer and the SSTable with the table bond range of the Li +1 layer to be combined, deleting the SSTable in the original Li +1 layer, and inserting the combined SSTable into the Li +1 layer. If the data amount of SSTable of the Li +1 layer is also equal to or greater than the upper limit, the combination of the Li +1 layer and the Li +2 layer is performed iteratively. As shown in fig. 3A, memory space is shown above the horizontal line, and persistent storage media is shown below the horizontal line. When 13 is written in the memory, 7 and 13 in the memory are written into an L0 layer, after 7 and 13 are written into an L0 layer, the data amount of SSTable of an L0 layer is equal to the upper limit, 1, 3, 7, 10, 13 and 19 in an L0 layer and 2, 4, 8, 12, 15 and 18 in an L1 layer are combined, SSTable in an original L1 layer is deleted, and SSTable generated by combination is inserted into the L1 layer.

The ordering maintained by the hierarchical compression mechanism can cause serious tail-lag problems, block the foreground request response, and cause serious write amplification.

As shown in fig. 3B, is a hierarchical compression mechanism for a key-value store system. The hierarchical merging mechanism distributes SSTable in different levels according to file size. The SStables of each layer may be arranged out of order, and there may be a range of intersecting table bonds between every two SStables of the same layer. And if the data quantity of the SSTable of the Li layer is equal to or larger than the upper limit, selecting one or more SStables of the Li layer which overlap with the table bond range of the Li +1 layer for merging in the memory, deleting the SSTable in the original Li +1 layer, and inserting the SSTable generated by merging into the Li +1 layer. If the data amount of SSTable of the Li +1 layer is also equal to or larger than the upper limit, the Li +1 layer is combined in an iteration mode.

SStables in a layered compression mechanism's key-value storage system are in an unordered state, as are the ranges of table keys between SStables. Resulting in a lower data read speed for the key-value storage system of the hierarchical compression scheme than for the key-value storage system of the hierarchical compression scheme. Although the key values in a single SSTable are in an ordered state, the data read speed of the log merge tree based storage system is lower than the data read speed of the B + tree based storage system. In particular, in the big data scenario, the amount of data is proliferating and the ordering between sstables cannot be completely abandoned.

As shown in fig. 3C, is a packet compression mechanism for a key-value store system. The grouping compression mechanism divides each layer of SSTable according to the table key range on the basis of the layering compression mechanism, and crossed table key ranges can exist among SSTable keys in the same group. By maintaining inter-group ordering, read performance can be improved. As shown in fig. 3C, when 13 is written in the memory, 7 and 13 in the memory are written in the L0 layer, and after 7 and 13 are written in the L0 layer, the data amount of SSTable in the L0 layer is equal to the upper limit (SSTable needs to write in an SSTable group whose group key range includes table keys of SSTable, that is, in an SSTable group whose 7 and 13 write group key range is 1-100, cannot write in an SSTable group whose group key range is 101-200), 1, 3, 55, 69, 7, 13 in the L0 layer and the SSTable in the L1 layer are merged (L1 layer has no SSTable in fig. 3C), the sst in the original L1 layer is deleted, and the merged SSTable is inserted in the L1 layer.

The nonvolatile storage medium may include a Phase Change Memory (PCM), a Resistive/Resistive random access Memory (ReRAM), a Memristor (memrisor), and the like. The nonvolatile storage medium has excellent characteristics of a memory and a persistent storage medium, and has good storage characteristics. The nonvolatile storage medium has storage delay close to that of the DRAM, and meanwhile, a Memory access unit (namely a cache line) similar to the DRAM is provided, and addressing can be carried out according to byte units, so that a Host (Host) can directly access the nonvolatile storage medium through a Memory Bus (Memory Bus) like accessing the DRAM, access through a hardware block device interface is avoided, and an access wall corresponding to the hardware block device interface is avoided. Meanwhile, the nonvolatile storage medium has nonvolatile property of a persistent storage medium, and data stored in the nonvolatile storage medium cannot be lost after the power supply is cut off. The storage density of the nonvolatile storage medium is larger than that of the DRAM, the capacity of the nonvolatile storage medium can reach TB level, and the nonvolatile storage medium is a storage medium which is used for supplementing the DRAM and constructing a mass storage system.

One implementation to improve the performance of a log merge tree key-value storage system is as follows.

Generally, the performance of a log Merge tree key value storage system is improved by using an NVM through an SLM-DB (Single-Level measure DB), and the consistency is improved by using the NVM to store MemTable and Immutable, while reducing the overhead of consistency. The SLM-DB converts a multi-Level data structure on a disk into a Single-Level data structure, and introduces a B + -Tree in the NVM as an index of the Single-Level data structure (Single Level), so that B + -Tree indexes of all key and value positions on the disk are established, and the index on the NVM is used for reducing the access to the disk when data is inquired. Shown in FIG. 4 as SLM-DB. Wherein, Persistent memory represents Persistent memory, and compact Log represents compressed Log.

Storing the B + -Tree indices for all keys on NVM requires a large amount of NVM space, and storing variable-length Key fields in the index of NVM requires complex NVM management mechanisms. B + -Tree indexes on NVM maintained by the SLM-DB bring excessive NVM cost, and due to the complexity of NVM management mechanism, the space utilization of NVM is low.

In order to better understand the log merge tree key value storage system based on the hybrid storage architecture, the optimization method and the related device disclosed in the embodiment of the present application, technical problems solved by the embodiment of the present application are explained below.

For solving the problem of reading and writing amplification of a log merge Tree key value storage system, the LSM-Tree based key value storage system converts a write request into sequential write to a persistent storage medium through a structure in a memory, and data are hierarchically distributed on different layers of the persistent storage medium according to the cold and hot degrees. But correspondingly introduces maintenance overhead, each compression operation will update the data index and filter on the persistent storage media multiple times, thereby increasing write amplification. Meanwhile, when data is read, data of different levels need to be accessed, a data block where the data corresponding to the Key is located needs to be read, and an index block and a filter block corresponding to the Key need to be read, so that extra reading overhead is increased.

How to solve the space management problem of the NVM. The NVM is used for accelerating the reading and writing of the key value storage system, but the existing NVM space management method has the problem of insufficient performance and is not matched with the requirement of the LSM-Tree-based key value storage system on data persistence.

In order to better understand the log merge tree key value storage system based on the hybrid storage architecture, the optimization method and the related device disclosed in the embodiment of the present application, first, the architecture of the log merge tree key value storage system based on the hybrid storage architecture of the present application is described below. For convenience of description, the "log merge tree key-value storage system based on the hybrid storage architecture" will be simply referred to as "log merge tree key-value storage system".

As shown in fig. 5, an architecture of a log merge tree key-value storage system based on a hybrid storage architecture is provided in an embodiment of the present application.

The hardware architecture of a log merge tree key-value storage system based on a hybrid storage architecture as shown in fig. 5 includes a host, a persistent storage medium, and a non-volatile storage medium. The host is connected with the nonvolatile storage medium, and the host is connected with the persistent storage medium. The host may be connected to the nonvolatile storage medium through a memory bus, and the host may be connected to the persistent storage medium through pcie (peripheral component interconnect express). The nonvolatile storage medium may be an Optane disk based on 3D Xpoint, and a Central Processing Unit (CPU) of the host performs data access and data persistence through a memory bus and a clfush quality instruction and a meffence instruction of an X86 instruction set.

The hybrid storage architecture also includes a volatile storage medium (memory) to which the host is coupled.

As shown in fig. 5, based on a persistent storage system (such as a stand-alone file system, a distributed file system, a storage system providing a data persistence method, and the like), a software architecture of a log merge tree Key-Value storage system based on a hybrid storage architecture accesses KV data (SSTable) in the persistent storage system in a Key-Value storage system (Key-Value store) manner, an Application program (Application) accesses the log merge tree Key-Value storage system through a KV interface to invoke a required storage service, and the log merge tree Key-Value storage system uses NVM through a lightweight NVM space management mechanism. Wherein the persistent Storage system runs on a persistent Storage medium (Storage).

In one embodiment of the application, a log merge tree key value storage system based on a hybrid storage architecture is applied to an electronic device, the hybrid storage architecture comprises a host, a persistent storage medium and a nonvolatile storage medium, the host is connected with the nonvolatile storage medium, and the host is connected with the persistent storage medium; the compression mechanism of the log merge tree key-value storage system may include a packet compression mechanism; the log merge tree key value storage system comprises two-stage filters, wherein a persistent storage medium stores a plurality of ordered character string surface layers, each ordered character string surface layer comprises a plurality of ordered character string table groups, each ordered character string table group comprises a plurality of ordered character string tables, and the two-stage filter corresponding to each ordered character string table group comprises a group filter corresponding to the ordered character string table group and a table filter corresponding to the ordered character string tables in the ordered character string table group.

In one embodiment of the present application, the two-stage filter is stored in a memory, a non-volatile storage medium, and/or a persistent storage medium of the electronic device. Wherein the two-stage filter may be stored in a non-volatile storage medium, optionally a persistent storage medium of the electronic device.

In one embodiment of the present application, the Group Filter may include a bloom Filter or a valley Filter, and the table Filter may include a bloom Filter, a quotient Filter or a valley Filter. Each ordered set of string tables in the persistent storage media corresponds to a group filter, which is a hash structure used to determine whether an object exists in the corresponding ordered set of string tables.

Fig. 6 is a schematic structural diagram of a two-stage filter according to an embodiment of the present disclosure. As shown in fig. 6, two-stage filters are stored in the NVM, respectively: one two-stage filter corresponding to ordered string table Group 1(Group1), and the other two-stage filter corresponding to ordered string table Group 2(Group 2). The dashed lines represent selective storage. The two filters stored in Storage may be selectively stored, that is, Storage may store two-stage filters shown by dotted lines, or Storage may not store two-stage filters shown by dotted lines. A two-stage filter corresponding to an ordered character string table group includes group element data corresponding to the ordered character string table group, table metadata (SSTable metadata) corresponding to the ordered character string tables in the ordered character string table group. Alternatively, the metadata may include key ranges, or the metadata may include key ranges and filters. For example, the metadata of the ordered string table group is the group metadata, the metadata of the ordered string table is the table metadata, the group metadata of an ordered string table group may include the group key range of the ordered string table group, and the table metadata of an ordered string table may include the table key range of the ordered string table. For another example, the group metadata for an ordered string table group may include a group key range and a group filter for the ordered string table group, and the table metadata for an ordered string table may include a table key range and a table filter for the ordered string table.

Although the packet compression mechanism maintains the orderliness of part of the SSTable, when judging whether the target key of the data to be searched exists in the ordered character string table group, all SStables in the ordered character string table group need to be accessed. In order to reduce the read amplification of the packet compression mechanism, the embodiment of the application constructs a two-stage filter structure comprising a group filter and a table filter by establishing the group filter of the ordered character string table group on the NVM, reduces invalid reads of the ordered character string table group through the group filter, and reduces the access of read operations to persistent storage media by storing the two-stage filter in the NVM.

In an embodiment of the application, the log merge tree key-value storage system based on the hybrid storage architecture creates a filter (i.e., a table filter) corresponding to all keys in a table for each ordered string table, and creates a filter (i.e., a group filter) corresponding to all keys in a group for each ordered string table group; a two-stage filter is composed of a group filter of an ordered character string table group and a table filter corresponding to an ordered character string table in the ordered character string table group, wherein one two-stage filter can comprise one or more table filters.

Understandably, the group filter of the ordered character string table group and the table filter corresponding to the ordered character string table in the ordered character string table group constitute a two-stage filter structure.

Fig. 7 is a schematic view illustrating access to a two-stage filter according to an embodiment of the present invention. As shown in fig. 7, the group filter in the two-stage filter is a 4-slot valley filter, and the table filter in the two-stage filter is a bloom filter located with 3 hash functions. The valley filter and the bloom filter are only examples to illustrate the access to the two-stage filter, and the practical application is not limited to this.

This embodiment describes only the access flow of the two-stage filter, without considering the selection process of SSTable in the ordered set of string tables. The target key 45 for the data to be looked up is known to hit in the 2 nd slot of index 2 in the group filter and is accessed to the location of

indices

1, 2, 16 in the table filter. The specific process of finding the target key 45 may include:

one hash value 2 is calculated by a first hash function given by the group filter, another hash value 3 is calculated by a second hash function given by the group filter, 4 slots of the group filter index 2 are traversed, and the second slot is determined as a corresponding item by the hash value in the slot, so that it is determined that the target key 45 is likely to be in the ordered string table group (i.e., the ordered string table group has a possibility of existence of the target key 45). If no slot is a corresponding item after the traversal is finished, the target key 45 is not in the ordered character string table group, so that the search process of SSTable files in the ordered character string table group is omitted, and invalid read operation is reduced.

Three

hash values

1, 2, and 16 are respectively calculated by three hash functions given by SSTable, and the corresponding index position is accessed, wherein the index 2 corresponding value is 0, so that the target key 45 is not judged to be in the ordered character string table group. If any index among the plurality of indexes corresponds to a value of 0, it can be determined that the target key does not exist in the ordered character string table group.

The embodiment of the application provides a data searching method of a log merge tree key value storage system based on a hybrid storage architecture, which is applied to electronic equipment, and specifically, the data searching method comprises the following steps:

101, obtaining a target key of data to be searched.

The data to be searched can be key value data, including a target key and a value corresponding to the target key.

102, searching a target key from the memory of the electronic equipment.

The log merge tree key value storage system can adopt cache mechanisms such as BlockCache and TableCache.

The target key can be first looked up from memory caches such as MemTable, immutale MemTable, etc.

And 103, if the target key is not searched in the memory of the electronic equipment, searching the target key from the persistent storage medium according to the two-stage filter.

The persistent storage medium may include an SSD, and if the target key is not found in the memory, the target key is found from the SSD according to the two-stage filter. Further, the searched target key or the search result may be returned in response to the query request.

In one embodiment of the present application, finding a target key from a persistent storage medium according to a two-stage filter comprises: according to the layer sequence, for any current ordered character string surface layer in the ordered character string surface layers, judging whether a target ordered character string table group for storing a target key exists in the current ordered character string surface layer or not according to the group key range of a plurality of ordered character string table groups in the current ordered character string surface layer; and if a target ordered character string table group for storing the target key exists in the current ordered character string surface layer, searching the target key from the target ordered character string table group according to the two-stage filter.

Wherein, the layer sequence may include the sequence of the layer sequence number, etc. For example, the target key lookup may be performed in order from the 0 th layer to the nth layer of the log merge tree. Before searching the target key in the current ordered character string table layer, whether the target ordered character string table group storing the target key exists in the current ordered character string table layer can be judged by adopting a searching method according to the group key range of a plurality of ordered character string table groups in the current ordered character string table layer. The search method may include a binary search method or a sequential search method, etc.

In an embodiment of the present application, if a target ordered character string table group storing a target key does not exist in a current ordered character string surface layer, the target key is searched from a next ordered character string surface layer of the current ordered character string surface layer.

In one embodiment of the present application, finding a target key from a target ordered set of strings according to a two-stage filter comprises: judging whether the target ordered character string table group has the possibility of including the target key or not according to the group filter in the two-stage filter (namely judging whether the target ordered character string table group possibly stores the target key or not); if the target ordered character string table group is determined to have the possibility of comprising a target key (namely the target key is determined to be possibly stored in the target ordered character string table group) according to the group filter in the two-stage filter, judging whether the target ordered character string table for storing the target key exists in the target ordered character string table group or not (based on a searching method) according to the table key range of a plurality of ordered character string tables in the current ordered character string table group; if a target ordered character string table for storing a target key exists in the target ordered character string table group according to the table key range of a plurality of ordered character string tables in the current ordered character string table group, judging whether the target ordered character string table has the possibility of including the target key (namely judging whether the target ordered character string table is possible to store the target key) according to the table filters in the two-stage filters; and if the target ordered character string table is determined to have the possibility of comprising the target key according to the table filter in the two-stage filter (namely the target key is determined to be possibly stored in the target ordered character string table), searching the data to be searched in the target ordered character string table according to the target key.

If the target ordered character string table group is judged to possibly store the target key according to the group filter, the target ordered character string table group really stores the target key or does not store the target key; and if the target ordered character string table group is judged to be impossible to store the target key according to the group filter, the target ordered character string table group does not really store the target key.

According to the method and the device, the target key can be searched from the target ordered character string table group according to the two-stage filter, the possibility that the target ordered character string table group comprises the target key can be judged firstly, if the target ordered character string table group comprises the possibility of the target key, the target ordered character string table group has higher probability of comprising the target key, then the ordered character string table in the target ordered character string table group is further judged, and finally whether the target key is included in the target ordered character string table group or not is accurately judged. By utilizing the two-stage filter, invalid searching can be avoided, so that the searching efficiency of the target key is improved.

In an embodiment of the present application, if it is determined that the target ordered string table set does not have the possibility of including the target key according to the set filter in the two-stage filter, or it is determined that the target ordered string table does not have the possibility of including the target key according to the table filter in the two-stage filter, the target key is searched from the next ordered string surface layer of the current ordered string surface layer.

(II) in one embodiment of the present application, the target key is looked up from multiple ordered character string surface layers in the persistent storage medium according to a two-stage filter in order of layer number. For example, the target key lookup may be performed in order from the 0 th layer to the nth layer of the log merge tree.

In one embodiment of the present application, for any current ordered string surface layer of the plurality of ordered string surface layers, a target key is looked up from the current ordered string surface layer according to a two-stage filter.

Specifically, judging whether a target key exists in the surface layer of the current ordered character string according to a two-stage filter; if a target key exists in the current ordered character string surface layer according to the two-stage filter, searching data to be searched according to the target key; and if the target key does not exist in the current ordered character string surface layer according to the two-stage filter, searching the target key from the next ordered character string surface layer of the current ordered character string surface layer.

In an embodiment of the present application, determining whether a target key exists in a current ordered character string surface layer according to a two-stage filter may include: and judging whether a target ordered character string table group exists in the current ordered character string surface layer or not according to a group filter in the two-stage filter, wherein the target ordered character string table group has the possibility of comprising a target key (namely the target key is possibly stored in the target ordered character string table group). And if the target ordered character string table group does not exist in the current ordered character string surface layer, determining that the target key does not exist in the current ordered character string surface layer. If a target ordered character string table group exists in the current ordered character string surface layer, judging whether the target ordered character string table group exists in a target ordered character string table according to a table filter in the two-stage filter, wherein the target ordered character string table has the possibility of comprising a target key; and if the target ordered character string table does not exist in the target ordered character string table group, determining that the target key does not exist in the current ordered character string surface layer. If the target ordered character string table exists in the target ordered character string table group, judging whether a target key exists in the target ordered character string table or not; and if the target key exists in the target ordered character string table, determining that the target key exists in the current ordered character string surface layer.

Optionally, judging whether a target ordered character string table group exists in the current ordered character string surface layer according to a group filter in the two-stage filter, where the possibility that the target ordered character string table group includes a target key includes: acquiring the group key range of a plurality of ordered character string table groups in the current ordered character string surface layer; determining one or more ordered character string table groups of which the group key range comprises a target key from a plurality of ordered character string table groups in the current ordered character string surface layer to obtain a determined ordered character string table group; judging whether the determined ordered character string table group has the possibility of including the target key or not through a group filter of the determined ordered character string table group; if the determined ordered character string table group has the possibility of comprising a target key (namely the target key is possibly stored in the ordered character string table group), determining that the target ordered character string table group exists in the current ordered character string surface layer, and taking the determined ordered character string table group as the target ordered character string table group; and if the determined ordered character string table group does not have the possibility of comprising the target key (namely, the ordered character string table group cannot store the target key), determining that the target ordered character string table group with the possibility of comprising the target key does not exist in the current ordered character string surface layer.

Specifically, one or more ordered character string table groups of which the group key range includes the target key can be determined from a plurality of ordered character string table groups in the current ordered character string surface layer based on a binary search method. The key combination ranges of the ordered character string table groups comprise the minimum key and the maximum key in the ordered character string table groups, and the key combination ranges among the ordered character string table groups are not overlapped, so that one or more ordered character string table groups possibly comprising the target key can be found based on a binary search method.

Understandably, in the searching process, for the current ordered character string table group in the current ordered character string surface layer, whether a target key possibly exists in the current ordered character string table group can be judged according to a filter of the current ordered character string table group; if the judgment is impossible, the search of the current ordered character string table group can be skipped, and the subsequent search is carried out; if the judgment is possible, in order to ensure that the target key really exists in the current ordered character string table group, further judgment needs to be made through a table filter of the ordered character string table in the current ordered character string table group, so that the misjudgment rate is reduced, the access times of the persistent storage medium are reduced, and the reduction of the reading performance caused by a compression mode is reduced.

Optionally, judging whether a target ordered character string table group exists in the current ordered character string surface layer according to a group filter in the two-stage filter, where the possibility that the target ordered character string table group includes a target key includes: acquiring ordered character string table groups in the current ordered character string surface layer one by one, and judging whether the ordered character string table groups have the possibility of including a target key or not through a group filter of the ordered character string table groups; if any one ordered character string table group in the current ordered character string surface layer has the possibility of comprising the target key, determining that the ordered character string table group with the possibility of comprising the target key exists in the current ordered character string surface layer, and taking the ordered character string table group with the possibility of comprising the target key as the target ordered character string table group; and if all the ordered character string table groups in the current ordered character string surface layer do not have the possibility of including the target key, determining that the target ordered character string table group with the possibility of including the target key does not exist in the current ordered character string surface layer.

Optionally, judging whether a target ordered string table exists in the target ordered string table group according to a table filter in the two-stage filter, where the possibility that the target ordered string table includes a target key includes: acquiring table key ranges of a plurality of ordered character string tables in a target ordered character string table group; determining one or more ordered character string tables of which the table key range comprises a target key from a plurality of ordered character string tables in the target ordered character string table group to obtain a determined ordered character string table; judging whether the determined ordered character string table has the possibility of comprising the target key or not through the table filter of the determined ordered character string table (namely judging whether the determined ordered character string table possibly has the target key or not through the table filter of the determined ordered character string table); if the determined ordered character string table has the possibility of comprising the target key (namely the determined ordered character string table may have the target key), determining that the ordered character string table having the possibility of comprising the target key exists in the target ordered character string table group, and taking the determined ordered character string table as the target ordered character string table; if the determined ordered character string table does not have the possibility of including the target key (namely the determined ordered character string table cannot have the target key), determining that the target ordered character string table having the possibility of including the target key does not exist in the target ordered character string table group.

Specifically, whether the target key may exist in the determined ordered character string table may be judged by a table filter of the determined ordered character string table according to the time sequence of the determined ordered character string table. For example, it is determined whether the target key may exist in table 1 first and then it is determined whether the target key may exist in table 2, wherein the generation time of table 1 is later than that of table 2.

Optionally, judging whether a target ordered string table exists in the target ordered string table group according to a table filter in the two-stage filter, where the possibility that the target ordered string table includes a target key includes: acquiring ordered character string tables in the target ordered character string table group one by one, and judging whether the ordered character string tables have the possibility of including the target key or not through a table filter of the ordered character string tables; if any one ordered character string table in the target ordered character string table group has the possibility of comprising the target key, determining that the target ordered character string table with the possibility of comprising the target key exists in the target ordered character string table group; and if all the ordered character string tables in the target ordered character string table group do not have the possibility of including the target key, determining that the target ordered character string table having the possibility of including the target key does not exist in the target ordered character string table group.

The embodiment of the application provides another data searching method of a log merge tree key value storage system based on a hybrid storage architecture, which is applied to electronic equipment, and specifically, the data searching method comprises the following steps: acquiring a target key of data to be searched; and searching the target key from the persistent storage medium according to the two-stage filter.

As shown in fig. 8, a flowchart of a data search method of a log merge tree key value storage system based on a hybrid storage architecture is provided in the embodiment of the present application. The electronic equipment judges whether the target key is found in the memory cache; if the target key is found in the memory cache, returning the target key value pair corresponding to the found target key; if the target key is not found in the memory cache, initializing the layer of the current ordered character string to be 0; in the current ordered character string table layer, searching an ordered character string table group comprising a target key by adopting a binary search method; judging whether the searched ordered character string table group contains a target key or not through a group filter in the NVM; if the searched ordered character string table group does not contain the target key through the group filter in the NVM, adding 1 to the layer of the current ordered character string table; judging whether the hierarchy of the current ordered character string surface layer is larger than or equal to the preset highest hierarchy; if the layer of the current ordered character string surface layer is larger than or equal to the preset highest layer, returning a search result without a target key; and if the hierarchy of the current ordered character string surface layer is smaller than the preset highest hierarchy, returning the process, and searching the ordered character string table group comprising the target key in the next layer of the current ordered character string surface layer by adopting a binary search method. If the searched ordered character string table group comprises the target key through the group filter in the NVM, searching the target key through the metadata of the ordered character string table in the ordered character string table group in the NVM and the table filter; if the ordered character string table in the ordered character string table group contains the target key, returning a target key value pair corresponding to the target key value; and if the ordered character string table in the ordered character string table group does not contain the target key, returning the process to the step of adding 1 to the hierarchy of the current ordered character string surface layer.

Fig. 9 is a schematic diagram of data search of a log merge tree key value storage system based on a hybrid storage architecture according to an embodiment of the present application. Wherein the dashed lines represent selective storage.

The target key for searching is known to be 45, the range of the MemTable key value is 3-15, the range of the Immutable key value is 1-10, the files on the SSD are divided into two layers, and each layer comprises a plurality of groups. Wherein the first two groups of the zero layer respectively comprise three SStables, the key value ranges of the two groups are respectively 1-30 and 31-60, and the target key 45 does not exist in the second group. The key values of the first two groups of the first layer are respectively 1-40 and 41-80, the first SSTable in the second group does not contain a target key, the second SSTable and the third SSTable in the second group both contain a target key 45, the third SStables in the second group are arranged from top to bottom according to a time sequence, and the generation time of the first SSTable in the second group is later than that of the third SSTable in the second group. The NVM stores metadata corresponding to the four ordered string tables, respectively. The search target key 45 specifically includes:

(1) the target key 45 is searched for by the MemTable and the executable MemTable in the memory, and it can be judged that 45 does not exist in the cache of the memory according to the key value range.

(2) A lookup is made for SSTable in the SSD. Firstly, searching in L0, and according to the group key value range stored in the group data of the L0 layer, performing binary search to determine that the data exists in the second group of the L0 layer.

(3) The second group of the L0 layer comprises three SStables, and the key value ranges are 31-41, 33-43 and 47-56 respectively. Before accessing a particular SSTable, access to the next level of files is continued by the group filter determining that the target key 45 does not exist in the second group at level L0.

(4) And determining that the target key 45 exists in the key value range where the second Group of the L0 layer is located in the L0 layer through a binary search method, and determining that the target key 45 exists in the first Group of the L1 layer and the second Group of the L1 layer through a Group index strategy according to information stored in the second Group of the L0 layer. That is, the range of key values in which the second group of the L0 layer is located overlaps with the range of key values in the first group of the L1 layer and the second group of the L1 layer, it is determined that the target key 45 may exist in the first group of the L1 layer and the second group of the L1 layer.

(5) And performing binary search in the first group of the L1 layer and the second group of the L1 layer according to the group key value range saved by the group metadata, and determining that the data exists in the second group of the L1 layer.

(6) The second group of the L1 layer comprises three SStables, and the key value ranges are 41-60, 44-55 and 45-75 respectively. Prior to accessing a particular SSTable, it is determined by the group filter that the target key 45 is present in the second group at the L1 level.

(7) According to the file of which the key range contains the target key 45 selected from the metadata information of the L1 layer, the key value ranges of the second group of three SStables of the L1 layer all contain the target key.

(8) The second group of three sstables of the L1 layer is secondarily filtered in sequence according to the generation time, and the secondary filtering can be performed from late to early according to the generation time.

(8-1) the presence or absence of the target bond 45 is judged by using the filter of the first SSTable of the second group of the L1 layer. If the key is judged to exist, an SSTable reading interface of RocksDB is used for accessing SSTable, a target key is searched in SSTable, and subsequent searching is carried out when the target key is not found finally; and if the judgment result is not present, directly carrying out subsequent searching.

(8-2) judging the existence of the target key 45 by using a second group of second SSTable filters of the L1 layer, searching for a key value pair (data to be searched) corresponding to the target key in the second SSTable, and returning.

The embodiment of the application provides a filter updating method of a log merge tree key value storage system based on a hybrid storage architecture, which is applied to electronic equipment, and specifically the filter updating method comprises the following steps:

and 201, determining any one of the ordered character string surface layers as a current ordered character string surface layer, wherein the ordered character string surface layers are stored according to a preset sequence.

202, obtaining the number of the ordered character string tables in the current ordered character string surface layer and the upper limit number of the preset tables.

And 203, if the number of the ordered character string tables in the current ordered character string surface layer is greater than or equal to the upper limit number of the preset table, selecting an ordered character string table group from the current ordered character string surface layer.

In one embodiment of the present application, selecting the ordered set of strings from the current ordered string table layer comprises: and selecting the ordered character string table group from the current ordered character string table layer according to the number and the generation time of the ordered character string tables.

Specifically, the ordered character string table group with the largest number of ordered character string tables and the earliest average generation time can be selected from the current ordered character string table layer.

And 204, generating an ordered character string table to be merged according to the selected ordered character string table group.

In one embodiment of the present application, generating the ordered string table to be merged according to the selected ordered string table group includes: merging and sorting the ordered character string tables in the selected ordered character string table group to obtain a first ordered character string table; and dividing the first ordered character string table into at least one second ordered character string table according to the key range of the next ordered character string surface layer of the current ordered character string surface layer to obtain the ordered character string table to be merged, which comprises at least one second ordered character string table.

In one embodiment of the present application, dividing the first ordered string table into at least one second ordered string table according to a key range of a next ordered string table of the current ordered string table comprises: if the table key range of the first ordered character string table is contained in any group key range of the next ordered character string surface layer of the current ordered character string surface layer, the first ordered character string table is not divided; if the table key range of the first ordered character string table is contained in any plurality of group key ranges of the next ordered character string surface layer of the current ordered character string surface layer, dividing the first ordered character string table according to any plurality of group key ranges; and if the table key range of the first ordered character string table is not contained in any group key range of the next ordered character string surface layer of the current ordered character string surface layer, uniformly dividing the first ordered character string table.

In one embodiment of the present application, the size of the first ordered string table is consistent with the size of the ordered string table of the next ordered string table to the current ordered string table. Table filters and table metadata corresponding to the first ordered string table are created on the NVM.

And 205, merging the to-be-merged ordered character string table to the next ordered character string surface layer of the current ordered character string surface layer to obtain a merged ordered character string surface layer.

In one embodiment of the present application, merging the to-be-merged ordered string table to a next ordered string table of the current ordered string table comprises: and in the next ordered character string surface layer of the current ordered character string surface layer, merging each second ordered character string table in the ordered character string tables to be merged into an ordered character string table group with the group key range containing the table key range of the second ordered character string table through multithreading.

And 206, updating the two-stage filter of the next ordered character string surface layer of the current ordered character string surface layer according to the merged ordered character string surface layer.

In one embodiment of the present application, the two-stage filter for updating the next ordered string surface layer of the current ordered string surface layer according to the merged ordered string surface layer comprises: generating corresponding group element data according to the ordered character string table group generated in the combined ordered character string surface layer, and creating a group filter corresponding to the generated ordered character string table group in a nonvolatile storage medium; and updating a group filter corresponding to the modified ordered character string table group in the nonvolatile storage medium according to the modified ordered character string table group in the merged ordered character string table layer.

The filter updating method of the embodiment updates the metadata and the filter in place on the NVM, thereby reducing the overhead of write amplification and recovery of the storage space.

Fig. 10 is a flowchart for updating a filter according to an embodiment of the present application. Determining that the surface layer of the current ordered character string is a Li layer, and selecting an ordered character string table group (combination group) of the Li layer to a memory; merging the ordered character string table groups in the memory and dividing (segmenting) the ordered character string table groups according to the Li layer; generating a new ordered character string table and a table filter, and writing the new ordered character string table and the table filter into a corresponding ordered character string table group of the Li +1 layer; and updating the metadata and the two-stage filter in the NVM in situ (namely updating the metadata and the two-stage filter corresponding to the ordered character string table group in the NVM according to the new ordered character string table group).

Specifically, the Li layer can be read to participate in SSTable to memory, which compresses to the Li +1 layer, for merge and split operations. One group with the largest number of Li layer files and the earliest total file time sequence (the total file time sequence is earliest, namely the average file generation time is earliest) in the SSD can be selected, all SSTable in the group (namely the ordered character string table group) are read into a memory for merging and sorting, the merged SSTable is divided according to the group division of the next layer, the key value ranges needing to be written to a target level after the division is finished cannot be overlapped, and the SSTable is not divided if the key value ranges are not overlapped with a plurality of groups. And after generating a new SSTable and a corresponding filter, selecting a group corresponding to the next layer for writing.

In particular, the metadata and filter information may be updated. Because the metadata and the filters of the SSTable and the metadata and the filters of the SSTable group are stored in the NVM, the metadata and the two-stage filters only need to be updated in place synchronously after the merging operation is completed, and Copy-on-Write (Copy-on-Write) technology is not needed to be used for multi-version.

Fig. 11 is a schematic diagram illustrating filter updating according to an embodiment of the present application. Wherein the dashed lines represent selective storage.

First, Memtable and Immunable Memtable in memory are converted, and then the compression process is performed from L0 to L1. Assuming that the files on the SSD are organized in two layers in total, the zeroth layer contains two groups, each containing three SStables, the key values of the two groups range from 1-30 and 31-60, respectively, where the first two SStables of the first group of L0 are generated earlier than the first two SStables of the second group of L0, respectively, and the last SStables of the first group of L0 are generated later than the last SStables of the second group of L1. The first layer comprises a group, the key value range of the group is 1-60, the group comprises SSTable, and the key value range is 10-55. The filter updating method specifically comprises the following steps:

(1) since the two sets of SSTable files of the L0 layer are the same in number, and the two SStables of the set 1 are earlier than the two SStables of the set 2, the set 1 with the earliest overall file timing sequence is selected for compression, and all the SStables in the set 1 are read into the memory for merging operation.

(2) And merging and sequencing all the SSTable read into the memory, and generating new SSTable files, filters and metadata.

(3) The new SSTable is split according to the range of group key values of the L1 layer. Because one group is arranged in the L1 layer, the range of key values is 1-60, the range of key values of the new SSTable file is 1-28, and the new SSTable file key values are in the group key value range of the ordered character string table group of the L1 layer, the new SSTable is directly written into the ordered character string table group of the L1 layer without being divided.

(4) The two-stage filter and metadata information in the NVM is updated synchronously.

The embodiment of the application provides a nonvolatile storage medium management method of a log merge tree key value storage system based on a hybrid storage architecture, which is applied to electronic equipment, and specifically the nonvolatile storage medium management method comprises the following steps:

301, a request to create a table filter or group filter is received.

302, convert the create request to a space allocation request for the non-volatile storage media.

303, obtaining a lock of an idle linked list of the nonvolatile storage medium, where the idle linked list includes a page number of an idle page of the nonvolatile storage medium.

And 304, extracting a plurality of page numbers from the idle linked list.

A storage management object is created 305 on the non-volatile storage medium based on the plurality of page numbers.

306, return the storage management object.

307, the lock of the free linked list is released.

In one embodiment of the present application, the memory space of the NVM is organized by page. The head page stores the whole information of the NVM page; the metadata page stores information of the NVM page; the data pages store specific data. The data pages may be managed through a bitmap (bitmap) and an idle linked list, maintaining currently used page information and idle page information. For the light-weight file calling use mode of the NVM package, the paging condition of the bottom NVM of the upper package is provided by providing a calling interface similar to a file system, so that the use of the NVM and the use of the memory are as convenient and fast, and the allocation and the recovery of the NVM are not required to be processed.

The allocation request of the embodiment adopts a short circuit mechanism to complete allocation directly through a bottom linked list, so that the NVM is ensured to complete response before the requesting thread uses the requested space.

The nonvolatile storage medium management method of this embodiment specifically includes a mutually exclusive application method.

FIG. 12 is a flowchart of a mutually exclusive application method provided by the embodiment of the present application. Specifically, the mutually exclusive application method may include: an allocation request (storage request) is received. Mutex locking is performed. And acquiring first _ free _ page _ num from the NVMPager and storing the first _ free _ page _ num into tmp, wherein page information is managed and described on the NVM through the NVMPager, the tmp is an intermediate variable, and the first _ free _ page _ num is a page number of a first free page in the free linked list. Then, the number of idle pages cnt is initially scanned to 0, and tmp! 0& & cnt < number of pages desired to be allocated is true. If tmp! 0& & cnt < the number of pages desired to be allocated is true, tmp is set to next _ page _ num of the page corresponding to tmp (page number of the next page), cnt is incremented by 1, and the process returns to the judgment of tmp! 0& & cnt < number of pages desired to be allocated is true. If tmp! If 0& & cnt < the number of pages desired to be allocated is false, it is determined whether cnt is equal to the number of pages desired to be allocated. If cnt is equal to the number of the pages expected to be allocated, modifying first _ free _ page _ num in NVMPager to be next _ page _ num of the page corresponding to tmp, and setting the next _ page _ num of the tmp page to be 0. Further, mutex unlocks and the dispensing is successful. If cnt is not equal to the number of pages expected to be allocated, mutex is unlocked and allocation fails.

The application method is further explained below.

a. A new filter is created in the NVM during the tail phase of the merge (compression) process.

The merge operation creates a new SSTable on the next ordered string table layer of the current ordered string table, and a corresponding new SSTable filter is created, requiring access to the NVM.

b. The NVM lightweight space management layer translates the create request to the group filter into a store request to the NVM space.

The NVM lightweight space management system comprises an NVM lightweight space management layer, which converts the creation request of the group filter into an application for NVM pages, abstracts the page set storing the filter, which needs to be accessed, into a file (the file is a continuous space in a logical sense, similar to the "file" of a general file system, in the following embodiment, the file related to the nonvolatile storage medium management method or the file related to the file service of the nonvolatile storage medium is a continuous space in a logical sense), and constructs an NVMFile handle in the memory when the file is opened or created, and the NVM space can be accessed and used through the NVMFile structure. The core members of the handle are the page number of the page occupied by the group filter on the NVM, and the offset of a logic offset value representing the reading and writing of the file handle, which is similar to the setting of the reading and writing positions of the file in the lseek method in the file operation. The NVMFile calculates the actual read/write address (the specific position of the specific page) on the NVM according to the logical offset value, and then directly and accurately accesses the corresponding content on the NVM in the user logical space by using the first address, which is mapped to the user logical space by mmap, of the NVM as the base address.

Filter construction, reading, etc. operations can be implemented on the NVM via NVMFile, shielding the underlying NVM paging case. The create operation on the group filter is converted to a create operation on NVMFile and then further converted to a scan of NVM free pages and an apply operation on new pages.

c. Acquiring a lock of the NVM idle linked list, picking a certain amount of pages from the idle page linked list, and creating an NVMemHandler object on the NVM by using the page numbers.

The NVM free-list is a lock-free list, but the use of mutex locks makes the NVM space allocation not allow allocation-allocation parallelism, not allow allocation-reclamation parallelism, and allow reclamation-reclamation parallelism, thereby ensuring the correctness of concurrent allocation and reclamation. The reason for not allowing allocation-allocation parallelism is to consider that each allocation operation requires a required number of pages to be extracted from the idle linked list at once, but lock-free concurrent linked lists that delete multiple elements at once are difficult to implement.

Nvmemhandles are used to describe logically contiguous NVM pages for managing and accessing NVM pages referenced by an NVM file.

d. And releasing the lock of the idle linked list and returning to the created NVMMemHandler.

The entire application for NVM space is in the locked phase. After acquiring the NVM space and creating the NVMMemHandler, releasing the lock of the NVM free page linked list.

The embodiment of the application provides another nonvolatile storage medium management method of a log merge tree key value storage system based on a hybrid storage architecture, which is applied to electronic equipment, and specifically, the nonvolatile storage medium management method includes:

in response to a data failure of a table filter or a group filter, a reclamation request for space stored by the non-volatile storage media is triggered 401.

402, add the recycle request to the recycle queue and notify the dispatch thread of the recycle request.

And 403, creating a plurality of recycle threads according to the recycle requests in the recycle queue by the scheduling thread, so as to recycle the failure pages by the plurality of recycle threads.

And 404, updating an idle linked list according to the failure page, wherein the idle linked list comprises the page number of the idle page of the nonvolatile storage medium.

405, wait for the recycle thread to end.

In one embodiment of the present application, updating the idle linked list according to the stale page includes: acquiring a header page number of an idle linked list; assigning the header page number to the next page number of the failure page; and re-determining the page number of the failed page as the head page number. Specifically, the first _ free _ page _ num (header page number, i.e. page number of the first NVMPageDescriptor (node) in the NVMPager) may be obtained by atomic load; assigning the next _ page _ num (next page number) of the NVMPageDescriptor (failure page) needing to be recycled as first _ free _ page _ num; if the first _ free _ page _ num is not changed through the atomic operation CAS, the page number of the NVMPageDescriptor is determined as the first _ free _ page _ num again.

The nonvolatile storage medium management method of this embodiment specifically includes a concurrent recovery method.

For recycle requests, after the requesting thread submits the sequence of page numbers that need to be freed, it may not be necessary to ensure that NVM space management returns the specified page, and the requestor may continue to execute other processing logic while avoiding accessing the address space of the freed page.

Fig. 13 is a flowchart of a concurrent recovery method provided in the embodiment of the present application. Specifically, the concurrent recovery method may include: receiving a recovery request; acquiring a request queue lock _ guard; inserting a recycle request into a request queue; notify _ one informs the NVM of scheduling thread; acquiring a request queue lock _ guard by an internal scheduling thread of the NVMPager; and the NVMPager internal scheduling thread judges whether the request queue is empty. Wait to wake up by cv. The NVMPager internal scheduling thread responds to the notice of notify _ one to wake up. And if the request queue is not empty, popping a part of the recovery requests in the current queue by the NVMPager internal scheduling thread. The NVMPager internal scheduling thread automatically unlocks lock _ guard or unlocks lock _ guard in response to an unlock request. Then, mutex locking; the NVMPager internal scheduling thread creates a plurality of threads to execute the recovery thread task; multiple threads of the thread join are dispatched inside the NVMPager to complete the task of the recovery thread; mutex unlocks, and execution judges whether the request queue is empty. The recycle execution thread atom load acquires first _ free _ page _ num in NVMPager. The recovery execution thread sets next _ page _ num in the NVMPageDescriptor to be recovered to first _ free _ page _ num. And the recycling execution thread atomic operation CAS judges whether the current idle chain table head is first _ free _ page _ num or not, and if the current idle chain table head is first _ free _ page _ num, the idle chain table head is set as the page number to be recycled. The reclaim execution thread determines whether the CAS operation updates the free link header. If yes, ending the process; if not, executing the atomic load to acquire first _ free _ page _ num in the NVMPager.

The recovery process is further explained below.

e. And (3) the metadata such as a filter and the like fails due to the merging (compression) process, and the recovery request of the NVM file is triggered.

When one SSTable is compressed and fails, the number of the page where the NVMemHandler is located is obtained, the offset value is calculated through the page number, base address addressing is carried out on the NVM base address according to the offset value, the NVMemHandler on the NVM can be accessed, and the NVM page can be conveniently and safely recovered.

f. And adding the recycle request into a recycle queue and informing the internal scheduling thread.

Operations on the reclaim queue all require acquiring a lock. The recycle request with the most released space is preferentially processed in the recycle requests, so that the situation that the allocation requests cannot be met when the number of the allocable pages of the NVM is small is reduced.

g. And popping up a plurality of recovery requests by the internal scheduling thread, creating a plurality of work thread recovery pages and updating the idle linked list.

When a recovery request comes, the NVMPager pops up the request in the waiting queue, and when the recovery request is less than a preset frequency (under a general condition), an internal scheduling thread of the NVMPager can directly create a plurality of threads to concurrently execute the recovery request in the current waiting queue so as to concurrently execute a recovery operation; if the number of the recovery requests exceeds the preset thread number, the NVMPager merges the popped recovery requests into one recovery operation, and the recovered pages are returned once in the internal scheduling thread of the NVMPager.

g1, acquiring first _ free _ page _ num of the current whole NVM space.

g2, setting the next _ page _ num field of the page descriptor of the NVM page to be reclaimed to first _ free _ page _ num.

g3, setting the current free link header to first _ free _ page _ num through CAS atomic operation.

The CAS (compare And set) technique is a special atomic operation, And the general call form is CAS (V, E, N), where V is the address to be updated, determines whether the data in V is the same as E, And if so, sets V to N. The CAS technique ensures that no other thread operates successfully during one attempt and successfully inserts data into the linked list.

h. Waiting for all recycle threads to end.

Fig. 14 is a logical relationship diagram of a data structure of a management method for a non-volatile storage medium according to an embodiment of the present application. FIG. 14 shows the data structure logical relationship of the NVM lightweight space management layer. The NVM lightweight space management layer is a component of the NVM lightweight space management system, the page information is managed and described on the NVM through NVMPager and NVMPagerDescriptor, and the specific file service is provided through NVMmemHandler. NVM space is accessed and used in memory by NVMFile structures.

As shown in fig. 15, a schematic diagram is created for the filter provided in the embodiment of the present application. The call flow of the NVM lightweight space management system is described by the process of creating a group filter at the end of a compaction operation. In the end stage of the one compression process, filters are created in the NVM by the NVM lightweight space management system while new SSTable files are written to the SSD.

The NVM lightweight space management layer translates the creation of the group filter into an application for NVM space. And the NVMPager schedules the space allocation request under the condition of concurrency, extracts a certain number of pages from the linked list of the idle pages, and creates an NVMemHandler object on the NVM by using the page numbers of the pages. The created NVMemHandler object is passed back to the KV system (RocksDB). And creating the NVMFile object for subsequent use, and persisting the NVMemHandler to the SSD to prevent power loss.

Fig. 16 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device 160 includes a memory 1601, a processor 1602, and computer readable instructions, such as a log merge tree key-value storage system optimization program based on a hybrid storage architecture, stored in the memory 1601 and executable on the processor 1602. The processor 1602, when executing the computer readable instructions, implements the steps in the above described embodiment of the log merge tree key-value storage system optimization method based on the hybrid storage architecture.

Those skilled in the art will appreciate that the schematic diagram 16 is merely an example of a computer device 160 and is not intended to limit the computer device 160 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the computer device 160 may also include input output devices, network access devices, buses, etc.

The Processor 1602 may be a Central Processing Unit (CPU), or may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor 1602 may be any conventional processor or the like, and the processor 1602 is the control center for the computer device 160 and connects the various components of the overall computer device 160 using various interfaces and lines.

The memory 1601 may be used to store computer readable instructions and the processor 1602 implements various functions of the computer device 160 by executing or executing computer readable instructions or modules stored in the memory 1601 and invoking data stored in the memory 1601. The memory 1601 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the computer device 160, and the like. In addition, the Memory 1601 may include a hard disk, a Memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Memory Card (Flash Card), at least one disk storage device, a Flash Memory device, a Read-Only Memory (ROM), a Random Access Memory (RAM), or other non-volatile/volatile storage devices.

The modules integrated by the computer device 160 may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by the present application, and can also be realized by hardware related to computer readable instructions, which can be stored in a computer readable storage medium, and when the computer readable instructions are executed by a processor, the steps of the above described method embodiments can be realized. Where the computer readable instructions comprise computer readable instruction code, the computer readable instruction code may be in source code form, object code form, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer readable instruction code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, computer memory, Read Only Memory (ROM), Random Access Memory (RAM), or the like.

The embodiment also provides a computer storage medium, where a computer instruction is stored in the computer storage medium, and when the computer instruction runs on an electronic device, the electronic device is caused to execute the above related method steps to implement the log merge tree key value storage system optimization method based on the hybrid storage architecture in the above embodiment.

The embodiment also provides a computer program product, which, when running on an electronic device, causes the electronic device to execute the above related steps, so as to implement the log merge tree key value storage system optimization method based on the hybrid storage architecture in the above embodiment.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; when the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip can execute the log merging tree key value storage system optimization method based on the hybrid storage architecture in the above method embodiments.

The electronic device, the computer storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, so that the beneficial effects achieved by the electronic device, the computer storage medium, the computer program product, or the chip may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the module or unit is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed to a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A kind of log merge tree key value storage system, based on mixing the storage framework, apply to the electronic device, characterized by that:

the hybrid storage architecture comprises a host, a persistent storage medium and a nonvolatile storage medium, wherein the host is connected with the nonvolatile storage medium and the persistent storage medium, the persistent storage medium stores a plurality of ordered character string surface layers, each ordered character string surface layer comprises a plurality of ordered character string table groups, and each ordered character string table group comprises a plurality of ordered character string tables;

the log merge tree key value storage system adopts a grouping compression mechanism and comprises two stages of filters corresponding to the ordered character string table groups, wherein the two stages of filters comprise group filters corresponding to the ordered character string table groups and table filters corresponding to the ordered character string table.

2. The log merge tree key storage system of claim 1, wherein the two-stage filter is stored on the non-volatile storage medium.

3. The log merge tree key-value store system of claim 1, wherein the group filter comprises a bloom filter or a valley filter and the table filter comprises a bloom filter, a quotient filter, or a valley filter.

4. The log merge tree key storage system of claim 1, wherein the hybrid storage architecture further comprises a volatile storage medium or memory.

5. A data lookup method based on the log merge tree key-value storage system as claimed in any one of claims 1 to 4, the data lookup method comprising:

acquiring a target key of data to be searched;

and searching the target key from the persistent storage medium according to the two-stage filter.

6. The data lookup method as claimed in claim 5 wherein said looking up said target key from said persistent storage media in accordance with said two-stage filter comprises:

searching the target key from the memory of the electronic equipment;

and if the target key is not searched from the memory of the electronic equipment, searching the target key from the persistent storage medium according to the two-stage filter.

7. The data lookup method as claimed in claim 5, wherein after said obtaining the target key of the data to be looked up, the data lookup method further comprises:

searching the target key from the memory of the electronic equipment;

8. The data lookup method as claimed in claim 5 wherein said retrieving said target key from said persistent storage media in accordance with said two-stage filter comprises:

according to the layer sequence, for any current ordered character string surface layer in the ordered character string surface layers, judging whether a target ordered character string table group for storing the target key exists in the current ordered character string surface layer according to the group key range of a plurality of ordered character string table groups in the current ordered character string surface layer;

and if a target ordered character string table group for storing the target key exists in the current ordered character string surface layer, searching the target key from the target ordered character string table group according to the two-stage filter.

9. The data lookup method as claimed in claim 8 wherein if the target ordered string table set storing the target key does not exist in the current ordered string surface layer, the target key is looked up from a next ordered string surface layer of the current ordered string surface layer.

10. The data lookup method as claimed in claim 8 wherein said looking up said target key from said set of target ordered strings according to said two-stage filter comprises:

judging whether the target ordered character string table group has the possibility of comprising the target key or not according to the group filter in the two-stage filter;

if the possibility that the target ordered character string table group comprises the target key is determined according to the group filter in the two-stage filter, judging whether the target ordered character string table group stores the target key or not according to the table key range of a plurality of ordered character string tables in the current ordered character string table group;

if a target ordered character string table storing the target key exists in the target ordered character string table group according to the table key range of a plurality of ordered character string tables in the current ordered character string table group, judging whether the target ordered character string table has the possibility of including the target key according to the table filter in the two-stage filter;

and if the target ordered character string table is determined to have the possibility of comprising the target key according to the table filter in the two-stage filter, searching the data to be searched in the target ordered character string table according to the target key.

11. The data lookup method as claimed in claim 10 wherein if it is determined from said group filter in said two-stage filter that said target ordered string table group does not have a likelihood of including said target key, or if it is determined from said table filter in said two-stage filter that said target ordered string table does not have a likelihood of including said target key, then said target key is looked up from a next ordered string surface layer in said current ordered string surface layer.

12. The data lookup method as claimed in claim 5 wherein said looking up said target key from said persistent storage media in accordance with said two-stage filter comprises:

and searching the target key from the plurality of ordered character string surface layers in the persistent storage medium by using the two-stage filter according to the layer sequence.

13. The data lookup method as claimed in claim 12 wherein for any one of the plurality of ordered string surfaces, for a current ordered string surface, looking up the target key from the current ordered string surface according to the two-stage filter comprises: judging whether the target key exists in the current ordered character string surface layer or not according to the two-stage filter;

if the target key exists in the current ordered character string surface layer according to the two-stage filter, searching the data to be searched according to the target key;

and if the target key does not exist in the current ordered character string surface layer according to the two-stage filter, searching the target key from the next ordered character string surface layer of the current ordered character string surface layer.

14. The data lookup method as claimed in claim 13 wherein said determining whether the target key exists in the current ordered string surface according to the two-stage filter comprises:

judging whether a target ordered character string table group exists in the current ordered character string surface layer or not according to the group filter in the two-stage filter, wherein the target ordered character string table group has the possibility of comprising the target key;

if the target ordered character string table group does not exist in the current ordered character string surface layer, determining that the target key does not exist in the current ordered character string surface layer;

if the target ordered character string table group exists in the current ordered character string surface layer, judging whether a target ordered character string table exists in the target ordered character string table group or not according to the table filter in the two-stage filter, wherein the target ordered character string table has the possibility of including the target key; if the target ordered character string table does not exist in the target ordered character string table group, determining that the target key does not exist in the current ordered character string surface layer;

if the target ordered character string table exists in the target ordered character string table group, judging whether the target key exists in the target ordered character string table or not;

and if the target key exists in the target ordered character string table, determining that the target key exists in the current ordered character string surface layer.

15. The data lookup method as claimed in claim 14 wherein said determining whether a target ordered string table set exists in the current ordered string surface layer according to the set filter in the two-stage filter, the possibility that the target ordered string table set includes the target key comprises:

acquiring the group key range of a plurality of ordered character string table groups in the current ordered character string surface layer;

determining one or more ordered character string table groups of which the group key range comprises the target key from a plurality of ordered character string table groups in the current ordered character string surface layer to obtain a determined ordered character string table group;

judging whether the determined ordered character string table group has the possibility of comprising the target key or not through a group filter of the determined ordered character string table group;

if the determined ordered character string table group has the possibility of comprising the target key, determining that an ordered character string table group having the possibility of comprising the target key exists in the current ordered character string surface layer, and taking the determined ordered character string table group as the target ordered character string table group;

and if the determined ordered character string table group does not have the possibility of comprising the target key, determining that the target ordered character string table group which has the possibility of comprising the target key does not exist in the current ordered character string surface layer.

16. The data lookup method as claimed in claim 14 wherein said determining whether a target ordered string table set exists in the current ordered string surface layer according to the set filter in the two-stage filter, the possibility that the target ordered string table set includes the target key comprises:

acquiring ordered character string table groups in the current ordered character string surface layer one by one, and judging whether the ordered character string table groups have the possibility of including the target key or not through a group filter of the ordered character string table groups;

if any one ordered character string table group in the current ordered character string surface layer has the possibility of comprising the target key, determining that a target ordered character string table group with the possibility of comprising the target key exists in the current ordered character string surface layer;

and if all the ordered character string table groups in the current ordered character string surface layer do not have the possibility of including the target key, determining that no target ordered character string table group with the possibility of including the target key exists in the current ordered character string surface layer.

17. The data lookup method as claimed in claim 14, wherein said determining whether a target ordered string table exists in the target ordered string table group according to the table filter in the two-stage filter, the target ordered string table having a likelihood of including the target key comprises:

acquiring the table key ranges of a plurality of ordered character string tables in the target ordered character string table group;

determining one or more ordered character string tables of which the table key range comprises the target key from a plurality of ordered character string tables in the target ordered character string table group to obtain a determined ordered character string table;

judging whether the determined ordered character string table has the possibility of comprising the target key or not through a table filter of the determined ordered character string table;

if the determined ordered character string table has the possibility of comprising the target key, determining that an ordered character string table with the possibility of comprising the target key exists in the target ordered character string table group, and taking the determined ordered character string table as the target ordered character string table;

and if the determined ordered character string table does not have the possibility of comprising the target key, determining that the target ordered character string table which has the possibility of comprising the target key does not exist in the target ordered character string table group.

18. The data lookup method as claimed in claim 14 wherein said determining whether a target ordered string table exists in said target ordered string table set according to said table filter in said two-stage filter, said target ordered string table set having a likelihood of including said target key comprises:

acquiring ordered character string tables in the target ordered character string table group one by one, and judging whether the ordered character string tables have the possibility of including the target key or not through a table filter of the ordered character string tables;

if any one ordered character string table in the target ordered character string table group has the possibility of comprising the target key, determining that a target ordered character string table with the possibility of comprising the target key exists in the target ordered character string table group;

and if all the ordered character string tables in the target ordered character string table group do not have the possibility of including the target key, determining that the target ordered character string table having the possibility of including the target key does not exist in the target ordered character string table group.

19. A filter updating method based on the log merge tree key value storage system according to any one of claims 1 to 4, the filter updating method comprising:

determining any one of the ordered character string surface layers as a current ordered character string surface layer, wherein the ordered character string surface layers are stored according to a preset sequence;

acquiring the number of ordered character string tables in the current ordered character string surface layer and the upper limit number of a preset table;

if the number of the ordered character string tables in the current ordered character string surface layer is larger than or equal to the upper limit number of the preset tables, selecting an ordered character string table group from the current ordered character string surface layer;

generating an ordered character string table to be merged according to the selected ordered character string table group;

merging the to-be-merged ordered character string table to a next ordered character string surface layer of the current ordered character string surface layer to obtain a merged ordered character string surface layer;

and updating a two-stage filter of the next ordered character string surface layer of the current ordered character string surface layer according to the merged ordered character string surface layer.

20. The filter updating method of claim 19, wherein said selecting an ordered set of strings from said current ordered set of strings comprises:

and selecting an ordered character string table group from the current ordered character string table layer according to the number and the generation time of the ordered character string tables.

21. The filter updating method of claim 19, wherein generating the ordered string table to be merged from the selected ordered string table group comprises:

merging and sorting the ordered character string tables in the selected ordered character string table group to obtain a first ordered character string table;

and dividing the first ordered character string table into at least one second ordered character string table according to the key range of the next ordered character string surface layer of the current ordered character string surface layer to obtain the ordered character string table to be merged, which comprises the at least one second ordered character string table.

22. The filter updating method of claim 21, wherein said dividing the first ordered string table into at least one second ordered string table according to the key range of the next ordered string table of the current ordered string table comprises:

if the table key range of the first ordered character string table is contained in any group key range of the next ordered character string surface layer of the current ordered character string surface layer, the first ordered character string table is not divided;

if the table key range of the first ordered character string table is contained in any plurality of group key ranges of the next ordered character string surface layer of the current ordered character string surface layer, dividing the first ordered character string table according to the any plurality of group key ranges;

and if the table key range of the first ordered character string table is not contained in any group key range of the next ordered character string surface layer of the current ordered character string surface layer, uniformly dividing the first ordered character string table.

23. The filter update method of claim 19, wherein the merging the to-be-merged ordered string table to a next ordered string table of the current ordered string table comprises:

and in the next ordered character string surface layer of the current ordered character string surface layer, merging each second ordered character string table in the ordered character string tables to be merged into an ordered character string table group with a group key range containing the table key range of the second ordered character string table through multithreading.

24. The filter updating method of claim 19, wherein said updating the two-stage filter for the next ordered string surface layer of the current ordered string surface layer based on the merged ordered string surface layer comprises:

generating corresponding group element data according to the ordered character string table group generated in the combined ordered character string surface layer, and establishing a corresponding group filter in the nonvolatile storage medium;

and updating the corresponding group filter in the nonvolatile storage medium according to the modified ordered character string table group in the combined ordered character string table layer.

25. A non-volatile storage medium management method based on the log merge tree key-value storage system according to any one of claims 1 to 4, the non-volatile storage medium management method comprising:

receiving a create request for the table filter or the group filter;

converting the creation request into a space allocation request for the non-volatile storage medium;

acquiring a lock of an idle linked list of the nonvolatile storage medium, wherein the idle linked list comprises page numbers of idle pages of the nonvolatile storage medium;

extracting a plurality of page numbers from the idle linked list;

creating a storage management object in the nonvolatile storage medium according to the plurality of page numbers;

returning the storage management object;

releasing the lock of the idle linked list.

26. A non-volatile storage medium management method based on the log merge tree key-value storage system according to any one of claims 1 to 4, the non-volatile storage medium management method comprising:

triggering a reclamation request for space stored by the non-volatile storage media in response to a data failure of the table filter or the set of filters;

adding the recovery request into a recovery queue, and informing a scheduling thread of the recovery request;

creating a plurality of recovery threads according to the recovery requests in the recovery queue through the scheduling thread so as to recover the failure pages through the plurality of recovery threads;

updating an idle linked list according to the failure page, wherein the idle linked list comprises the page number of the idle page of the nonvolatile storage medium;

and waiting for the recovery thread to end.

27. The non-volatile storage media management method of claim 26, wherein the updating the free-link list based on the stale pages comprises:

acquiring a header page number of the idle linked list;

assigning the header page number to a next page number of the failure page;

and re-determining the page number of the failure page as a header page number.

28. A computer device comprising a processor and a memory; the memory to store instructions; the processor is configured to call an instruction in the memory, so that the computer device executes the log merge tree key-value storage system based on the hybrid storage architecture according to any one of claims 1 to 4, or so that the computer device executes the log merge tree key-value storage system optimization method based on the hybrid storage architecture according to any one of claims 5 to 27.

29. A computer-readable storage medium storing at least one instruction which, when executed by a processor, implements the hybrid storage architecture based log merge tree key-value storage system of any one of claims 1 to 4, or which, when executed by a processor, implements the hybrid storage architecture based log merge tree key-value storage system optimization method of any one of claims 5 to 27.