KR20210058118A - Casedb: low-cost put-intensive key-value store for edge computing - Google Patents

Abstract

Disclosed is a low-cost put-intensive key-value storage device for edge computing to increase write performance. According to one embodiment of the present invention, a method executed in the key-value storage device comprises the following steps: determining a memory component related to data information according to a size of data as a memory component and a disk component are configured, wherein the memory component includes a Memtable and a C buffer (CBuffer); separating data and metadata by flushing metadata including a key and a bloom filter to the disk component as the determined memory component processes the data information; and delaying compaction of a string sort table by merging the separated metadata to perform data management tasks.

Description

CaseDB: Low-cost put-intensive key-value storage device for edge computing {CASEDB: LOW-COST PUT-INTENSIVE KEY-VALUE STORE FOR EDGE COMPUTING}

The following description relates to a key-value storage device.

Until now, cloud storage has been provided to users to manage and share data through a central data center. Since these servers cannot meet the needs of current end users in terms of transmission speed and data consistency, the number of data that can be stored in this kind of centralized server has increased exponentially. In the new era of big data, a new distributed computing paradigm called edge computing is emerging as a solution to data management and communication problems. Edge computing provides a decentralized solution that brings data computation and storage closer to where it is needed to improve response time and save bandwidth. Not only does it shorten data transfer times, it also provides inexpensive hardware options for data storage and processing. Local edge servers don't require as much CPU power and hardware resources as big data servers, while the amount of data they send and receive is considerably less than in data centers. The main purpose of edge servers is to enable high communication speeds for locally connected IoT devices, and to exchange data with cloud servers that are broadcast to all other edge servers.

In addition to the advantages of edge computing over centralized networks, there are certain vulnerabilities that must be reduced while designing edge servers. One of the most important factors to consider for an edge server is internal data management. Until now, NoSQL databases are the most promising in big data systems because they do not rely on heavy SQL queries and have a simple interface. The KV (key-value) store is the fastest and simplest representation of a NoSQL database. KV Store considers only keys and values as data representations and provides public methods such as Put, Get, and Delete for data management. So far, several efforts have been made to improve the read or write throughput of the KV store. The B+ tree indexes each item into a multi-item bucket to provide very good read performance, but suffers from writing and updating difficulties. In contrast, log-structured file systems have the great advantage of writing data, but poor performance when accessing that data. The log structured merge tree (LSM-tree) has become one of the most widely used data structures for implementing KV stores. The LSM-tree uses a multi-level structure, and data is written in sorted order at each level. This significantly improves the read performance.

For example, LevelDB, used in Google's data center, is open source with major KV stores and clear documentation, and is easy to customize. LevelDB sorts data at each level and improves read throughput by using an LSM tree. However, it performs a significant amount of read and write I/O operations, resulting in a significant disadvantage of amplifying writes. Write amplification simply means inconsistency while looking for KV items on the hard disk drive (HDD). As data size increases, data items must be rewritten repeatedly because they do not stay at a stable physical address. Moreover, edge servers use only limited hardware sources. Because edge computing servers are small hardware devices placed in multiple locations including a small area, the size of the computing server is expected to be as small as an embedded board. However, even the most well-designed embedded boards have difficulty connecting to external hardware. Specifically, unlike an embedded board, the server includes a PCI expansion slot for directly attaching an SSD or HDD to the motherboard. This greatly increases the data read and write latency of the external HDD.

Key-value storage that expands LevelDB by introducing new functions by separating keys and filters from values (data values) of different files to reduce compaction I/O to solve common problems in LevelDB Apparatus and method can be provided.

The method performed in the key-value storage device includes determining a memory component related to data information according to the size of data as a memory component and a disk component are configured-Memtable and C buffer (CBuffer) as the memory components. Including -; Separating data and metadata by flushing metadata including a key and a bloom filter to a disk component as the determined memory component processes the data information; And merging the separated metadata to delay compaction of the string sorting table to perform data management.

The determining of the memory component may include checking the data size to classify the following values according to a preset criterion, inserting a value less than the sorted preset criterion into the C buffer, and a value less than the preset criterion. Including the step of inserting the remaining values that are not classified as in the Memtable, and the separating step includes maintaining a small value less than a preset reference inserted in the C buffer until a threshold value is reached, and the merged values are critical. When the value exceeds the value, the value combined with the group key is inserted into the Memtable, and when the Memtable becomes full as the remaining values are inserted into the Memtable, it is converted into a read-only table (immutable Memtable) and the pool state after a certain period of time. Flushing the data of the disk to a disk component, creating a new Memtable, and inserting incoming data thereafter, and the performing of the data management operation includes: separating metadata and data by the flushing, and The metadata including the and bloom filter is stored as a level 0 metadata layer, and the value data is stored separately in a level 0 string sorted table. When the metadata layer becomes full, the first compact Data items that are updated and deleted while the metadata files of two adjacent levels are selected from the key range of the preset criteria and read from the memory component, and the metadata existing in the memory component are merged and sorted. It is deleted from the metadata as it is recorded in the delete-batch log, and after merging the metadata, a new metadata file is created at a lower level, and the first compaction is performed to delete the metadata. have.

The performing of the data management operation may include performing a second compaction in which data information existing at level 0 of the string sorting table is moved to a destination of the string sorting table as the first compaction is completed, and In the second compaction, metadata files of all levels in a preset range and a string sorting table for referencing metadata are read as a memory component, and a string is merged into the metadata file and sorted. After reading the sort table, data is moved to a new string sort table according to the metadata, and a reference from the metadata is updated to a new string sort table.

The determining of the memory component includes, upon receiving a read request for a specific key from an application, determining whether the specific key is included in the merged item by the C buffer or inserted into the Memtable, and the separation The step of: extracting a key-value item by searching for the merged item from a metadata file, searching for the merged key-value item in the metadata file, and searching for the merged key-value item in the metadata file- If the value does not exist, a scan is started in the database, and when a search is executed in the Memtable, the data information contained in the read-only Memtable is moved to the disk component, and the data information for each level is checked with the Bloom filter. , The performing of the data management operation includes scanning the metadata file when the bloom filter returns positive feedback and obtaining a reference to the string sorting table, and when the bloom filter returns negative feedback, the scan item It may include the step of skipping the corresponding level and continuing the search from a level following the corresponding level.

The performing of the data management operation may include inserting the same key with the updated value to update the key-value item.

The performing of the data management operation may include removing the deleted key and the updated key from the metadata file when the compaction starts.

The key-value storage device uses a metadata file having a fixed size, and the metadata file has a constant size and is composed of at least one offset block and a filter block in a key-value format, and the metadata file Allocates a block to the bloom filter of, uses the bloom filter as metadata, stores data in a key-value format in the offset block, the key is the key of the data, and the value is in the string sorting table holding the key. May be a reference to.

The key-value storage device includes at least one processor embodied to execute computer-readable instructions contained in a memory, and the size of data according to the configuration of the memory component and the disk component by the at least one processor A memory component related to data information is determined according to the memory component-including a Memtable and a C buffer (CBuffer) as the memory component -, and a key and a bloom filter are included as the data information is processed by the determined memory component. Data management can be performed by flushing metadata to a disk component to separate data and metadata, and by merging the separated metadata to delay compaction of a string sorting table.

The key-value storage device according to the embodiment avoids small writes by buffering/merging key-value items in memory, and separating the value and the key for metadata compaction, thereby reducing I/O cost and writing performance. It is possible to provide a method and system to improve the.

The key-value storage device according to an embodiment may achieve write throughput and reduce write amplification without degrading read performance.

1 is a diagram illustrating a key-value storage device according to an exemplary embodiment.
FIG. 2 is a diagram illustrating a layout of a string arrangement table file in a key-value storage device according to an exemplary embodiment.
3 is a diagram illustrating a metadata file layout in a key-value storage device according to an exemplary embodiment.
4 and 5 are diagrams for describing compaction in a key-value storage device according to an exemplary embodiment.
6 is a flowchart illustrating a method of performing a data management operation in a key-value storage device according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

A key-value storage device and method for delaying compaction of a string sort table by separating a metadata file and using a bloom filter as metadata and merging the metadata can be provided.

1 is a diagram illustrating a key-value storage device according to an exemplary embodiment.

In the embodiment, with the aim of reducing compaction I/O, CaseDB (hereinafter referred to as'key-value (hereinafter referred to as ``key-value'') expands LevelDB by introducing a new function by separating keys and filters from values of different files. Key-Value) storage device') can be provided. Referring to FIG. 1, a string sorted table (SSTable) in Level DB includes both metadata and data, and the entire file is read into memory for compaction. The string sorting table may mean a data storage unit that stores sequentially created data in a sorted order. However, most logical and arithmetic operations are performed on keys and values and can be read to simply update the position. Accordingly, the compilation process can be redesigned to separate the string sort table into metadata using the key and delay merging of the string sort table. The key-value storage device can drastically reduce the number of reads and writes of an SSD.

The SSD's interface is not directly connected to the edge server's built-in board, and the bloom filter of the metadata file can be moved to maintain read throughput. In addition, the metadata constitutes a small ratio of the total data less than or equal to a preset standard, and the compaction according to the embodiment may allow relatively small I/O. For example, a key-value storage device can allow relatively less I/O than that of LevelDB. Below. It is possible to provide a key-value storage device that reduces compaction I/O and maintains the multilevel ordered structure of LevelDB.

The key-value storage device can constitute a memory component and a disk component. The key-value storage device may include a new memory component, a C buffer, along with a memory component, Memtable. In addition, the key-value storage device may adopt an architecture and data unit to reduce read and write I/O and increase throughput.

The C buffer can be used to merge small values from the front. In other words, the C buffer can be used to merge small key-value items. The key-value storage device can create a new section on the disk that stores the metadata file. The key-value storage device can configure a plurality of levels in the metadata file, similar to the string sorting table of the level DB. Each metadata file can correspond to multiple string sort tables at the same level. 2 and 3, a key and a reference for a string sort table and a bloom filter may be included in the metadata file. In addition, the key-value storage device may create an additional log file called delete_batch, which is a temporary file for performing a compaction process that tracks deprecated (deprecated) data. The key-value storage device prevents versions from being duplicated at different levels and can efficiently remove deprecated data from the database. Key-Value Storage Separates key and value, but maintains an LSM-tree structure for both.

Referring to FIGS. 2 and 3, the key-value storage device may use different layers for different metadata files including a key for a value, a reference, and a bloom filter. Although a key and a reference are required as meta information for compaction, it is possible to consider performing scanning and reading while executing the writing process. The key-value device may include a bloom filter in the metadata file to improve search performance.

As shown in FIG. 3, in the key-value device, the metadata file has a constant size and may be composed of an offset block and a filter block in a key-value (KV) format. The offset block stores data in KV format, of which the key is the data key, and the value is a reference to the string sorting table holding the key. In addition, blocks may be allocated to the bloom filter in order to improve read performance. The metadata size can be calculated as in Equation 1.

Equation 1:

Here, M is the size of metadata, B is the size of the bloom filter, K is the size of the key, O is the size of the offset, and n is the number of items in the metadata file.

Equation 1 will be described as an example. (K + O) is always set once and remains constant for the entire database, and B is calculated according to n. Accordingly, the variable that greatly affects the size of the metadata file is the number of items in the file. For example, a key-value storage device may use 16 bytes for each key and 20 bytes for an offset. The key-value storage device may calculate the metadata size by assigning a value to n and changing the size of the value to check the ratio of the string sorting table. For example, when the value is 50 bytes, the share of metadata can be derived as 73%. This percentage can decrease as the size of the value increases. This is because the number of inputs decreases as the size of a single item increases in a fixed workload. You can see that metadata sharing affects the size of the bloom filter. Moreover, even if the size of the value increases, the size of the key remains constant, and thus the share of metadata sharing may decrease. Accordingly, it can be assumed that in order to efficiently use the key-value storage device, it is necessary to avoid writing a small size to the database. The key-value storage device may merge a small value into one large item by using a C buffer and insert a large item into the Memtable. In addition, in consideration of the scale of metadata sharing, the key-value storage device can easily manage the size of data at each level and find data by using a fixed-size metadata file. The key-value storage device makes it easy to manage a multi-layered structure like a string sorting table.

The sharing of metadata may increase as the size of the inserted data decreases. The key-value storage device may arrange the C buffer as a memory component. The C buffer can load KV items into an arbitrary memory buffer and merge them into a specific size. At this time, all the merged data may be recorded in the metadata file to allow reading from the merged data. In other words, the C buffer ensures that all data is at least larger than a certain size. We ensure that the metadata separation method works perfectly in all cases.

In the embodiment, it is assumed that C buffer allocation is possible because checkpoints can be used if the Put method is redefined before inserting data. The key-value storage device may use the threshold value to check whether to directly merge or insert data into the database. In the key-value storage device, if the size of the data is larger than the threshold value, the data is directly inserted into the Memtable, one of the memory components, and no space is allocated to the C buffer. At this time, if the size of the data is smaller than the threshold, the key-value storage device inserts the data into the C buffer and delays the value of the value stored in the C buffer until the size of the value reaches the smallest possible size. You can create an item. After merging the KV items, the key-value storage device can generate a unique key for the group, store the log in the metadata file, and insert it into the Memtable. At this time, before inserting into the Memtable, a small KV item may be merged into a larger item.

The number of items merged by the C buffer may vary depending on the merge size and data size. For example, if 4KB is used as the threshold value and the merge size, all items smaller than 4KB are merged using the C buffer, and can be recorded in the metadata file. In this case, this may include the original key of the data and the data group key. Moreover, since the size and index of the group's data are also recorded, using this information, the data can be easily extracted from the merged group.

6 is a flowchart illustrating a method of performing a data management operation in a key-value storage device according to an exemplary embodiment.

In step 610, the key-value storage device may determine a memory component related to data information according to the size of the data as the memory component and the disk component are configured. The key-value storage device may check the data size of the data information and determine a Memtable or C buffer, which is a memory component to which data information is to be inserted, according to the data size.

In step 620, as the key-value storage device processes data information in the determined memory component, the disk component may flush the metadata including the key and the bloom filter to separate the data and the metadata. In this case, after the metadata including the key and the bloom filter is flushed to the metadata layer, compaction may be performed. When data is flushed to a disk component, the key and value can be separated, the key can be offset to the corresponding value, and the bloom filter can be implemented in the metadata file where the key exists.

In step 630, the key-value storage device may perform data management by merging the separated metadata to delay compaction of the string sorting table. Compaction can be run on both keys and values. Compressing the metadata file first may delay value compaction. The key-value storage device may perform a first compaction and a second compaction. In the first compaction, only metadata is compressed and the position of the key can be updated while maintaining the same reference to the value. In the second compaction, you can rewrite the value to the final level along the key. The key-value storage device can maintain the key and value in the LSM-tree even if the key and value are separated into different layers. Data management operations may include writing, reading, updating, and deleting. Referring to FIGS. 4 and 5, data management operations including writing, reading, updating, and deleting in a key-value storage device will be described.

First of all, a data management task for writing will be described. The data insertion process of the key-value storage device can check the data size and classify a small value according to the data size. The key-value storage device may insert a small value into the C buffer according to classifying a small value, and insert the remaining values (eg, normal values) that are not classified as small values into the Memtable. At this time, the small value inserted into the C buffer may be maintained until the size of the C buffer reaches a threshold value. In the key-value storage device, as KV items of the remaining values are directly inserted into the memtable, when the memtable becomes full, it is converted to read-only and a new memtable can be created. In this case, the new Memtable may mean becoming an immutable Memtable. The key-value storage device can flush the data of the immutable Memtable to the disk component. Immutable Memtables are moving queues that flush data to disk components. As shown in FIG. 1, when the metadata is flushed, it can be separated into original data and metadata. In this case, the key-value device may flush the metadata including the key and the bloom filter to the metadata layer and then perform compaction. Metadata including keys and bloom filters is stored as a level 0 metadata layer, and value data is separately stored in a level 0 string sort table, and compaction can be performed when the metadata layer becomes full. In contrast, the string sorting table only sits at level 0 and can remain at level 0 until metadata compaction is complete. The most important factor to consider in the architecture of the key-value device is that the compaction process employs a plurality (eg, two) of the main parts, as shown in FIG. 4. Metadata files of a plurality of adjacent levels may be selected within a preset key range and read into a memory. During merging and sorting of metadata in memory, updated and deleted items are written to the delete_batch log and can be deleted (logically deleted) from the metadata. Since this data is not accessible via metadata, it will never be read even if the original data exists in the KV repository. After merging the metadata, the key-value storage device may create a new file at a lower level and delete the metadata file before merging. In this case, the lower level may mean a lower level based on the current level. In an embodiment, a compaction process may be performed for all metadata levels.

The first compaction of the key-value storage device contains the metadata file, implying that the metadata does not contain the value, whereas the rewrite of the sorted item is faster in the key-value storage device than in the string sort table. do. Also, since metadata is only a small percentage of the total database, reading one metadata file from a key-value storage device and rewriting it after compaction is the same as performing the same process on hundreds of string sorting tables.

After creating the metadata file, the key-value storage device may perform a second compaction in which all data in level 0 of the string sort table is moved to the final destination of the string sort table. Referring to FIG. 5, it is for explaining the second compaction. Until now, metadata files have been compacted and placed at a specific level. In the second compaction, all metadata files of all levels of a specific range and a string sorting table for referencing the metadata may be read into the memory. Since the keys are already merged and sorted into the metadata file, there is no need for a heavy CPU load in the second compaction process. After reading the string sort table, you can move the data to a new string sort table according to the metadata. Here, check the delete_batch file and skip data that is no longer used. Finally, references from metadata can be updated with a new string sorting table. Then, all metadata files are rewritten to disk and the old values are deleted. When a new string sort table is rewritten, it is recorded at a specific level as indicated in the sorted metadata.

Equation 2:

As the first compaction and the second compaction are executed, the key-value storage device may have a structure similar to that of the level DB. The key-value storage device provides a multi-level structure of a string sorting table, and data can be sorted on a level basis. In addition, key-value storage devices can use the delete_batch log file to track data that is no longer in use and delete it before the write process is complete. The key-value storage device has a great advantage over LevelDB when it delays the compaction of the original data and places the data into the target string sorting table immediately. This advantage can be proved numerically by calculating a WAR (write amplification ratio) using Equation 2. In Equation 2, a new variable P representing a metadata sharing factor may be used. The value of P is expected to be less than 1 and is usually a very small number. For example, if the value size is 4 KB and the metadata sharing is 1%, then P = 0.01, so the right side of the equation is removed. In other words, WAR depends not only on the number of levels and the adjacent level splitting factor, but also on the metadata sharing factor. Even if the number of levels increases according to the database size according to the basic configuration (size(Li+1)/size(Li) = 10), a very small P decreases the weight of the second term in Equation 2 and writes amplification is expected to be between 2 and 3. do. In Equation 2, 2 represents a specific number of writes of the string sorting table in the database. The string sort table can be created once when flushed and then rewritten at the end of the write process. This means that each datum is handled at least twice. Also, if P increases, the case should be considered. This situation occurs when the KV item size is really small and the metadata acquires a larger share.

In addition, data management tasks for reading will be described. Read throughput is a very important factor because in real life the read rate is never less than the write rate. Level DB achieves high read throughput by ensuring level-based sorting data. Since the key-value storage device inherits the general data structure from the level DB, it can also provide compatible read performance. A multi-layer read layer can be used to improve the performance of the key-value storage device. As the key-value storage device receives a read request for a specific key from an application, it may first determine whether the key belongs to the item merged by the C buffer or directly inserted into the Memtable. Considering that the KV store proposed in the embodiment aims to be used by an edge server for a connected car, it can be determined through a data source. Typically, the key contains information about which sensor is transmitting data. The key-value storage device can extract this information and infer the size of the value. The merged item can be searched for the merged KV item by searching in the metadata file, and the necessary value can be extracted. After retrieving the merged KV item in the metadata file, or if the key is not in the metadata file, you can start a scan in the database. The search is run on the memtable and the read-only memtable is first moved to the disk component. You can start at the highest level and check the data into the bloom filter. If the bloom filter returns positive feedback, the metadata file can be scanned and a reference to a specific string sorting table can be obtained. If the bloom filter returns negative feedback, the scan item is not at that level (the level corresponding to the negative feedback returned), so the level can be skipped and the search can continue from the next level. In the case of updated data, the previous value will remain at a certain level until the compaction reaches the corresponding one. If data is found at a higher level, it means that it will never be retrieved until the previous value is removed. The key-value storage device satisfies the version constraint rule.

In addition, data management tasks for update and deletion will be described. Another important data operation using the KV store is performing updates and deletions. The key-value storage device provides an erasing method, but no update method is required. Since the key-value storage device is a log structured file system, updating data means deleting old data and inserting new data. Accordingly, to update an existing KV entry, you must insert the same key with the updated value. The KV store itself recognizes the same two keys and holds the last inserted key.

In addition to write operations, erase operations are placed in memory to be performed immediately. When compaction starts, the deleted and updated keys are removed from the metadata file, and this file is logically deleted. However, the original data is stored in the string sorting table as deprecated data that must be permanently removed. To track such data, the key-value storage device uses a delete_batch file that stores metadata of a corresponding item, and may perform initialization and execution in the first compaction and the second compaction, respectively. When reading the string sort table into memory in the second compaction, it is possible to check whether data is deprecated using the delete_batch log.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (8)

In the method performed in the key-value storage device,
Determining a memory component related to data information according to the size of data as the memory component and the disk component are configured-including a Memtable and a C buffer (CBuffer) as the memory component;
Separating data and metadata by flushing metadata including a key and a bloom filter to a disk component as the determined memory component processes the data information; And
Merging the separated metadata to delay compaction of the string sorting table to perform data management
A method that is stored in a key-value device comprising a. The method of claim 1,
Determining the memory component comprises:
Check the data size and classify the following values according to the preset criteria, insert the values below the classified criteria into the C buffer, and insert the remaining values not classified as values below the preset criteria into the Memtable. Steps to do
Including,
The separating step,
The values below the preset standard inserted in the C buffer are maintained until the threshold value is reached, and when the merged values exceed the threshold value, the value combined with the group key is inserted into the memtable, and the remaining values are inserted into the memtable. Accordingly, when the memtable becomes full, it is converted into an immutable memtable, and after a certain period of time, the full data is flushed to the disk component, and a new memtable is used to store key value data. To insert the incoming data after creating
Including,
The step of performing the data management operation,
Metadata and data are separated by the flushing, and metadata including keys and bloom filters are flushed to the metadata layer to store metadata separately from value, and when metadata becomes full, primary compaction The two adjacent level metadata files are selected from the key range of the preset criteria and read from the memory component, and the data items that are updated and deleted while merging and sorting the metadata existing in the memory component are delete-batch is deleted from the metadata as it is recorded in the log, and after merging the metadata, creating a new metadata file at a lower level and deleting the metadata
A method performed in a key-value storage device comprising a. The method of claim 1,
The step of performing the data management operation,
As the execution of the first compaction is completed, a second compaction is performed in which data information present in level 0 of the string sorting table is moved to the destination of the string sorting table, and in the second compaction, a preset range Read the string sorting table for referencing metadata files and metadata of all levels of the memory component as a memory component, read the string sorting table as the key is merged and sorted into the metadata file, and then read the string sorting table into the metadata. Data is moved to the new string sort table accordingly, and the reference from the metadata is updated to the new string sort table.
A method performed in a key-value storage device comprising a. The method of claim 1,
Determining the memory component comprises:
Upon receiving a read request for a specific key from an application, determining whether the specific key is included in the merged item by the C buffer or inserted into the Memtable
Including,
The separating step,
A key-value item is extracted by searching for the merged item from a metadata file, and the merged key-value item is searched for in the metadata file, and the merged key-value does not exist in the metadata file. If not, start a scan in the database, and when a search is executed in the Memtable, the data information contained in the read-only Memtable is moved to the disk component, and the data information for each level is checked with the Bloom filter.
Including
The step of performing the data management operation,
If the bloom filter returns positive feedback, the metadata file is scanned and a reference to the string sorting table is obtained. If the bloom filter returns negative feedback, the scan item skips the corresponding level and follows the corresponding level. Steps to continue the search from the level
A method performed in a key-value storage device comprising a. The method of claim 1,
The step of performing the data management operation,
Inserting the same key with the updated value to update the key-value item
A method performed in a key-value storage device comprising a. The method of claim 1,
The step of performing the data management operation,
When the compaction starts, removing the deleted key and the updated key from the metadata file
A method performed in a key-value storage device comprising a. The method of claim 1,
The key-value storage device uses a metadata file of a fixed size,
The metadata file has a constant size and is composed of at least one offset block and a filter block in a key-value format, allocates a block to a bloom filter of the metadata file, and uses the bloom filter as metadata,
The offset block stores data in a key-value format, where the key is the key of the data, and the value is a reference to the string sorting table holding the key.
Method performed in a key-value storage device, characterized in that. In the key-value storage device,
At least one processor implemented to execute computer readable instructions contained in memory
Including,
By the at least one processor,
As a memory component and a disk component are configured, a memory component related to data information is determined according to the size of the data-and the memory components include Memtable and C buffer (CBuffer),
As the determined memory component processes the data information, the metadata including the key and the bloom filter are flushed to the disk component to separate data and metadata,
Merging the separated metadata to delay the compaction of the string sorting table to perform data management
Key-value storage device.

Images