WO2018133762A1

WO2018133762A1 - File merging method and apparatus

Info

Publication number: WO2018133762A1
Application number: PCT/CN2018/072641
Authority: WO
Inventors: 郑主能
Original assignee: 广州市动景计算机科技有限公司
Priority date: 2017-01-17
Filing date: 2018-01-15
Publication date: 2018-07-26

Abstract

Disclosed are a file merging method and apparatus. The method comprises: appending to write an appended data block after a first file, a value in a data block of a second file being written therein; appending to write a new index block after the appended data block, the new index block being generated based on an index block of the first file and an index block of the second file, logical addresses of all keys in the index block of the first file and the index block of the second file and corresponding values thereof in the data block of the first file and the appended data block being respectively recorded in leaf nodes of a new B + tree; and appending to write a new file header after the new index block so as to record metadata information of a merged new file. Accordingly, when two files are merged, a value of one file just needs to be directly appended to write into the other file, thereby improving write performance; and merged index blocks are a new B + tree, which makes it convenient to read values in a merged large file by means of a search, thereby improving read performance.

Description

File merging method and device

Technical field

The present invention relates to the field of data storage technologies, and in particular, to a method and apparatus for merging files stored in an external memory.

Background technique

Looking at the storage engine of today's databases, the underlying data structure is either a B-tree or its variant B+ tree, or an LSM tree. The former has better read-friendliness, while the latter has better write-friendliness. Although these two things seem to have both fish and bear's paws, in the greedy Internet world, they are eager to have a data storage solution that is compatible with both reading and writing. Although the data structure used in LevelDB seems to combine LSM and B-tree, it is not thorough enough. First, in the strict sense, it is not a B-tree, but a simple multi-fork tree; second, its Key (key) and The value (value) is stored together, which is not conducive to the optimization of the index. This optimization is especially important when doing data merging.

Specifically, the files stored in the disk in LevelDB are divided into multiple levels, and there are many files (SSTable files) in different levels. In order to reduce redundancy and improve readability, the SSTable files need to be merged, because in the SSTable file. The keys and corresponding values are stored together, so when merging LevelDB files, all key-value pairs need to be fetched one by one to build a new file. The merge process is more complicated and reduces readability. Write performance.

In recent years, with the rise of NoSql, various KV-type storage engines have emerged. There are caches, but also for persistence, which is representative of LevelDB in the field of persistence. LevelDB is a high-performance KV storage engine developed by Google, inspired by Google's BigTable.

Although LevelDB can play very good performance in the small data volume scenario, in the case of large data volume (hundreds of G) and high frequency write, LevelDB is reading, writing, merging, data cleaning, restart recovery, etc. Many aspects have exposed its shortcomings.

Summary of the invention

An object of the embodiments of the present invention is to provide a file merging method and apparatus for data merging.

According to an aspect of an embodiment of the present invention, a file merging method is provided. The file is stored in an external memory, including a file header, a data block, and an index block. The file header is used to record metadata information of the file, and the data block is used for storing. The index block is used to store the key corresponding to the value in the form of a B+ tree, wherein the logical addresses of all the keys and their corresponding values in the data block are respectively recorded in the leaf nodes in the B+ tree, and the method includes: The first file is additionally written with an additional data block in which the value in the data block of the second file is written; after the additional data block is additionally written, the new index block is additionally written, and the new index block is based on the index block of the first file and the second The index block generated by the file, the index block of the first file and all the valid keys in the index block of the second file and their corresponding values are recorded in the data block of the first file and the logical address in the additional data block respectively. In the leaf node in the B+ tree; a new file header is additionally written after the new index block to record the metadata information of the merged new file.

The keys and values of the file described in the embodiment of the present invention are stored separately, and the keys are stored in the form of a B+ tree. Therefore, when the two files are merged, one file can be kept unchanged, and the value of the other file can be directly added and written, thereby improving the writing performance. And the merged index block is a new B+ tree, and the value in the merged file can be conveniently read according to the new index block, and the read performance of the merged file is not affected.

Optionally, the metadata information may include one or more of the following:

The number of keys in the index block;

The range of keys in the index block;

The height of the B+ tree;

The logical address of the first leaf node in the B+ tree;

The number of internal nodes in the B+ tree.

Therefore, when the corresponding target value is read according to the request key, whether the request key is within the range of the key of the file can be determined according to the metadata information in the file header of the file, and if the determination is yes, the file is further Finding in the index block can reduce unnecessary lookups.

Optionally, all nodes constituting the B+ tree are physically stored contiguously.

Thus, the B+ tree can be physically stored continuously by utilizing the local preloading feature of the disk, so that the index block of the file to be merged can be obtained by simply traversing successive disk blocks in the process of reconstructing the index block.

Optionally, the file merging method may further include: updating a file header of the first file according to the new file header to replace the metadata information in the file header of the first file with the metadata information in the new file header.

Since the append write is a destructive write, the present invention can avoid the damage caused to the file by the abnormal situation during the merge process by setting the double file header.

Optionally, the file includes a front file header at the head of the file and a back file header at the end of the file. The contents of the front file header and the last file header are the same, and the previous file header of the first file is updated according to the new file header as a new file. The header of the previous file, with the new header as the header of the new file.

Thus, when the merge is completed normally, the front and back headers of the new file can be updated normally, and can be used to view the metadata information in the new file.

Optionally, the file merging method may further include: when the step of writing the metadata information of the new file in the new file header is wrong, the new file is restored to the first file before the merge according to the file header of the first file. And/or in the case of an error in the step of updating the header of the first file, the header of the first file is re-updated according to the new header.

Therefore, when an error occurs in the process of writing a new file header, since the file header of the first file has not been updated, the file in the merge process can be restored to the first file before the merge according to the file header of the first file. When an error occurs during the process of updating the header of the first file, it can be re-updated according to the new header.

Optionally, the file merging method may further include: performing the following steps to read the target value corresponding to the request key from the target file: obtaining a file header and an index block of the target file; determining, according to the file header, whether the request key is at the file header Within the range of the indicated key; in the case where it is determined that the request key is in the range, based on the B+ tree structure of the index block, the leaf node corresponding to the request key is searched in the index block; stored according to the found leaf node The value corresponding to the key reads the target value at the logical address in the data block in the target file.

According to another aspect of the embodiments of the present invention, a file merging apparatus is further provided, where the file is stored in an external memory, including a file header, a data block, and an index block, where the file header is used to record metadata information of the file, and the data block is used. For storing values, the index block is used to store the key corresponding to the value in the form of a B+ tree, wherein the logical addresses of all the keys and their corresponding values in the data block are respectively recorded in the leaf nodes in the B+ tree, and the device includes a first writing unit, configured to write an additional data block after the first file, wherein a value in the data block of the second file is written; a B-tree generating unit, configured to be based on the index block of the first file and the second The index block of the file generates a new B+ tree, and all the valid keys in the index block of the first file and the index block of the second file and their corresponding values are respectively recorded in the data blocks of the first file and the logical addresses in the additional data block. In the leaf node in the new B+ tree; the second write unit is configured to additionally write a new index block after appending the data block, wherein the new B+ tree is written; and the third write unit is used after the new index block Add a new header is written to the new file meta merged data record.

Optionally, the metadata information may include one or more of the following:

The number of keys in the index block;

The range of keys in the index block;

The height of the B+ tree;

The logical address of the first leaf node in the B+ tree;

The number of internal nodes in the B+ tree.

Optionally, the file merging device may further include: an updating unit, configured to update the file header of the first file according to the new file header, to replace the metadata in the file header of the first file with the metadata information in the new file header. information.

Optionally, the file includes a front file header at the head of the file and a rear file header at the end of the file. The contents of the front file header and the back file header are the same, and the update unit updates the previous file header of the first file according to the new file header. The front file header of the new file, with the new file header as the post file header of the new file.

Optionally, the file merging device may further include: a first restoring unit, where the step of writing the metadata information of the new file in the new file header fails, the new file is restored according to the file header of the first file The first file before the merge; and/or the second restore unit is configured to re-update the file header of the first file according to the new file header if the step of updating the file header of the first file is erroneous.

Optionally, the file merging device may further include a reading unit, configured to read the target value corresponding to the request key from the target file, where the reading unit may include: an acquiring module, acquiring a file header and an index of the target file a determining module, according to the file header, determining whether the request key is within the range of the key indicated by the file header; and the finding module, in the case of determining that the request key is in the range, searching in the index block based on the B+ tree structure of the index block Corresponding to the leaf node of the request key; the reading module reads the target value in the logical address in the data block in the target file according to the value corresponding to the key stored by the found leaf node.

The key and value of the file described in the file merging method and apparatus of the embodiment of the present invention are separately stored, wherein the key is stored in the form of a B+ tree, thereby maintaining one file when merging the two files Moves the value of another file directly to the previous file, improves the write performance, and reconstructs the index block that stores the key in the form of a B+ tree. The new index block can be easily read from the merged file. The value of the merged file will not be affected.

Another object of embodiments of the present invention is to provide a new database management method and database system.

According to an aspect of an embodiment of the present invention, a database management method is provided for storing a plurality of pieces of data, wherein each piece of data includes a corresponding key and value, the method comprising: writing a plurality of pieces of data into an external memory The log file; the data in the log file is written into the memory table in the internal memory, wherein the data written in the memory table is stored in order according to the size of the key; when the size of the memory table exceeds a predetermined threshold, the memory table is converted Is a read-only memory table, and writes subsequent data in the log file to the new memory table; writes the data in the read-only memory table to the external accessor to obtain the first-level storage file; and merges two or More first-level storage files to get second-level storage files.

Thus, the file finally stored in the external memory has only two layers, and the redundancy is low, which is convenient to find.

Optionally, the data block management method may further include: specifying a primary file name of the first-level storage file by using a first naming rule; and specifying a primary file name of the second-level storage file by using a second naming rule, the first naming rule It is different from the second naming rule in order to distinguish whether the storage file is a first-level storage file or a second-level storage file based on the main file name.

Therefore, it can be confirmed whether it belongs to the first-level storage file or the second-level storage file according to the main file name of the storage file.

Optionally, the memory table may be composed of a hash table, where the hash table includes one or more hash buckets, each hash bucket corresponds to one jump table, and each piece of data in the memory table constitutes an element of the jump table. Among them, the order of the elements in the jump table is arranged in order according to the size of the keys.

Inserting the memory table before jumping the table can reduce the lock granularity. For concurrent read and write operations, if the keys are different, the fast lookup insertion can be performed in the jump table corresponding to the respective hash bucket. On the other hand, the expansion is expanded. While the size of the memory table is not increased, the size of the jump table is not increased, which can reduce the probability that the jump table becomes a linear search as the amount of data becomes larger, thereby improving the overall search efficiency.

Optionally, the data block management method may further include: maintaining a read-only memory table queue in the internal memory, where data in the read-only memory table is not all written to the external memory, and when the size of the new memory table exceeds a predetermined threshold , converts the new memory table into another read-only memory table and puts it into the read-only memory table queue.

Therefore, by maintaining the memory table queue, it is possible to cope with the problem of blocking when the high frequency is written, because the data is too late to be merged, and the memory table is full.

Optionally, the data structure of the storage file may include: a file header for recording metadata information of the storage file, a data block for storing the value, and an index block for storing the key corresponding to the value in the form of a B+ tree. The logical addresses of all the keys and their corresponding values in the data block are respectively recorded in the leaf nodes in the B+ tree, and all the nodes constituting the B+ tree are physically stored continuously.

Optionally, the step of merging the two first-level storage files may include: additionally writing an additional data block after the first storage file, where the value in the data block of the second storage file is written; and appending the data block after appending the data block Writing a new index block, the new index block is generated based on the index block of the first storage file and the index block of the second storage file, the index block of the first storage file and all the keys in the index block of the second storage file and The corresponding value is recorded in the leaf node of the new B+ tree in the data block of the first storage file and the logical address in the additional data block; the new file header is additionally written after the new index block to record the merged new file. Metadata information for the file.

Therefore, when the two files are merged, one file can be kept unchanged, and the value of the other file can be directly added and written, thereby improving the writing performance. And the merged index block is a new B+ tree, and the value in the merged file can be conveniently read according to the new index block, and the read performance of the merged file is not affected.

Optionally, the metadata information may include one or more of the following: the number of keys in the index block; the range of keys in the index block; the height of the B+ tree; the logical address of the first leaf node in the B+ tree; and the B+ tree The number of internal nodes.

Optionally, the database management method may further include: updating a file header of the first storage file according to the new file header to replace the metadata information in the file header of the first storage file with the metadata information in the new file header.

Since the additional write is a destructive write, the embodiment of the present invention can avoid the damage caused to the file by the abnormal situation during the merge process by setting the double file header.

Optionally, the file may include a front file header located at a file header and a subsequent file header located at a tail of the file, and the contents of the front file header and the subsequent file header are the same, and the front file header of the first storage file is updated according to the new file header, as The front file header of the new file, with the new file header as the post file header of the new file.

Optionally, the database management method may further include: when the step of writing the metadata information of the new file in the new file header is wrong, the new file is restored to the first before the merge according to the file header of the first storage file. The file is stored; and/or in the case where the step of updating the header of the first stored file is erroneous, the header of the first stored file is re-updated according to the new header.

Therefore, when an error occurs in the process of writing a new file header, since the file header of the first storage file has not been updated, the file in the merge process can be restored to the first before the merge according to the file header of the first storage file. The file is stored, and when an error occurs in the process of updating the header of the first stored file, it can be re-updated according to the new header.

Optionally, the database management method may further include: searching for a key corresponding to the request key in the memory table, in response to the request for finding the target value corresponding to the request key, and reading the target value in the case of finding; If the request key is not found in the memory table, the read-only memory table is searched for the key corresponding to the request key, and the target value is read in the case of the search; the read-only memory table is not found. In the case of the request key, it is chronologically searched for whether each of the first-level storage files in the disk has a key corresponding to the request key, and in the case of the search, the target value is read; and in each of the first-level storage files. If it is not found, use the binary search method to find whether the second-level storage file in the disk has the key corresponding to the request key, and read the target value if found.

Optionally, the database management method may further include: acquiring a file header and an index block of the target storage file in response to the request for reading the target value corresponding to the request key from the target storage file; determining, according to the file header, whether the request key is Within the range of the key indicated by the file header; in the case of determining that the request key is within the range of the key indicated by the file header, searching for the leaf node corresponding to the request key in the index block based on the B+ tree structure of the index block; In the case of the search, the target value is read in the logical address in the data block in the target storage file according to the value corresponding to the key stored by the found leaf node.

Optionally, the database management method may further include: in response to restarting the request for restoring the internal memory, constructing the second-level storage file list according to the size order of the range of the keys included in the second-level storage file; storing according to the first level The file serial number order of the file, constructing a first-level storage file list; determining, according to the first-level storage file list and the second-level storage file list, the writing progress of the data in the log file being written to the first-level storage file; According to the writing progress, the data in the log file that is not written to the first-level storage file is written to the memory table in the internal memory.

According to another aspect of the embodiments of the present invention, there is also provided a database system comprising: an internal memory and an external memory, wherein the internal memory is used to write a plurality of pieces of data to a log file in the external memory, and the external memory will log the file The data in the internal memory is written into the memory table in the internal memory, wherein the data written in the memory table is stored in order according to the size of the key. When the size of the memory table exceeds a predetermined threshold, the internal memory converts the memory table into a read-only memory table. The external memory writes the subsequent data in the log file to the new memory table, and the internal memory writes the data in the read-only memory table to the external accessor to obtain the first-level storage file, and the external memory merges two or more. The first level stores the text to get the second level storage file.

Optionally, the external storage specifies a primary file name of the first-level storage file by using a first naming rule, and specifies a primary file name of the second-level storage file by using a second naming rule, where the first naming rule is different from the second naming rule. In order to distinguish whether the storage file is a first-level storage file or a second-level storage file based on the main file name.

Optionally, the memory table is composed of a hash table, the hash table includes one or more hash buckets, and each hash bucket corresponds to one jump table, and each piece of data in the memory table constitutes an element of the jump table, wherein The order of the elements in the jump table is ordered according to the size of the keys.

Optionally, the read-only memory table queue is maintained in the internal memory, and the data in the read-only memory table is not all written to the external memory, and when the size of the new memory table exceeds a predetermined threshold, the external memory converts the new memory table into a new memory table. Make another read-only memory table and put it into a read-only memory table queue.

Optionally, the external memory merges the two first-level storage files by performing an operation of: additionally writing the additional data block after the first storage file, wherein the value in the data block of the second storage file is written; and appending the data block Then, a new index block is additionally written, and the new index block is generated based on the index block of the first storage file and the index block of the second storage file, and all of the index block of the first storage file and the index block of the second storage file are valid. The key and its corresponding value are recorded in the leaf node of the new B+ tree in the data block and the additional data block of the first storage file respectively; the new file header is additionally written after the new index block to record the merge Metadata information after the new file.

With the database management method and the database system of the embodiment of the present invention, the file finally stored in the external memory has only two hierarchical structures, and the file redundancy is low, which is convenient to find.

Other features and advantages of the present invention will become apparent from the Detailed Description of the <RTIgt;

DRAWINGS

The accompanying drawings, which are incorporated in FIG

1 and 3 are diagrams showing the data structure of a file involved in the file combining scheme of the present invention.

2 is a diagram showing the B+ tree structure of an index block of the present invention.

FIG. 4 is a schematic flow chart showing a file merging method according to an embodiment of the present invention.

5 and 6 show schematic views of a file merge state based on the present invention.

FIG. 7 shows a schematic flow chart of a method of reading data in an object file.

FIG. 8 is a functional block diagram showing a file merging apparatus according to an embodiment of the present invention.

FIG. 9 is a schematic diagram showing the structure in which the reading unit can also have a function module.

FIG. 10 is a schematic diagram showing a hardware configuration of an electronic device in which an embodiment of the present invention can be performed.

11 is a block diagram showing the structure of a database system according to an embodiment of the present invention.

FIG. 12 is a flow chart showing the data storage between the internal memory 110 and the external memory 120.

Figure 13 is a static diagram showing the process of storing data.

Figure 14 is a flow chart showing a complete lookup.

Figure 15 is a flow chart showing the lookup inside a file.

FIG. 16 is a schematic flow chart showing restart recovery according to an embodiment of the present invention.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the invention unless otherwise specified.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

In the examples shown and discussed herein, any specific values are to be construed as illustrative only and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

FIG. 1 is a block diagram showing a hardware configuration of an electronic device 1000 in which an embodiment of the present invention can be implemented.

The electronic device 1000 can be a portable computer, a desktop computer, a mobile phone, a tablet, or the like. As shown in FIG. 1, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, or the like. The memory 1200 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone jack, and the like. The communication device 1400 can, for example, perform wired or wireless communication, and specifically can include Wifi communication, Bluetooth communication, 2G/3G/4G/5G communication, and the like. The display device 1500 is, for example, a liquid crystal display, a touch display, or the like. Input device 1600 can include, for example, a touch screen, a keyboard, a somatosensory input, and the like. The user can input/output voice information through the speaker 1700 and the microphone 1800.

The electronic device shown in Figure 1 is merely illustrative and is in no way meant to limit the invention, its application or use. In the embodiment of the present invention, the memory 1200 of the electronic device 1000 is configured to store an instruction for controlling the processor 1100 to perform any of the file merging methods provided by the embodiments of the present invention or Database management method. It will be understood by those skilled in the art that although a plurality of devices are illustrated for electronic device 1000 in FIG. 1, the present invention may relate only to some of the devices therein, for example, electronic device 1000 relates only to processor 1100 and storage device 1200. A technician can design instructions in accordance with the disclosed aspects of the present invention. How the instructions control the processor for operation is well known in the art and will not be described in detail herein.

This embodiment mainly proposes a scheme of merging files stored in an external memory such as a hard disk, a floppy disk, an optical disk, or a USB disk. The key and value of the file described in the file merging method and apparatus of the present embodiment are separately stored, wherein the key is stored in the form of a B+ tree, whereby when merging the two files, one file can be kept. Moves the value of another file directly to the previous file, improves the write performance, and reconstructs the index block that stores the key in the form of a B+ tree. The new index block can be easily read from the merged file. Value, the read performance of the merged file will not be affected. Before describing in detail the file merging scheme of the present embodiment, the data structure of the file in the file merging scheme of the present embodiment will be described first.

FIG. 1 is a schematic diagram showing the data structure of a file in the file combining scheme of the present embodiment. As shown in FIG. 1, the file described in this embodiment can be physically divided into a file header, a data block, and an index block by blocks, and each block can be composed of a plurality of pages. The page referred to in this embodiment is the minimum unit of one I/O, which is generally an integer multiple of the system page, and the size of the pages of different types of blocks may be different.

The data block is used to store the value. The index block is used to store the key corresponding to the value in the form of a B+ tree. As shown in FIG. 2, the B+ tree is composed of a leaf node, an internal node, and a root node. The form of the B+ tree here is a person skilled in the art. As is well known, it will not be repeated here. It should be noted that each leaf node in the B+ tree corresponds to a key, and the logical addresses of all the keys and their corresponding values in the data block are respectively recorded in the leaf nodes in the B+ tree. That is, only the key is stored in the leaf node of the B+ tree, and no value is stored. Instead, the offset of the page in the data block where the value is located and the offset of the value in the page can be stored.

Optionally, all the nodes (root node, internal node, and leaf node) constituting the B+ tree are physically and continuously stored, thereby utilizing the local preloading feature of the disk to quickly acquire all nodes in the B+ tree, and the merge can be improved. The efficiency of building a new B+ tree in the process (the merge process will be explained in more detail below).

The header is used to record the metadata information of the file. The metadata information may include the number of keys in the index block, the range of keys in the index block, the height of the B+ tree, the logical address of the first leaf node in the B+ tree, and the number of internal nodes in the B+ tree.

So far, the data structure of the file in the file merging scheme of the present embodiment is briefly explained with reference to FIG. The data structure of the file shown in FIG. 1 is only an example, and it should be understood that it can also have various modifications. For example, as shown in FIG. 3, the file header of the file may include a front file header and a back file header, and the metadata information of the file recorded by the front file header and the subsequent file header may be the same. For another example, the file described in this embodiment may further include a filter, and the filter may be used to determine whether the accessed key is in the file. For example, the filter may be a Bloom filter, and the access does not exist. Key, you can use the Bloom filter to quickly determine that the key does not exist, and do not have to go to the B+ tree to query. Because the Bloom filter is actually a hash table, you can judge the existence of the key in the complexity of O(1), and the search time complexity of the B+ tree is O(logn), so you can set the Bloom filter. Improve search efficiency, which can improve read performance.

The file combining scheme of this embodiment will be described in detail below with reference to FIGS. 4 to 9. FIG. 4 is a schematic flowchart showing a file merging method according to an embodiment of the present invention, including steps S210 to S230. The method can combine two or more files. For convenience of description, the first file and the second file are combined as an example for description.

Referring to FIG. 4, in step S210, an additional data block is additionally written after the first file, in which the value in the data block of the second file is written.

Here, the freshness of the second file may be greater than the first file, that is, the second file may be stored later in the external memory, and the first file may be previously stored in the external memory.

Since the values and keys in the file described in this embodiment are separately stored, when the first file and the second file are merged, the value in the data block of the second file may be additionally appended after the first file. Here, a block in which a value is added after the first file is referred to as an additional data block. That is to say, the value in the data block of the second file can be rewritten in the additional data block after the first file, so that the end of the file F and the address of the additional data block are consecutive.

After the value in the data block of the second file is additionally appended to the first file, new index information can be created, that is, in step S220, the new index block is additionally written after the additional data block.

Here, the new index block is generated based on the index block of the first file and the index block of the second file. As described above, the freshness of the second file may be greater than the first file, so the key value in the second file may be a modification, deletion, replacement, etc. of the key value in the first file, and thus for the first file and The same key exists in the index block of the second file, and the key in the second file with higher freshness can be selected as the valid key, and the key in the first file is discarded, thereby constructing a new index block.

That is to say, the keys in the generated new index block are all valid keys, and the corresponding values are all valid values. The key in the new index block is also stored in the form of a B+ tree which is regenerated according to the index block of the first file and the index block of the second file, and thus may be referred to as a new B+ tree. The index block of the first file and all the valid keys in the index block of the second file and their corresponding values are respectively recorded in the data block of the first file and the logical address in the additional data block in the leaf of the new B+ tree. In the node.

As described above, the index block of the first file and all the nodes of the B+ tree in the index block of the second file are physically stored continuously, so that in the process of reconstructing the new B+ tree, the local portion of the disk can be utilized. The feature of the preloading feature is that the index block of the first file and the index block of the second file can be obtained by simply traversing successive disk blocks, thereby improving the construction efficiency of the new B+ tree.

After constructing a new B+ tree to generate a new index block, the index block in the first file is invalidated and replaced by the new index block. Among them, the invalidity mentioned here means that in the subsequent search process, the new index block is used for searching, and the old index block is no longer used. That is, after generating a new index block, the old index block may not be deleted.

In step S230, a new file header is additionally written after the new index block to record the metadata information of the merged new file.

The metadata information of the new file may include the number of keys in the new index block, the range of keys in the new index block, the height of the new B+ tree, the logical address of the first leaf node in the new B+ tree, and the internal nodes in the new B+ tree. Number and so on. After generating a new file header, you can delete the second file and free up storage space.

FIG. 5 is a schematic diagram showing a merge process of merging G files into F files according to an embodiment of the present invention.

According to FIG. 5 and the description above with reference to FIG. 3, in the merge process, the F file is unchanged, and only the value in the G file needs to be additionally written into the F file, and a new index block and a new file header are generated. Compared with the existing LevelDB, it is necessary to take out the key value and reconstruct the comparison. The merge process is simpler, and according to the merged B+ tree, the value corresponding to the key in the file can be conveniently found, and the read performance is improved. .

FIG. 6 is a schematic diagram showing another example of a merge process of merging G files into F files according to an embodiment of the present invention.

Different from FIG. 5, both the F file and the G file in FIG. 6 include a front file header located at the head of the file and a rear file header located at the end of the file. The contents of the front file header and the last file header are the same.

Different from the above-mentioned merge process, after the new file header is additionally written, the previous file header of the F file can be updated according to the new file header as the front file header of the new file, and the new file header is used as the new file header. The file header of the file.

This allows you to maintain two file headers during the file merge process. This is because the append write during the merge process is a kind of "destructive write", that is, when the G file is merged into the F file, the F file is destroyed. Among them, the destructive writing mentioned here refers to the merge of the G file into the F file, and the new file header of the merged new file records the metadata information of the merged new file, and the file header of the F file before the merge. It is invalid, so if no protection measures are taken, the F file will not be repaired once the merge process fails. Therefore, the present invention adopts a method of maintaining a double file header, and can solve the problem that the file cannot be recovered due to an abnormal situation.

Specifically, when the merge is completed normally, the first two files of the new file can be updated normally and are the same. In the event of an abnormal situation that needs to be restored, it is ok to take the header of the file as it is.

An exception occurs if a new file header has not been written at the end. Since the header of the header at this time has not been updated, it is still intact, but it is only old. Through this file header, the residual information of the last merged unfinished can be truncated to get an old version of the complete file.

An exception occurs if the file header is updated before. Since the new file header is already complete at this time, as long as the new file header is used for recovery. That is, the previous file header can be re-updated with the new header to ensure the integrity and consistency of the two headers in the initial state.

FIG. 7 is a schematic flowchart showing a method of reading a target value corresponding to a request key from a file.

Referring to FIG. 7, in step S310, a file header and an index block of the target file are acquired.

In step S320, it is determined according to the file header whether the request key is within the range of the key indicated by the file header. If not, it indicates that the value corresponding to the request key does not exist in the target file, and the reading ends.

In the case where it is determined that the request key is in the range, step S330 is performed to find a leaf node corresponding to the request key in the index block based on the B+ tree structure of the index block. When the leaf node corresponding to the request key is not found in the index block, it indicates that the value corresponding to the request key does not exist in the target file, and the reading ends. In the case of finding, step S340 may be performed to read the target value in the logical address in the data block in the target file according to the value corresponding to the key stored by the found leaf node.

FIG. 8 is a functional block diagram showing a file merging apparatus according to an embodiment of the present invention. The functional modules of the file combining apparatus 500 may be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present invention. It will be understood by those skilled in the art that the functional modules described in FIG. 7 can be combined or divided into sub-modules to implement the principles of the above described invention. Accordingly, the description herein may support any possible combination, or division, or further limitation of the functional modules described herein.

The file merging device 500 shown in FIG. 8 can be used to implement the detecting method shown in FIG. 3 to FIG. 6. The following is only a brief description of the functional modules that the file merging device 500 can have and the operations that can be performed by the functional modules. For details of the reference, refer to the description above with reference to FIG. 3 to FIG. 6 , and details are not described herein again.

As shown in FIG. 8, the file combining apparatus 500 includes a first writing unit 510, a B-tree generating unit 520, a second writing unit 530, and a third writing unit 540.

The first writing unit 510 is configured to write an additional data block after the first file, wherein the value in the data block of the second file is written.

The B-tree generating unit 520 is configured to generate a new B+ tree based on the index block of the first file and the index block of the second file, all the keys in the index block of the first file and the index block of the second file, and each key The logical addresses in the data block and the additional data block of the first file are respectively recorded in the leaf nodes in the new B+ tree;

The second writing unit 530 is configured to additionally write a new index block after appending the data block, in which a new B+ tree is written.

The third writing unit 540 is configured to additionally write a new file header after the new index block to record the metadata information of the merged new file.

As shown in FIG. 8, the file combining apparatus 500 may also optionally include an updating unit 550. The update unit 550 can update the file header of the first file according to the new file header to replace the metadata information in the file header of the first file with the metadata information in the new file header.

Specifically, the file may include a front file header located at the head of the file and a subsequent file header located at the end of the file, and the contents of the front file header and the subsequent file header are the same. The update unit 550 can update the previous file header of the first file as the previous file header of the new file according to the new file header, and use the new file header as the post file header of the new file.

As shown in FIG. 8, the file combining apparatus 500 may further include a first restoring unit 560 and a second restoring unit 570.

The first restoration unit 560 may restore the new file to the first file before the merge according to the file header of the first file in the case where the step of writing the metadata information of the new file in the new file header is erroneous.

The second restoration unit 570 may re-update the file header of the first file according to the new file header in the case where the step of updating the file header of the first file is erroneous.

As shown in FIG. 8, the file combining apparatus 500 may further include a reading unit 580. The reading unit 580 can read the target value corresponding to the request key from the target file. FIG. 8 is a functional block diagram showing functional modules that a reading unit can have.

As shown in FIG. 9, the reading unit 580 may include an obtaining module 581, a determining module 583, a searching module 585, and a reading module 587.

The obtaining module 581 can acquire the file header and the index block of the target file, and the determining module 583 can determine, according to the file header, whether the request key is within the range of the key indicated by the file header. In the case where it is determined that the request key is within the range, the lookup module 585 can look up the leaf node corresponding to the request key in the index block based on the B+ tree structure of the index block. The reading module 587 can read the target value in the logical address in the data block in the target file according to the value corresponding to the key stored by the found leaf node.

In this embodiment, an electronic device is further provided, including a memory and a processor. The memory is configured to store executable instructions, and the processor is configured to execute the electronic device to perform any one of the file merging methods provided in the embodiment according to the control of the executable instructions. In one example, the electronic device can be an electronic device 1000 as shown in FIG.

The file merging method and apparatus according to the present embodiment have been described in detail above with reference to the accompanying drawings. According to this embodiment, the keys and values of the file can be stored separately, and the keys are stored in the form of a B+ tree. Therefore, when the two files are merged, one file can be kept unchanged, and the value of the other file can be directly added and written, thereby improving the writing performance. And the merged index block is a new B+ tree, and the value in the merged file can be conveniently read according to the new index block, and the read performance of the merged file is not affected.

In the prior art, LevelDB exposes many shortcomings in many aspects such as reading, writing, merging, data cleaning, restarting recovery, etc. To this end, this embodiment proposes a new database management method and database system.

FIG. 11 is a block diagram showing the structure of a database system according to an embodiment of the present invention. As shown in FIG. 1, the database system 100 of the present invention mainly includes an internal memory 110 and an external memory 120. The internal memory 110 and the external memory 120 can cooperate to complete data storage.

FIG. 12 is a flow chart showing the cooperation between the internal memory 110 and the external memory 120 to implement data storage.

Referring to FIG. 12, first in step S110, a plurality of pieces of data to be stored may be written by the internal memory 110 to a log file in the external memory 120. Each piece of data includes corresponding keys and values, and the log files can be sequentially written in the order in which the data arrives.

Then, step S120 may be performed to write the data in the log file to the memory table in the internal memory 110 by the external memory 120. Among them, the data written in the memory table can be stored in order according to the size of the key. For example, the data stored in the memory table may adopt a jump table structure, so that the data stored in the memory table is arranged in an order according to the size of the key. For example, the memory table may be composed of a hash table, where the hash table may include one or more hash buckets, each hash bucket corresponds to one jump table, and each data in the memory table constitutes one of the jump tables. An element in which the order of the elements in the jump table is ordered in order of the size of the key.

Therefore, a hash table is embedded before the jump table, so that the lock granularity can be reduced on the one hand, and for concurrent read and write operations, if the keys are not the same, the fast lookup can be performed in the jump table corresponding to the respective hash bucket. insert. On the other hand, while expanding the size of the memory table, the size of the jump table is not enlarged, which can reduce the probability that the jump table becomes a linear search as the amount of data becomes larger, and improves the overall search efficiency.

When the data written in the memory table is gradually increased, so that the size of the memory table exceeds a predetermined threshold, the internal memory 110 can convert the memory table into a read-only memory table (step S130), at which time the internal memory 110 is not written in the log file. The data can be written to a new memory table. As the name suggests, read-only memory tables can only be read and cannot be written.

It should be noted that the log file in the external memory 120 and the memory table in the internal memory 110 may have a one-to-one correspondence, that is, for a key-Value data, it may be written to the log file, and then from the log file. When writing to a memory table, when the size of the memory table exceeds a predetermined threshold and needs to be converted into a read-only memory table, the newly arrived data can be written into a new log file, and the data in the new log file can be written into the new memory table.

After the memory table is converted into the read-only memory table, step S140 may be performed, and the data in the read-only memory table may be written into the external memory 120 by the internal memory 110 to obtain the first-level storage file. The external memory 120 may perform step S150 to merge two or more first-level storage files stored therein to obtain a second-level storage file.

In addition, the external memory 120 may specify a primary file name of the first-level storage file according to the first naming rule, and may specify a primary file name of the second-level storage file according to the second naming rule, where the first naming rule and the second naming The rules can be set differently to distinguish whether the storage file is a first-level storage file or a second-level storage file based on the primary file name. For example, "_0" can be added after the main file name of the first-level storage file, and "_1" is added after the main file name of the second-level storage file. That is, the first level storage file and the second level storage file can be named by xxx_0.hdb, xxx_1.hdb respectively.

So far, the storage flow of the external memory and the internal memory in the database system to realize the persistent storage of data to the external memory is briefly explained in conjunction with FIG. Figure 13 is a static diagram showing the process of storing data.

As shown in Figure 13, the read-only memory table queue can be maintained in the internal memory. The data in the read-only memory table is not all written to the external memory, and the new memory table will be new when the size of the new memory table exceeds a predetermined threshold. Convert to another read-only memory table and put it into a read-only memory table queue. Therefore, by maintaining the memory table queue, it is possible to cope with the problem of blocking when the high frequency is written, because the data is too late to be merged, and the memory table is full.

The structure of the database system of the present embodiment and the data storage flow of the database system for persistently storing data in the external memory have been described so far with reference to FIGS. 11 to 13. The following describes the process of merging the storage files stored in the external storage, the data search process, and the data recovery process when the database system is restarted under special circumstances.

First, the storage file consolidation process

Before the detailed description of the file merging process of the present invention, the data structure of the storage file stored in the external memory is first described only.

In Fig. 1, a schematic diagram of a data structure of a storage file stored in an external memory has been shown. As shown in FIG. 1, the files described in the present invention can be physically divided into file headers, data blocks, and index blocks by blocks, and each block can be composed of a plurality of pages. Among them, the page mentioned in this paper is the minimum unit of I/O, which is generally an integer multiple of the system page. The size of the pages of different types of blocks can be different.

The data block is used to store the value. The index block is used to store the key corresponding to the value in the form of a B+ tree. The form of the B+ tree is well known to those skilled in the art and will not be described here. It should be noted that each leaf node in the B+ tree corresponds to a key, and the logical addresses of all the keys and their corresponding values in the data block are respectively recorded in the leaf nodes in the B+ tree. That is, only the key is stored in the leaf node of the B+ tree, and no value is stored. Instead, the offset of the page in the data block where the value is located and the offset of the value in the page can be stored.

So far, the data structure of the storage file stored in the external memory has been briefly explained in conjunction with FIG. The data structure of the file shown in FIG. 1 is only an example, and it should be understood that it can also have various modifications. For example, as shown in FIG. 3, the file header of the storage file may include a front file header and a back file header, and the metadata information of the file recorded by the front file header and the subsequent file header may be the same. For another example, the storage file may further include a filter, and the filter may be used to determine whether the accessed key is in the file, for example, the filter may be a Bloom filter, and for accessing a key that does not exist, the Bronze may be used. The filter quickly determines that the key does not exist, and does not need to go to the B+ tree to query. Because the Bloom filter is actually a hash table, you can judge the existence of the key in the complexity of O(1), and the search time complexity of the B+ tree is O(logn), so you can set the Bloom filter. Improve search efficiency, which can improve read performance.

The merging process of the storage file will be described in detail below with reference to FIGS. 4 to 6. FIG. 4 is a schematic flow chart showing a method of merging storage files according to an embodiment of the present invention. The method may combine two or more storage files, wherein two or more first-level storage files may be combined into one second-level storage file, or two or more seconds may be combined. The level stores the file and generates a new second-level storage file. For the convenience of description, the merging process of the storage file of this embodiment is described here by taking the first storage file and the second storage file as an example.

Referring to FIG. 4, in step S210, an additional data block is additionally written after the first storage file, wherein the value in the data block of the second storage file is written.

Here, the freshness of the second storage file may be greater than the first storage file, that is, the second storage file may be stored later in the external storage, and the first storage file may be previously stored in the external storage.

Since the value and the key in the storage file are separately stored, when the first storage file and the second storage file are merged, the value written in the data block of the second storage file may be appended after the first storage file, where A block in which a write value can be added after the first storage file is referred to as an additional data block. That is to say, the value in the data block of the second storage file can be rewritten in the additional data block after the first storage file, so that the end of the file F and the address of the additional data block are consecutive.

After the value written in the data block of the second storage file is added after the first storage file, new index information can be created, that is, in step S220, the new index block is additionally written after the additional data block.

Here, the new index block is generated based on the index block of the first storage file and the index block of the second storage file. As described above, the freshness of the second storage file may be greater than the first storage file, so the key value in the second storage file may be a modification, deletion, replacement, etc. of the key value in the first storage file, and thus The same key existing in the index block of the first storage file and the second storage file may select a key in the second storage file with higher freshness as a valid key, and discard the key in the first storage file to construct a new key Index block.

That is to say, the keys in the generated new index block are all valid keys, and the corresponding values are all valid values. The key in the new index block is also stored in the form of a B+ tree, which is regenerated according to the index block of the first storage file and the index block of the second storage file, and thus may be referred to as a new B+ tree. The index key of the first storage file and all the valid keys in the index block of the second storage file and their corresponding values are respectively recorded in the new B+ tree in the data block of the first storage file and the logical address in the additional data block. In the leaf node.

As described above, all the nodes of the B+ tree in the index block of the first storage file and the index block of the second storage file are physically stored continuously, so that the disk can be utilized in the process of reconstructing the new B+ tree. The local preloading feature can obtain the index block of the first storage file and the index block of the second storage file by simply traversing successive disk blocks, thereby improving the construction efficiency of the new B+ tree.

After constructing a new B+ tree to generate a new index block, the index block in the first storage file is invalidated and replaced by the new index block. Among them, the invalidity mentioned here means that in the subsequent search process, the new index block is used for searching, and the old index block is no longer used. That is, after generating a new index block, the old index block may not be deleted.

The metadata information of the new file may include the number of keys in the new index block, the range of keys in the new index block, the height of the new B+ tree, the logical address of the first leaf node in the new B+ tree, and the internal nodes in the new B+ tree. Number and so on. After generating a new file header, you can delete the second storage file and free up storage space.

FIG. 5 is a schematic diagram showing a merge process of merging G files into F files.

According to FIG. 5 and the description above with reference to FIG. 4, in the merge process, the F file is unchanged, and only the value in the G file needs to be additionally written into the F file, and a new index block and a new file header are generated. Compared with the existing LevelDB, it is necessary to take out the key value and reconstruct the comparison. The merge process is simpler, and according to the merged B+ tree, the value corresponding to the key in the file can be conveniently found, and the read performance is improved. .

FIG. 6 is another schematic diagram showing a merge process of merging G files into F files.

Thus, in the file merging process, two file headers can be maintained, because the additional write during the merge process is also a kind of "destructive write", that is, when the G file is merged into the F file, it will be destroyed. F file. The destructive writing mentioned here refers to the fact that when the G file is merged into the F file, the new file header of the merged new file records the metadata information of the merged new file, and the F file before the merge. The file header is invalid, so if no protection measures are taken, the F file will not be repaired once the merge process fails. By maintaining the double file header, you can solve the problem that the file cannot be recovered due to abnormal conditions.

Second, the data search process

According to the description of the data storage process above, when the data is persistently stored in the storage file in the external storage, the storage process is first written to the memory table, then written to the read-only memory table, and then written to the external The first level storage file in the memory, the first level storage file is merged into the second level storage file. Therefore, the freshness of the data is decremented according to the memory table, the read-only memory table, the first-level storage file, and the second-level storage file.

Therefore, when reading data, it can be read from the memory table first. If the memory table cannot be read, and then read from the read-only memory table, if the read-only memory table cannot be read, then The first-level storage file in the external storage is searched, the first storage file is not found, and then the second-level storage file is searched.

Figure 14 is a flow chart showing a complete lookup. Referring to FIG. 6, step S410 may first be performed to find in the memory table whether there is a key corresponding to the request key. For example, when the data in the memory table is stored in the form of a hash table and a jump table, it may first be located according to the request key to the specific hash bucket in the memory table, and then searched in the corresponding jump table.

If you find it in the memory table, you can read it directly. If it is not found in the memory table, it can continue to find in the read-only memory table in the internal memory whether or not there is a key corresponding to the request key (step S420). Wherein, when the memory memory maintains a read-only memory table queue with multiple read-only memory tables, the read-only memory table in the read-only memory table queue can be searched one by one in chronological order.

If it is not found in the read-only memory table, it can be searched from the first-level storage file in the external memory. Here, it can be chronologically searched whether each first-level storage file in the external storage has a request key corresponding to it. Key (step S430).

In the case where it is found that a certain first-level storage file has a key corresponding to the request key, the value corresponding to the request key can be read from the first-level storage file. If it is not found, it can be searched from the second-level storage file in the external storage. Here, it can be used to find whether the second-level storage file has the value corresponding to the request key in the convenient time of the binary search (step S440).

In the case where it is found that a certain second-level storage file has a key corresponding to the request key, the value corresponding to the request key can be read from the second-level storage file. In the case that it is not found, it indicates that the request key and the corresponding value are not stored in the database system.

Figure 15 is a flow chart showing the lookup inside a file. Referring to FIG. 15, the file header and the index block of the target storage file may be first acquired (step S510), and then step S520 is performed to determine, according to the file header, whether the request key is within the range of the key indicated by the file header, and if not, indicating the target storage. The value corresponding to the request key does not exist in the file, and the reading ends.

In the case that it is determined that the request key is in the range, step S530 may be performed to find a leaf node corresponding to the request key in the index block based on the B+ tree structure of the index block. When the leaf node corresponding to the request key is not found in the index block, it indicates that the value corresponding to the request key does not exist in the target storage file, and the reading ends. In the case of finding, step S540 may be performed to read the target value according to the logical address in the data block in the target storage file according to the value corresponding to the key stored by the found leaf node.

Third, restart the recovery process

In LevelDB, restarting is an annoying thing. Because it needs to recover the data in the internal memory from the MANIFEST and Current two manifest files, as the amount of data grows, these two files may be very large, especially for the Current file, and the GB is also very common. As a result, it sometimes takes dozens of minutes to reboot. Worse, if the manifest file is lost, the entire library will be unavailable. In the database system of the present invention, since the description information of each file is completely described in its own index block, file header, and the like, and the information of these blocks is often not large, as long as the block is restarted, By reading the corresponding information, the metadata of the entire file can be completely recovered. Even if a file is damaged, it will not cause the entire library to be unavailable. Even a library of 100 gigabytes can be restarted in seconds.

Fig. 16 is a schematic flow chart showing the restart of the restart according to the present embodiment. The sequence between step S610 and step S620 is not required, and may be performed at the same time or at different times.

Referring to FIG. 16, in step S610, a second level storage file list is constructed. Specifically, the index block of the second-level storage file, the filter block (in some cases), the file header, and the like may be pre-loaded by means of memory mapping, and then the second-level storage file list is constructed according to the range order of the keys.

At step S620, a first level storage file list is constructed. Specifically, the index block of the first-level storage file, the filter block (in some cases), the file header, and the like may be pre-loaded by means of memory mapping, and then the first-level storage file list is constructed according to the range order of the keys.

After the first-level storage file list and the second-level storage file list are constructed, it is possible to determine that the write of the log file written to the first-level storage file is entered (step S630), so that the write progress can be made according to the progress. The memory table and the read-only memory table in the internal memory are constructed (step S640).

As described above, there may be multiple log files written in the external memory, which are respectively corresponding to the memory table (or the read-only memory table), so that it can be determined according to the constructed first file list and the second file list. The data in those log files in multiple log files is not written to the storage file. The log file that is not written to the storage file can then be converted to a read-only memory table, where the data in the log file can be written to the memory table for the last generated log file. Thus, the recovery of the memory table and the read-only memory table in the internal memory can be completed.

The database management method and database system according to the present invention have been described in detail above with reference to the accompanying drawings. According to the embodiment, the file finally stored in the external memory has only two hierarchical structures, and the file redundancy is low, which is convenient to find.

In the present embodiment, a database management method, a database system, and an electronic device as described below are also provided.

Aspect 1. A database management method for storing a plurality of pieces of data, wherein each of the pieces of data includes a corresponding key and a value, the method comprising:

Writing the plurality of pieces of data to a log file in an external memory;

Writing the data in the log file to the memory table in the internal memory, wherein the data written in the memory table is stored in an orderly manner according to the size of the key;

When the size of the memory table exceeds a predetermined threshold, converting the memory table into a read-only memory table, and writing subsequent data in the log file to a new memory table;

Writing data in the read-only memory table to an external accessor to obtain a first-level storage file;

Combine two or more first level storage files to get a second level storage file.

Aspect 2. The data block management method of aspect 1, further comprising:

Specifying a primary file name of the first level storage file in a first naming rule;

Specifying a primary file name of the second-level storage file by using a second naming rule, the first naming rule being different from the second naming rule, so as to distinguish whether the storage file is a first-level storage file or a second based on the primary file name. Level storage file.

The database management method according to aspect 1, wherein the memory table is composed of a hash table, the hash table includes one or more hash buckets, and each hash bucket corresponds to one jump table. Each piece of data in the memory table constitutes an element of the hop table, wherein the order of the elements in the hop table is ordered in order according to the size of the key.

Aspect 4. The database management method of aspect 1, further comprising:

Maintaining a read-only memory table queue in the internal memory, the data in the read-only memory table is not all written to the external memory, and when the size of the new memory table exceeds a predetermined threshold, the new memory table is converted Make another read-only memory table and put it into the read-only memory table queue.

The database management method of aspect 1, wherein the data structure of the storage file comprises:

a file header for recording metadata information of the storage file;

a data block for storing values;

An index block, configured to store, in a B+ tree, a key corresponding to the value, wherein a logical address of all the keys and their corresponding values in the data block are respectively recorded in a leaf node in the B+ tree, And all nodes constituting the B+ tree are physically stored continuously.

Aspect 6. The database management method of aspect 5, wherein the step of merging the two first level storage files comprises:

Appending an additional data block after the first storage file, wherein the value in the data block of the second storage file is written;

Adding a new index block after the additional data block, the new index block being generated based on an index block of the first storage file and an index block of the second storage file, where the first storage file is All valid keys in the index block and the index block of the second storage file and their corresponding values are respectively recorded in the new B+ tree in the data block of the first storage file and the logical address in the additional data block. In the leaf node;

A new file header is additionally written after the new index block to record metadata information of the merged new file.

Aspect 7. The database management method of aspect 6, wherein the metadata information comprises one or more of the following:

The number of keys in the index block;

a range of keys in the index block;

The height of the B+ tree;

The logical address of the first leaf node in the B+ tree;

The number of internal nodes in the B+ tree.

Aspect 8. The database management method of aspect 6, further comprising:

Updating a file header of the first storage file according to the new file header to replace metadata information in a file header of the first storage file with metadata information in the new file header.

Aspect 9. The database management method according to aspect 8, wherein

The file includes a front file header located at a file header and a subsequent file header located at a tail of the file, and the content of the front file header and the subsequent file header are the same.

Updating a front file header of the first storage file according to the new file header as a front file header of the new file, and using the new file header as a post file header of the new file.

Aspect 10. The database management method of aspect 8 or 9, further comprising:

In the case that the step of writing the metadata information of the new file in the new file header is in error, the new file is restored to the first storage file before the merge according to the file header of the first storage file; and/or

In the case where the step of updating the file header of the first storage file is erroneous, the file header of the first storage file is re-updated according to the new file header.

Aspect 11. The database management method of any of aspects 1-9, further comprising:

Responding to a request for finding a target value corresponding to the request key, searching in the memory table for whether there is a key corresponding to the request key, and reading the target value if found;

If the request key is not found in the memory table, searching for the key corresponding to the request key from the read-only memory table, and reading the target value if found ;

If the request key is not found in the read-only memory table, whether each of the first-level storage files in the external memory has a key corresponding to the request key is searched one by one in time series, in the search Reading the target value if it is; and

If not found in each of the first-level storage files, use a binary search to find whether the second-level storage file in the disk has a key corresponding to the request key, in the case of finding The target value is read below.

Aspect 12. The database management method of aspect 11, further comprising:

Obtaining a file header and an index block of the target storage file in response to the request to read the target value corresponding to the request key from the target storage file;

Determining, according to the file header, whether the request key is within a range of keys indicated by the file header;

In a case of determining that the request key is within a range of a key indicated by the file header, searching for a leaf node corresponding to the request key in the index block based on a B+ tree structure of the index block;

In the case of the search, the target value is read at a logical address in the data block in the target storage file according to the value corresponding to the key stored by the found leaf node.

Aspect 13. The database management method of any of aspects 1-9, further comprising:

Responding to restarting the request to restore the internal memory, constructing the second-level storage file list according to the size order of the range of the keys included in the second-level storage file;

Constructing a first-level storage file list according to the file serial number order of the first-level storage file;

Determining, according to the first-level storage file list and the second-level storage file list, a writing progress of the data in the log file being written to the first-level storage file;

According to the write progress, a memory table and a read-only memory table in the internal memory are constructed.

Aspect 14. A database system comprising: an internal memory and an external memory, wherein

The internal memory is used to write a plurality of pieces of data into a log file in an external memory.

The external memory writes data in the log file to a memory table in an internal memory, wherein data written in the memory table is stored in an orderly manner according to a size of a key.

When the size of the memory table exceeds a predetermined threshold, the internal memory converts the memory table into a read-only memory table, and the external memory writes subsequent data in the log file to a new memory table.

The internal memory writes data in the read-only memory table into an external accessor to obtain a first-level storage file.

The external memory merges two or more first level storage files to obtain a second level storage file.

Aspect 15. The database system of aspect 14, wherein

The external storage specifies a primary file name of the first-level storage file by a first naming rule, and specifies a primary file name of the second-level storage file by a second naming rule, the first naming rule and the The second naming rule is different to distinguish whether the storage file is a first-level storage file or a second-level storage file based on the primary file name.

The database system according to aspect 14, wherein the memory table is composed of a hash table, the hash table includes one or more hash buckets, and each hash bucket corresponds to a jump table. Each piece of data in the memory table constitutes an element of the hop table, wherein the order of the elements in the hop table is ordered in order according to the size of the key.

Aspect 17. The database system of aspect 14, wherein

Maintaining a read-only memory table queue in the internal memory, wherein data in the read-only memory table is not all written to the external memory, and when the size of the new memory table exceeds a predetermined threshold, the external memory will be new The memory table is converted into another read-only memory table and placed in the read-only memory table queue.

The database system of aspect 14, wherein the data structure of the storage file comprises:

a file header for recording metadata information of the storage file;

a data block for storing values;

The database system of aspect 18, wherein the external memory merges the two first level storage files by performing the following operations:

Adding a new index block after the additional data block, the new index block being generated based on an index block of the first storage file and an index block of the second storage file, where the first storage file is All the keys in the index block and the index block of the second storage file and their corresponding values are respectively recorded in the new B+ tree in the data block of the first storage file and the logical address in the additional data block. In the leaf node;

Aspect 20. An electronic device comprising:

a memory for storing executable instructions;

And a processor, configured to execute the management method of any one of the databases described in aspects 1-13, according to the control of the executable instruction. The invention can be a system, method and/or computer program product. The computer program product can comprise a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement various aspects of the present invention.

The computer readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, with instructions stored thereon A raised structure in the hole card or groove, and any suitable combination of the above. A computer readable storage medium as used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, a light pulse through a fiber optic cable), or through a wire The electrical signal transmitted.

The computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages. Source code or object code written in any combination, including object oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. carried out. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider to access the Internet) connection). In some embodiments, the customized electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams can be implemented by computer readable program instructions.

The computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or a processor of other programmable data processing apparatus to produce a machine such that when executed by a processor of a computer or other programmable data processing apparatus Means for implementing the functions/acts specified in one or more of the blocks of the flowcharts and/or block diagrams. The computer readable program instructions can also be stored in a computer readable storage medium that causes the computer, programmable data processing device, and/or other device to operate in a particular manner, such that the computer readable medium storing the instructions includes An article of manufacture that includes instructions for implementing various aspects of the functions/acts recited in one or more of the flowcharts.

The computer readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on a computer, other programmable data processing device or other device to produce a computer-implemented process. Thus, instructions executed on a computer, other programmable data processing apparatus, or other device implement the functions/acts recited in one or more of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a portion of an instruction that includes one or more components for implementing the specified logical functions. Executable instructions. In some alternative implementations, the functions noted in the blocks may also occur in a different order than those illustrated in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented in a dedicated hardware-based system that performs the specified function or function. Or it can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Numerous modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements in the various embodiments of the embodiments, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

A file merging method, the file being stored in an external memory, including a file header, a data block, and an index block, wherein the file header is used to record metadata information of the file, and the data block is used to store a value, the index The block is configured to store the key corresponding to the value in the form of a B+ tree, wherein the logical addresses of all the keys and their corresponding values in the data block are respectively recorded in the leaf nodes in the B+ tree, the method include:

Appending an additional data block after the first file, wherein the value in the data block of the second file is written;

Adding a new index block after the additional data block, the new index block being generated based on the index block of the first file and the index block of the second file, the index block of the first file and All the valid keys in the index block of the second file and their corresponding values are respectively recorded in the leaf nodes in the new B+ tree in the data block of the first file and the logical address in the additional data block;

A new file header is additionally written after the new index block to record metadata information of the merged new file.
The file merging method according to claim 1, wherein the metadata information comprises one or more of the following:

The number of keys in the index block;

a range of keys in the index block;

The height of the B+ tree;

The logical address of the first leaf node in the B+ tree;

The number of internal nodes in the B+ tree.
The file merging method according to claim 1 or 2, wherein all nodes constituting the B+ tree are physically stored contiguously.
The file merging method according to any one of claims 1 to 3, further comprising:

Updating a file header of the first file according to the new file header to replace metadata information in a file header of the first file with metadata information in the new file header.
A file merging method according to any one of claims 1 to 4, wherein

The file includes a front file header located at a file header and a subsequent file header located at a tail of the file, and the content of the front file header and the subsequent file header are the same.

Updating a front file header of the first file according to the new file header as a front file header of the new file, and using the new file header as a post file header of the new file.
The file merging method according to any one of claims 1 to 5, further comprising:

In the case where the step of writing the metadata information of the new file in the new file header is in error, the new file is restored to the first file before the merge according to the file header of the first file; and/or

In the case where the step of updating the file header of the first file is erroneous, the file header of the first file is re-updated according to the new file header.
The file merging method according to any one of claims 1 to 7, further comprising the step of: reading a target value corresponding to the request key from the object file:

Obtain the file header and index block of the target file;

Determining, according to the file header, whether the request key is within a range of keys indicated by the file header;

In a case of determining that the request key is within the range, searching for a leaf node corresponding to the request key in the index block based on a B+ tree structure of the index block;

The target value is read at a logical address in a data block in the target file according to a value corresponding to the key stored by the found leaf node.
A file merging device, the file being stored in an external memory, including a file header, a data block, and an index block, wherein the file header is used to record metadata information of the file, and the data block is used to store a value, the index The block is configured to store the key corresponding to the value in the form of a B+ tree, wherein the logical addresses of all the keys and their corresponding values in the data block are respectively recorded in leaf nodes in the B+ tree, the device include:

a first writing unit, configured to write an additional data block after the first file, where the value in the data block of the second file is written;

a B-tree generating unit, configured to generate a new B+ tree based on the index block of the first file and the index block of the second file, all of the index block of the first file and the index block of the second file The valid key and its corresponding value are respectively recorded in the data block of the first file and the logical address in the additional data block in the leaf node in the new B+ tree;

a second writing unit, configured to additionally write a new index block after the additional data block, where the new B+ tree is written;

And a third writing unit, configured to additionally write a new file header after the new index block, to record metadata information of the merged new file.
The file merging device according to claim 8, wherein the metadata information comprises one or more of the following:

The number of keys in the index block;

a range of keys in the index block;

The height of the B+ tree;

The logical address of the first leaf node in the B+ tree;

The number of internal nodes in the B+ tree.
The file merging device according to claim 8 or 9, further comprising:

And an updating unit, configured to update a file header of the first file according to the new file header to replace metadata information in a file header of the first file with metadata information in the new file header.
A file merging device according to any one of claims 8 to 10, wherein

The file includes a front file header located at a file header and a subsequent file header located at a tail of the file, and the content of the front file header and the subsequent file header are the same.

The update unit updates a front file header of the first file as a front file header of the new file according to the new file header, and uses the new file header as a post file header of the new file.
The file merging device according to any one of claims 8 to 11, further comprising:

a first restoring unit, configured to restore the new file to the first before the merge according to the file header of the first file in the case that the step of writing the metadata information of the new file in the new file header is in error File; and/or

a second restoring unit, configured to re-update the file header of the first file according to the new file header if an error occurs in the step of updating the file header of the first file.
The file merging device according to any one of claims 8 to 12, further comprising: a reading unit, configured to read a target value corresponding to the request key from the target file, wherein the reading unit comprises:

Obtain a module to obtain a file header and an index block of the target file;

The determining module determines, according to the file header, whether the request key is within a range of keys indicated by the file header;

a finding module, in a case of determining that the request key is within the range, searching for a leaf node corresponding to the request key in the index block based on a B+ tree structure of the index block;

The reading module reads the target value in a logical address in a data block in the target file according to a value corresponding to the key stored by the found leaf node.
An electronic device comprising:

a memory for storing executable instructions;

And a processor, configured to execute the electronic device to perform the file merging method according to any one of claims 1-7 according to the control of the executable instruction.