CN114328545B

CN114328545B - Data storage and query method, device and database system

Info

Publication number: CN114328545B
Application number: CN202210203048.7A
Authority: CN
Inventors: 朱炳鹏; 江进; 付治钧; 宋杰; 袁琳
Original assignee: Beijing Ant Cloud Financial Information Service Co ltd
Current assignee: Ant Yunchuang Digital Technology Beijing Co ltd
Priority date: 2022-03-03
Filing date: 2022-03-03
Publication date: 2022-07-08
Anticipated expiration: 2042-03-03
Also published as: CN114328545A; WO2023165272A1

Abstract

The embodiment of the specification provides a data storage and query method, a data storage and query device and a database system. The database system comprises a memory layer and a data storage medium. When data storage is carried out, data to be stored are stored in a memory table in a memory layer. And when the memory table is full of data, writing the stored data of the memory table into the data storage medium layer by layer. When the data storage medium is written layer by layer, data merging is carried out on the storage data to be written and all storage data in the data storage file of the current layered storage layer; and dividing the Value of the merged storage data into main data blocks and corresponding additional data blocks, and determining the storage positions of the main data blocks and the corresponding additional data blocks in the data storage medium. Each main data block and the corresponding additional data block are then stored to a corresponding storage location of the data storage medium. When data query is carried out, data query is carried out layer by layer from a memory layer of the database system.

Description

Data storage and query method, device and database system

Technical Field

The embodiments of the present specification generally relate to the field of data storage, and in particular, to a data storage method and apparatus, a data query method and apparatus, and a database system, which are applicable to large Value data storage.

Background

In a traditional data storage mechanism, a database system processes data to be stored into Key-value (KV) data, and stores the KV data in a memory layer or a data storage medium in the database system. In order to improve the data writing performance, the database system performs data organization on data to be stored in a hierarchical organization mode, for example, a structure similar to LSM Tree is adopted, then the data to be stored after the data organization is divided into different data storage files, and the different data storage files are stored in different hierarchical storage layers of a memory layer or a data storage medium. For example, the database system calculates a Hash Value of a Key of the data to be stored, and divides Value corresponding to the Key into different Hash Value ranges (Hash ranges) according to the calculated Hash Value, and then divides the data to be stored into different data storage files based on the Hash ranges.

A data storage and query scheme is disclosed in patent application publication No. CN 113407550A. In the data storage and query scheme, when data storage is performed, the data storage device stores data to be stored in a memory layer and determines a corresponding data index according to an index mechanism based on Hash. And after the data stored in the memory layer meets the preset conditions, writing the data in the memory layer into a data storage medium, generating a corresponding data index file according to an index mechanism based on the perfect hash, and loading the data index file into the memory layer. When data query is carried out, the data query device queries the memory data of the memory layer firstly, when no matching result exists in the memory data, index query is carried out in a data index file of the data storage medium stored in the memory layer, data reading is initiated to the data storage medium according to the queried data index, and therefore corresponding query data are obtained.

However, the database system may support multi-modal hash key data storage, for example, normal KV storage, wide table data storage like HBase, or graph storage by encoding together points and one-degree-edge data into one value.

Value itself is particularly large when the Value in the data to be stored is particularly large, for example when a Key of a wide table contains a very large number of columns of data (Column), or a graph node has a very large number of one degree edges (e.g., a celebrity may have tens or hundreds of millions of supporters, meaning that there are tens or hundreds of millions of one degree edges for this graph node), or for ordinary KV storage. In this case, if values are simply stored together, data that needs to be read by IO during querying may be very large, thereby causing a severe system overhead to the storage engine server, and causing a high querying delay, which severely restricts data querying performance.

Disclosure of Invention

In view of the foregoing, embodiments of the present specification provide data storage and query schemes. In the data storage and query scheme, a database system comprises a memory layer and a data storage medium with a layered storage structure. When data storage is carried out, data to be stored are stored in a memory table in a memory layer of a database system. And when the memory table is full of data, writing the stored data of the memory table into the data storage medium layer by layer. When the data storage medium is written layer by layer, data merging is carried out on the storage data to be written and all storage data in the data storage file of the current layered storage layer; dividing the Value of the merged storage data into main data blocks and corresponding additional data blocks, determining the storage positions of the main data blocks and the corresponding additional data blocks in the data storage medium, and then storing the main data blocks and the corresponding additional data blocks in the corresponding storage positions of the data storage medium. When data query is carried out, data query is carried out layer by layer from a memory layer of the database system. According to the data storage and query scheme, the large Value is divided into a plurality of data blocks to be stored in the data storage file, and the additional data blocks are read as required according to the query condition and the metadata information in the main data block during data query, so that the storage and query efficiency of the large Value data is improved, and the system overhead (IO overhead) of a database system is reduced.

According to an aspect of an embodiment of the present specification, there is provided a data storage method including: writing data to be stored into a memory table of a memory layer of a database system, wherein the data to be stored comprises Value in a Key-Value pair; responding to the memory table being full, starting from a top layered storage layer in storage layers of a data storage medium of the database system, executing data writing on the storage data of the memory table layer by layer according to the following mode until lower data writing is not triggered any more: reading all storage data in a data storage file of a current layered storage layer; merging the written upper storage data with the read storage data; dividing Value corresponding to each Key in the merged storage data into a main data block and a corresponding additional data block; determining a data storage file in which each main data block is located and the offset of each main data block in the data storage file based on the common hash value and the perfect hash value of the key corresponding to each main data block, wherein each data storage file corresponds to a hash value range of the common hash value based on the key; and writing data into each main data block and the corresponding additional data block according to the determined data storage file and the determined offset in the data storage file, wherein each written main data block comprises metadata, and the metadata comprises the number of the additional data blocks, the offset of each additional data block in the data storage file and the data length.

Optionally, in one example of the above aspect, one or more of the respective primary data blocks do not include a corresponding additional data block, and for a primary data block that does not include an additional data block, no metadata is included when writing to the data storage file.

Optionally, in an example of the above aspect, the lower data write is triggered in response to a file size of one or more data storage files in the current hierarchical storage tier after completion of the current data write reaching or exceeding a predetermined threshold, the written storage data including storage data in the data storage files in the current hierarchical storage tier that reached or exceeded the predetermined threshold.

Optionally, in one example of the above aspect, the additional data blocks in the data storage file are stored contiguously or separately.

Optionally, in an example of the above aspect, the merged storage data performs data slicing according to one of the following data slicing algorithms: a data segmentation algorithm based on data query frequency; a data segmentation algorithm based on data columns in the original data table; a data segmentation algorithm based on a data timestamp range; graph data slicing algorithms based on edge type, number of edges, or edge index.

Optionally, in an example of the above aspect, the number of tiered storage layers of the data storage medium includes a predetermined number of layers, and/or each tiered storage layer includes a predetermined number of data storage files, respectively.

Optionally, in one example of the above aspect, the number of data storage files included in each hierarchical storage tier increases from top to bottom tier to tier.

According to another aspect of embodiments of the present specification, there is provided a data query method including: responding to a received data query request initiated by a user, performing data query in a memory table of a memory layer of a database system, wherein the database system performs data storage according to the data storage method; in response to that the data query result for the memory layer does not completely satisfy the data query condition in the data query request, executing data query from a top layered storage layer of a storage layer of the database system in the following manner until the data query condition is completely satisfied or the query of a bottom layered storage layer is completed: in response to that the data query result for the memory layer or the upper layered storage layer does not completely satisfy the data query condition, positioning a data storage file in the current layered storage layer according to the common hash value of the key of the target data of the data query request and positioning the offset and the data length of the main data block in the data storage file according to the perfect hash value of the key; reading the main data block from the data storage file based on the offset and the data length of the positioned main data block; analyzing the read main data block to obtain a data query result; in response to that the data query result for the main data block does not completely satisfy the data query condition, acquiring the offset and the data length of an additional data block recorded in the main data block; reading all the additional data blocks from the data storage file based on the acquired offset and data length of the additional data blocks; analyzing each read additional data block to obtain a data query result; and providing all the obtained data query results to the user.

Alternatively, in one example of the above aspect, the data reading of the respective additional data blocks is performed in parallel.

According to another aspect of embodiments of the present specification, there is provided a data storage device including: the memory writing unit writes data to be stored into a memory table of a memory layer of the database system, wherein the data to be stored comprises Value in a Key-Value pair; and a data storage medium writing unit, which is used for responding to the full writing of the memory table and writing the storage data of the memory table into the data storage medium of the database system, wherein the data storage writing unit comprises: the data reading module is used for reading all storage data in the data storage file of the current layered storage layer; the data merging module merges the written upper-layer storage data and the read storage data; the data segmentation module is used for segmenting Value corresponding to each Key in the merged storage data into a main data block and a corresponding additional data block; the storage position determining module is used for determining a data storage file where each main data block is located and the offset of each main data block in the data storage file based on the common hash value and the perfect hash value of the key corresponding to each main data block respectively, and each data storage file corresponds to a hash value range based on the common hash value of the key; and the data writing module is used for writing data into each main data block and corresponding additional data blocks according to the determined data storage file and the offset in the data storage file, each written main data block comprises metadata, and the metadata comprises the number of the additional data blocks, the offset of each additional data block in the data storage file and the data length, wherein the data storage writing unit performs layer-by-layer data writing on the storage data of the memory table from a top-layer layered storage layer in a storage layer of a data storage medium of the database system until lower-layer data writing is not triggered any more.

Optionally, in one example of the above aspect, the data storage medium writing unit performs lower layer data writing on the storage data in the data storage files in the current hierarchical storage layer that meet or exceed a predetermined threshold in response to a file size of one or more data storage files in the current hierarchical storage layer meeting or exceeding the predetermined threshold.

Optionally, in an example of the foregoing aspect, the data slicing module performs data slicing on the merged storage data according to one of the following data slicing algorithms: a data segmentation algorithm based on data query frequency; a data segmentation algorithm based on data columns in the original data table; a data segmentation algorithm based on a data timestamp range; graph data segmentation algorithm based on edge type, number of edges or edge index.

According to another aspect of embodiments of the present specification, there is provided a data query apparatus including: the memory query unit is used for responding to a received data query request initiated by a user and performing data query in a memory table of a memory layer of the database system, and the database system performs data storage according to the data storage method; the data storage medium query unit is used for responding to the data query result aiming at the memory layer and not completely meeting the data query condition in the data query request, and performing data query on the data storage medium of the database system: and a query result providing unit that provides all the obtained data query results to the user, wherein the data storage medium querying unit includes: the main data block positioning module is used for positioning a data storage file in the current hierarchical storage layer according to a common hash value of a key of target data of the data query request and positioning the offset and the data length of a main data block in the data storage file according to a perfect hash value of the key in response to that a data query result aiming at the memory layer or the upper hierarchical storage layer does not completely meet the data query condition; the main data block reading module is used for reading the main data block from the data storage file based on the offset and the data length of the positioned main data block; the main data block query module analyzes the read main data block to obtain a data query result; the additional data block positioning module is used for responding that the data query result aiming at the main data block does not completely meet the data query condition, and acquiring the offset and the data length of the additional data block recorded in the main data block; the additional data block reading module is used for reading all the additional data blocks from the data storage file based on the acquired offset and data length of the additional data blocks; the additional data block query module is used for analyzing each read additional data block to obtain a data query result; the data storage medium query unit starts to execute data query from a top layered storage layer of a storage layer of the database system until the data query condition is completely met or the query of a bottom layered storage layer is completed.

Optionally, in an example of the above aspect, the additional data block reading module reads each additional data block from the data storage file in parallel.

According to another aspect of embodiments of the present specification, there is provided a database system including: a data storage device as described above; a data querying device as described above; a memory layer; and a data storage medium.

According to another aspect of embodiments of the present specification, there is provided a data storage device including: at least one processor, a memory coupled to the at least one processor, and a computer program stored in the memory, the at least one processor executing the computer program to implement the data storage method as described above.

According to another aspect of embodiments of the present specification, there is provided a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform a data storage method as described above.

According to another aspect of embodiments of the present specification, there is provided a computer program product comprising a computer program which is executed by a processor to implement the data storage method as described above.

According to another aspect of embodiments of the present specification, there is provided a data query apparatus including: at least one processor, a memory coupled to the at least one processor, and a computer program stored in the memory, the at least one processor executing the computer program to implement the data query method as described above.

According to another aspect of embodiments of the present specification, there is provided a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform a data query method as described above.

According to another aspect of embodiments of the present specification, there is provided a computer program product comprising a computer program which is executed by a processor to implement the data query method as described above.

Drawings

A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the drawings, similar components or features may have the same reference numerals.

FIG. 1 illustrates an example schematic of a database system according to embodiments of the present description.

FIG. 2 illustrates an example schematic of a data storage medium according to an embodiment of this specification.

FIG. 3 illustrates an example schematic of a data storage file according to an embodiment of this specification.

FIG. 4 illustrates an example flow diagram of a data storage method in accordance with an embodiment of the present description.

FIG. 5 illustrates an example flow diagram of a layer-by-layer data writing method for a data storage medium in accordance with an embodiment of the present description.

FIG. 6 illustrates an example schematic of raw data to be stored in accordance with an embodiment of the present description.

FIG. 7 illustrates an example flow diagram of a data merge process in accordance with an embodiment of the present description.

FIG. 8 illustrates an example flow diagram of a data query method in accordance with embodiments of the present specification.

FIG. 9 illustrates an example block diagram of a data storage device in accordance with an embodiment of this specification.

FIG. 10 illustrates an example block diagram of a data query device in accordance with an embodiment of this specification.

FIG. 11 illustrates an example schematic diagram of a data storage device implemented based on a computer system in accordance with an embodiment of the present description.

FIG. 12 illustrates an example schematic diagram of a computer-system-implemented data query device in accordance with an embodiment of the present description.

Detailed Description

The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thereby implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. Changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as needed. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with respect to some examples may also be combined in other examples.

As used herein, the term "include" and its variants mean open-ended terms, meaning "including but not limited to. The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first," "second," and the like may refer to different or the same object. Other definitions, whether explicit or implicit, may be included below. The definition of a term is consistent throughout the specification unless the context clearly dictates otherwise.

Before describing embodiments according to the present specification in conjunction with the accompanying drawings, a brief description will be given of several terms mentioned in the embodiments of the specification.

The normal hash operation is to perform a hash operation on data by using a normal hash function. Hash collisions (Hash Collision) may occur by performing a Hash operation according to a general Hash function. In other words, for two different data, performing a hash operation according to a common hash function may result in the same hash value.

Perfect Hash operation is a Hash operation without Hash collision, which maps N input integer data sets to M integer spaces based on a Perfect Hash Function (Perfect Hash Function), wherein,

. In the perfect hash operation, for a given data set, the hash values obtained by processing any of keys 1 and keys 2 by the perfect hash function H are different, that is,

。

in this specification, raw data is processed as Key-Value pair (KV) data, and Value corresponding to each Key is stored in a database system. Examples of raw data may include, but are not limited to, normal KV data, wide-list data, graph data, and the like. For ordinary KV data, each Key corresponds to a Value. For the KV data processed by the wide table data and the graph data, there may be multiple data columns (columns) for one Key, or multiple first-degree edges for one graph node, in other words, one Key may correspond to multiple values. In this specification, the data to be stored includes Value in KV data, which refers to data to be stored into a database system.

A database system, a data storage method, a data storage apparatus, a data query method, and a data query apparatus according to embodiments of the present specification will be described below with reference to the accompanying drawings.

FIG. 1 illustrates an example schematic diagram of a database system 100 in accordance with embodiments of the present description. The database system 100 may also be referred to as a server side of a database application for providing data storage services and data query services. The database system 100 may be implemented, for example, using devices having computing capabilities.

As shown in FIG. 1, database system 100 includes a memory layer 110 and a data storage medium 120. The memory layer 110 may include a memory pool composed of at least one memory (e.g., a memory of a computer device). The memory layer 110 may also be referred to as internal memory and main memory. The memory layer 110 may be used, for example, to temporarily store operation data in a CPU of the computing device and data exchanged with an external data storage medium such as a Disk (Disk). The data storage medium 120 may also be referred to as an external memory. When a computing device reads data in the data storage medium 120, the data in the data storage medium 120 needs to be read into a memory by initiating a data read operation, such as an IO operation, to the data storage medium. The data storage medium 120 may be, for example, various nonvolatile storage media such as a magnetic disk device, a memory card, and the like. Magnetic disks refer to memories that store data using magnetic recording techniques. Examples of magnetic disks may include various forms of (Soft Disk) floppy disks and Hard disks (Hard disks), for example.

Database system 100 also includes data storage 130. The data storage device 130 is configured to store data to be stored to the memory layer 110 or the data storage medium 120 of the database system.

The database system 100 also includes a data querying device 140. The data query device 140 is configured to perform a data query in the memory layer 110 or the data storage medium 120 of the database system in response to a data query request initiated by a user, and provide the data query result to the user.

In some embodiments of the present description, a data storage medium may be layered into at least two layered storage tiers, each of the at least two layered storage tiers having a hierarchical relationship therebetween, thereby forming a layered storage tier structure. FIG. 2 shows an example schematic of a data storage medium 120 according to an embodiment of the present description.

As shown in FIG. 2, data storage medium 120 is layered into 3 layered storage tiers, namely, layered storage tier L0, layered storage tier L1, and layered storage tier L2. Layered storage L0 is the top storage layer of the layered storage tier structure, layered storage L1 is the lower layered storage tier of layered storage L0, and layered storage L2 is the lower layered storage tier of layered storage L1. Note that the number of layered layers of the layered storage layer structure may be a predetermined number of layers set in advance.

Each hierarchical storage tier may include a predetermined number of data storage files (e.g., Bin files). In some embodiments, the number of data storage files of each hierarchical storage layer may be the same. In some embodiments, the number of data storage files of each hierarchical storage layer may be different. For example, in some examples, when the data storage medium is layered, the number of data storage files set for each layered storage layer may increase sequentially layer by layer from the top layered storage layer. As shown in FIG. 2, the number of Bin files provided for hierarchical storage L0, hierarchical storage layer L1, and hierarchical storage layer L2 increases sequentially from top to bottom from top storage layer L0, i.e., hierarchical storage layer L0 contains 1 Bin file, hierarchical storage layer L0 contains 2 Bin files, and hierarchical storage layer L2 contains 4 Bin files. It is noted that fig. 2 shows only an example of a hierarchical storage structure of a data storage medium. In other embodiments, a tiered storage structure containing other numbers of data storage files may be employed. Preferably, each hierarchical storage layer may contain a number of data storage files that is a power of 2.

And the data storage file in the layered storage layer is used for storing Value of the data to be stored. When writing the Value of the data to be stored into the data storage file, it is necessary to segment the Value corresponding to each Key in the data to be stored into the main data and the corresponding appended block data, and determine the storage location (i.e., the offset from the head address of the data storage file) of the corresponding Value in the data storage file based on the Key. And then writing the cut main data and the corresponding additional block data into the corresponding data storage file.

In the example of FIG. 3, the data storage file includes a plurality of primary data blocks, e.g., primary data block 1, primary data block 2, through primary data block M. Each main data block corresponds to partial or all Value of one Key. Optionally, each main data block may also include one or more additional data blocks. For example, the main data block 1 may include additional data blocks 1, 2 through N. The additional data blocks 1 to N are used to store the remaining values of the keys corresponding to their master data blocks. In other words, for Value of a Key in the data to be stored, it can be split into a main data block and one or more additional data blocks. In the case that Value of one Key is not large, all its values can be sliced into main data blocks without additional data blocks.

It is to be noted that, in the data storage file of the present specification, as shown in fig. 3, the master data blocks corresponding to the respective keys are stored consecutively, and the storage locations of the master data blocks corresponding to the respective keys in the data storage file may be determined based on the perfect hash values of the keys. Further, the corresponding additional data blocks of the respective main data blocks may be stored continuously, as shown in fig. 3, or may be stored separately.

When writing a data storage file, a main data block needs to contain corresponding metadata (if there are additional data blocks) in addition to the sub data blocks contained in the record. The metadata comprises the number of the additional data blocks, the offset of each additional data block in the data storage file and the data length. As shown in fig. 3, for the main data block 1, it includes N additional data blocks, so that the number of the additional data blocks is N. In addition, the metadata also records the offsets of the additional data blocks 1 to N in the data storage file and the corresponding data lengths. The offset of the additional data block recorded in the metadata is used to indicate an offset of a head address of a storage location of the additional data block in the data storage file with respect to a head address of the data storage file. Each additional data block is recorded to a corresponding location of the data storage file, and the recorded additional data block may contain 1 or more sub data blocks. For example, the additional data block 1 contains sub data blocks X +1 through X + Y.

FIG. 4 illustrates an example flow diagram of a data storage method in accordance with an embodiment of the present description. The data storage method illustrated in fig. 4 may be performed by the data storage device 130 in the database system. In some embodiments, the data storage 130 may be implemented as a thread (hereinafter referred to as a data storage thread) of a database system (database application server) for implementing data storage.

As shown in fig. 4, when data to be stored needs to be written into the database system, the data to be stored is written into a memory table (MemTable) of a memory layer of the database system. For example, at 410, the data storage thread monitors whether data to be stored is received. The data to be stored may be, for example, data to be stored entered by a user via a client device of a database application. And if the data to be stored is not received, the data storage thread continuously monitors. If the data to be stored is received, the data storage thread writes the data to be stored to the memory table MemTable at 420.

FIG. 6 illustrates an example schematic of raw data to be stored in accordance with an embodiment of the present description. As shown in fig. 6, the original data to be stored is key-value pair data. Each piece of Key-Value pair data may include a Key-Value pair (i.e., Key-Value) consisting of 1 Key (Key) and at least one Value (Value). In the example of fig. 3, the Key of each Key-value pair data is a data ID, e.g., a user's identification number, registration number, system number, etc. Value _1 represents the user's name, Value _2 represents the user's age, and Value _5 represents the user's occupation, etc. In this specification, the term "data to be stored" refers to Value in the original data.

At 430, it is determined whether the memory table in the memory layer is full. If the memory table is judged to be full, at 440, the storage data of the memory table is written into the storage layer of the data storage medium in a layer-by-layer writing mode from the top layered storage layer in the storage layers of the data storage medium of the database system. And if the memory table is judged to be not full, the current data storage process is finished. Optionally, a return to 410 may be made to continue monitoring for receipt of data to be stored.

Fig. 5 illustrates an example flow diagram of a layer-by-layer data writing method 500 for a data storage medium in accordance with an embodiment of the present description. FIG. 5 illustrates an example of a data writing process for a current hierarchical storage layer of a layer-by-layer data writing process.

As shown in FIG. 5, upon triggering a data write to the current hierarchical storage tier, at 510, all of the storage data of the current hierarchical storage tier, i.e., the storage data in all of the data storage files (Bin files) in the current hierarchical storage tier, is read. The data write may be triggered in response to a memory table being full or a file size of one or more data storage files in an upper hierarchical storage layer reaching or exceeding a predetermined threshold. The written storage data comprises storage data of a memory table or storage data in a data storage file reaching or exceeding a preset threshold value in an upper layered storage layer.

At 520, the written storage data is merged with the read storage data in the current hierarchical storage layer. FIG. 7 illustrates an example flow diagram of a data merge process in accordance with an embodiment of the present description. As shown in fig. 7, when data merging is performed, for the same Key, if the Value of Value is updated, the old data, for example, the data corresponding to ID1, is replaced with the new data. Alternatively, if the first data has a Key that does not exist in the old data, the data corresponding to the Key, for example, the data corresponding to ID5, is incremented. Then, based on the updated stored data in the memory layer, a corresponding memory data index is generated using a first hash-based indexing mechanism.

At 530, the merged stored data is sliced into individual main data chunks and corresponding additional data chunks. For example, the storage data corresponding to each key may be divided into a main data block and a corresponding additional data block.

In some embodiments, data slicing may be performed on the merged stored data in accordance with a data slicing algorithm. Examples of data slicing algorithms may include, but are not limited to: a data segmentation algorithm based on data use frequency; a data segmentation algorithm based on data columns in the original data table; a data segmentation algorithm based on a data timestamp range; graph data segmentation algorithm based on edge type, number of edges or edge index.

The data slicing algorithm based on the data use frequency refers to data slicing according to whether Value is the common Value. Whether Value is a commonly used Value may be determined based on various applicable methods in the art, for example, whether the hit frequency of data query (or the data use frequency) exceeds a predetermined Value. According to the data segmentation algorithm, the common Value can be stored in the main data block, and the less common Value can be stored in the additional data block, so that most data queries can avoid reading the additional data block.

A data slicing algorithm based on data columns in an original data table refers to data slicing performed based on data columns in an original data table (e.g., a wide table). For example, for a wide table with a large Value, data slicing may be performed in accordance with Column of the wide table. The data segmentation algorithm based on the data timestamp range refers to data segmentation based on the timestamp of the data to be stored. According to this data slicing method, each data block contains stored data within a range of time stamps. Graph data slicing algorithms based on edge type, number of edges, or edge index are data slicing algorithms applicable to graph data. In such a data slicing algorithm, the graph data may be data sliced according to the type of edge, the number of edges, or the edge index.

At 540, the data storage file in which each main data block is located and the offset of each main data block in the data storage file are determined based on the common hash value and the perfect hash value of the key corresponding to each main data block, respectively. Each data storage file corresponds to a range of hash values.

In this description, each hierarchical storage layer is arranged to correspond to a first range of hash values, which may be, for example, a range of integers, such as [0, K-1], where K is a power of 2. And the hash value range corresponding to each data storage file in each layered storage layer is the hash value range of the layered storage layer divided by the number of the data storage files of the layered storage layer. For example, for the data storage medium in FIG. 2, the hash values for the data storage files in hierarchical storage layer L0 range from [0, K-1], the hash values for the 2 data storage files in hierarchical storage layer L1 range from [0, K/2-1] and [ K/2, K-1], respectively, and the hash values for the 4 data storage files in hierarchical storage layer L1 range from [0, K/4-1], [ K/4, K/2-1], [ K/2, 3K/4-1] and [3K/4, K-1], respectively.

When determining the storage position, firstly, determining the data storage file where each main data block is located based on the common hash value of the key corresponding to each main data block. For example, the common hash value of the key corresponding to each main data block is calculated, and then the range of the hash value of the common hash value is determined based on the common hash value of the key corresponding to the main data block, so as to determine the located data storage file. For each main data block, if the hash value calculated based on the ordinary hash operation exceeds the set first hash value range (namely, the hash value range corresponding to the hierarchical storage layer), performing modulo operation by taking the number of the hash values in the first hash value range as a modulus, and obtaining a remainder which is the ordinary hash value. After the data storage file where each main data block is located is determined, for each data storage file, a perfect hash function corresponding to the data storage file is determined based on the keys of all the main data blocks contained in the data storage file, and the determined perfect hash function is used for performing perfect hash operation on the Key of each main data block in the data storage file, so that the offset of each main data block in the data storage file is determined.

After the storage location of each main data block is determined as described above, at 550, data writing is performed on each main data block and the corresponding additional data block according to the determined data storage file and the offset in the data storage file. When the written main data block further comprises one or more additional data blocks, the written main data block further comprises metadata, and the metadata comprises the number of the additional data blocks, the offset and the data length of each additional data block in the data storage file, so that the storage position of each additional data block is located by using the metadata, and the writing of the additional data blocks is performed.

Alternatively, after the storage data of the memory table is written into the storage layer of the data storage medium, the corresponding storage data may be deleted from the memory table, thereby freeing the corresponding storage space in the memory layer. Further, alternatively, after the storage data of the hierarchical storage layer is written to the lower hierarchical storage layer, the corresponding storage data may also be deleted from the hierarchical storage layer.

Returning to FIG. 5, after the data writing of the main data block and its corresponding additional data block is completed as above, at 560, it is determined whether to trigger a lower layer data write. For example, a determination is made as to whether a file size of one or more data storage files in a current hierarchical storage tier meets or exceeds a predetermined threshold. If the file size of one or more data storage files in the current hierarchical storage tier reaches or exceeds a predetermined threshold, then lower tier data writes are triggered, i.e., performed on the stored data of the one or more data storage files in the current hierarchical storage tier whose file size reaches or exceeds the predetermined threshold. If the file sizes of all the data storage files in the current hierarchical storage layer do not reach the preset threshold value, the data storage process is ended.

The data storage process according to the embodiment of the present specification is described below by taking an example as an example. In this example, the database system includes memory and hard disks, which are classified into L0-L3 hierarchies. The memory, L0 tier and L1 tier store data. The stored data is user data in a user information system, wherein key is user id, and value is various information of the user. For user id1, the name of user id1 is recorded in memory, the address of user id1 is recorded in layer L0, and the telephone number of user id1 is recorded in layer L1.

Assume that an identification number of user id1 needs to be inserted. The data in the memory table is stored by using a linked list, so that after the identity number of the user id1 is inserted into the memory table, two items of information exist in the linked list of the user id1, one item of information is the newly inserted identity number, and the other item of information is the originally stored name.

If the memory table is full after the identification number is inserted, the storage data of the memory table needs to be merged into the Bin file of the L0 layer. In this case, it is necessary to read old data (e.g., sub-data such as a telephone number) stored in the bin file of the L0 layer from the L0 layer, merge the read telephone number with stored data of the memory table (e.g., sub-data such as an id card, a name, etc. of the user id 1), and write the merged data into a new bin file of the L0 layer. The sub-data such as the telephone number, the identity card, the name and the like can be put into the main data block of the Bin file together, or can be opened in the main data block and the additional data block of the Bin file. For example, if the name is the most frequently accessed child data and the identification number and phone number are not commonly used child data, then the name may be placed in the main data block and the identification number, phone number, respectively, may be placed in 2 additional blocks. Further, in the case where the sub data is too much, if it is put in one main data block, the main data block is made too large, whereby the sub data can be cut into a main block and an additional block of appropriate sizes. Here, the data slicing manner of the main data block and the additional data block may be a custom manner.

After the data writing from the memory table to the L0 layer is completed, if the size of the Bin file of the L0 layer reaches a predetermined threshold, the Bin file of the L0 layer needs to be merged with the Bin file of the L1 layer, and the merged Bin file is written into the L1 layer, so that a new Bin file of the L1 layer is obtained. The data writing process of the L1 layer is similar to that of the L0 layer, and is not repeated here.

The data storage process according to the embodiment of the present specification is described above with reference to the drawings. After the data is stored in the database system according to the data storage method, data query can be performed in response to a data query request initiated by a user.

FIG. 8 illustrates an example flow diagram of a data query method 800 in accordance with embodiments of the present description. The data query method in fig. 8 may be performed by a data query apparatus in a database system. The data query apparatus may be implemented as a thread (hereinafter referred to as a data query thread) of a database system (database application server) for implementing data query.

As shown in fig. 8, at 810, in response to receiving a user-initiated data query request, a data query is performed in a memory table of a memory layer of a database system. In one example, the data query request may include a Key of the target data to be queried and a data query condition.

After the data query for the memory table is completed to obtain the data query result for the memory table, at 820, it is determined whether the data query for the memory table completely satisfies the data query condition in the data query request. Namely, whether the data query conditions to be queried in the data query request complete the corresponding data query is judged. For example, assuming that only the id card and name of the user id1 are stored in the memory table of the memory layer, and the data query condition indicates that the id card, name and registration date of the user id1 need to be queried, the data query condition in the data query request is not completely satisfied by the data query against the memory table because there is no registration date in the memory table.

If the data query for the memory table completely satisfies the data query condition in the data query request, the data query result obtained by the data query for the memory table is provided to the user at 811.

And if the data query aiming at the memory table does not completely meet the data query condition in the data query request, executing the data query layer by layer from the top layered storage layer of the database system until the data query condition is completely met or the query of the bottom layered storage layer is completed. Specifically, the operations of blocks 803 to 810 are performed cyclically.

At 803, for the current hierarchical storage layer, the data storage file in the current hierarchical storage layer is located according to the common hash value of the key of the target data to be queried of the data query request, and the offset and the data length of the main data block in the data storage file are located according to the perfect hash value of the key of the target data to be queried.

For example, a common hash value is obtained by performing a common hash operation using a key on the target data, and then it is determined which data storage file of the current hierarchical storage layer the common hash value falls within, thereby locating the data storage file in which the target data is located. After the located data storage file is located, performing perfect hash operation on the key of the target data to obtain a perfect hash value based on a perfect hash function corresponding to the data storage file, so as to locate the offset of the main data block in the data storage file, and determining the data length of the main data block based on the offset of the next main data block in the data storage file. It is noted that, when data storage is performed, since keys of respective main data blocks are known and stored continuously, offsets of respective main data blocks can be learned.

After locating the offset and data length of the main data block in the data storage file, at 804, the main data block is read from the data storage file based on the offset and data length of the located main data block. For example, the data query thread may set a callback function of the main data block, send an asynchronous IO read request to the operating system, and initiate an IO data read operation to the data storage medium by the operating system, thereby reading the main data block from the data storage file of the corresponding hierarchical storage layer of the data storage medium.

After reading the main data block from the data storage medium, at 805, the read main data block is parsed, and a data query is performed in the parsed main data block to obtain a data query result.

After the data query result for the main data block is obtained, at 806, it is determined whether the data query condition in the data query request is completely satisfied after the data query for the main data block is completed. If the data query condition in the data query request is completely satisfied after the data query of the main data block is completed, a data query result obtained by the data query of the main data block and the memory table is provided to the user at 811.

If the data query condition in the data query request is not completely satisfied after the data query of the main data block is completed, the offset and the data length of the additional data block recorded in the main data block are obtained in 807. That is, the offset and data length of each additional data block are obtained from the parsed main data block.

At 808, all additional data blocks are read from the local data storage file based on the obtained offset and data length of each additional data block. For example, after determining that the additional data block query needs to be executed, the data query thread may set a callback function of the additional data block, send an IO read request to the operating system for each additional data block according to the offset and the data length of the additional data block recorded in the main data block, and initiate an IO data read operation to the data storage medium by the operating system to read the additional data block from the data storage medium. And after the additional data block of the current hierarchical storage layer is returned by the operating system, the data query thread calls a callback function of the additional data block to execute processing. For example, the callback function of the additional data block may count the number of additional data blocks returned and start decoding all additional data blocks when they are returned. It is noted that in one example, the data read of the various additional data blocks may be performed in parallel.

At 809, each additional data block read is parsed to obtain data query results. And analyzing the read additional data block, and performing data query in the analyzed additional data block to obtain a data query result.

After the data query result for the additional data block is obtained, it is determined whether the data query condition in the data query request is completely satisfied after the data query for the additional data block is completed at 810. If the data query condition in the data query request is completely satisfied after the data query of the additional data block is completed, a data query result obtained by the data query of the memory table, the main data block and the additional data block is provided to the user at 811. Here, the data query results for the main data block and the additional data block include data query results for the main data block and the additional data block of all hierarchical storage layers up to (including) the current hierarchical storage layer.

And returning to 803, and executing 803 to 810 data query processing for the next hierarchical storage layer if the data query condition in the data query request is not completely met after the data query of the additional data block is completed.

By using the data storage and data query scheme, in each hierarchical storage layer, Value of the same Key can be separately stored in the main data block and one or more additional data blocks, and when data query is performed, query is performed in the main data block at first, and data query for the additional data blocks is performed only when target data does not exist in the main data block, that is, on-demand query is performed for the additional data blocks, so that the data query amount during data query can be reduced, and thus the data query efficiency is improved.

FIG. 9 illustrates an example block diagram of a data storage device 900 in accordance with embodiments of this specification. As shown in fig. 9, the data storage device 900 may include a memory writing unit 910 and a data storage medium writing unit 920. The data storage medium writing unit 920 includes a data reading module 921, a data merging module 923, a data splitting module 925, a storage location determining module 927, and a data writing module 929.

The memory writing unit 910 is configured to write data to be stored into a memory table of a memory layer of the database system, where the data to be stored includes a Key-Value pair. The operation of the memory write unit 910 may refer to the operation described above with reference to 420 of fig. 4.

The data storage medium writing unit 920 is configured to write the storage data of the memory table to the data storage medium of the database system in response to the memory table being full. Specifically, the data reading module 921, the data merging module 923, the data splitting module 925, the storage location determining module 927, and the data writing module 929 in the data storage writing unit 920 perform layer-by-layer data writing on the storage data of the memory table from a top layered storage layer in the storage layers of the data storage medium of the database system until lower layer data writing is no longer triggered.

The data reading module 921 is configured to read all storage data in the data storage file of the current layered storage tier. The data merge module 923 is configured to merge the written upper storage data with the read storage data.

The data splitting module 925 is configured to split the Value corresponding to each Key in the merged storage data into a main data block and a corresponding additional data block.

The storage location determining module 927 is configured to determine, based on the normal hash value and the perfect hash value of the key corresponding to each main data block, a data storage file in which each main data block is located and an offset of each main data block in the data storage file, where each data storage file corresponds to one hash value range.

The data writing module 929 is configured to perform data writing on each main data block and the corresponding additional data block according to the determined data storage file and the determined offset in the data storage file, where each written main data block includes metadata, and the metadata includes the number of the additional data blocks, the offset of each additional data block in the data storage file, and a data length.

In some embodiments, the data storage medium writing unit 920 may perform lower layer data writing on the storage data in the data storage files in the current hierarchical storage layer that meet or exceed a predetermined threshold in response to the file size of one or more data storage files in the current hierarchical storage layer meeting or exceeding the predetermined threshold.

In some embodiments, the data slicing module 925 may be configured to perform data slicing on the merged stored data according to one of the following data slicing algorithms: a data segmentation algorithm based on data use frequency; a data segmentation algorithm based on data columns in the original data table; a data segmentation algorithm based on a data timestamp range; graph data segmentation algorithm based on edge type, number of edges or edge index.

Fig. 10 illustrates an example block diagram of a data query device 1000 in accordance with an embodiment of this specification. As shown in fig. 10, the data query apparatus 1000 includes a memory query unit 1010, a data storage medium query unit 1020, and a query result providing unit 1030. The data storage medium querying unit 1020 includes a master data block locating module 1021, a master data block reading module 1022, a master data block querying module 1023, an additional data block locating module 1024, an additional data block reading module 1025, and an additional data block querying module 1026.

The memory query unit 1010 is configured to perform a data query in a memory table of a memory layer of the database system in response to receiving a data query request initiated by a user. The operation of the memory lookup unit 1010 may refer to the operation described above with reference to 801 of fig. 8.

The data storage medium querying unit 1020 is configured to perform a data query on the data storage medium of the database system in response that the data query result for the memory layer does not completely satisfy the data query condition in the data query request. The data storage medium querying unit 1020 executes a data query starting from a top hierarchical storage tier of storage tiers of the database system until a data query condition is completely satisfied or a bottom hierarchical storage tier query is completed.

In particular, upon a data query for each hierarchical storage layer, the master data block location module 1021, the master data block reading module 1022, the master data block querying module 1023, the additional data block location module 1024, the additional data block reading module 1025, and the additional data block querying module 1026 perform the following operations.

The main data block positioning module 1021 is configured to, in response to that a data query result for the memory layer or the upper hierarchical storage layer does not completely satisfy a data query condition, position a data storage file in a current hierarchical storage layer according to a common hash value of a key of target data of the data query request, and position an offset and a data length of a main data block in the data storage file according to a perfect hash value of the key of the target data. The main data block reading module 1022 is configured to read a main data block from a data storage file in which it is located based on the offset and data length of the located main data block. The main data block query module 1023 is configured to parse the read main data block to obtain a data query result.

The additional data block positioning module 1024 is configured to obtain an offset and a data length of an additional data block recorded in the main data block in response to that the data query result for the main data block does not fully satisfy the data query condition. The additional data block reading module 1025 is configured to read all the additional data blocks from the data storage file based on the offset and data length of the acquired additional data blocks. Additional data block query module 1026 is configured to parse the read individual additional data blocks to obtain data query results.

The query result providing unit 1030 is configured to provide all the obtained data query results to the user.

As described above with reference to fig. 1 to 10, a data storage method, a data storage apparatus, a data query method, a data query apparatus, and a database system according to an embodiment of the present specification are described. The above query storage device and data query device may be implemented by hardware, or may be implemented by software, or a combination of hardware and software.

FIG. 11 illustrates a schematic diagram of a data storage device 1100 implemented based on a computer system in accordance with an embodiment of the present description. As shown in fig. 11, the data storage device 1100 may include at least one processor 1110, a storage (e.g., non-volatile storage) 1120, a memory 1130, and a communication interface 1140, and the at least one processor 1110, the storage 1120, the memory 1130, and the communication interface 1140 are connected together via a bus 1160. The at least one processor 1110 executes at least one computer-readable instruction (i.e., the elements described above as being implemented in software) stored or encoded in memory.

In one embodiment, computer-executable instructions are stored in the memory that, when executed, cause the at least one processor 1110 to: writing data to be stored into a memory table of a memory layer of a database system, wherein the data to be stored comprises Value in a Key-Value pair; responding to the memory table being full, starting from a top layered storage layer in storage layers of a data storage medium of the database system, executing data writing on the storage data of the memory table layer by layer according to the following mode until lower data writing is not triggered any more: reading all storage data in a data storage file of a current layered storage layer; merging the written upper storage data with the read storage data; dividing Value corresponding to each Key in the merged storage data into a main data block and a corresponding additional data block; respectively determining a data storage file in which each main data block is located and an offset of each main data block in the data storage file based on the common hash value and the perfect hash value of the key corresponding to each main data block, wherein each data storage file corresponds to a hash value range of the common hash value based on the key; and writing data into each main data block and the corresponding additional data block according to the determined data storage file and the determined offset in the data storage file, wherein each written main data block comprises metadata, and the metadata comprises the number of the additional data blocks, the offset of each additional data block in the data storage file and the data length.

It should be appreciated that the computer-executable instructions stored in the memory, when executed, cause the at least one processor 1110 to perform the various operations and functions described above in connection with fig. 1-7 and 9 in the various embodiments of the present description.

FIG. 12 shows a schematic diagram of a data query device 1200 implemented based on a computer system according to an embodiment of the present description. As shown in fig. 12, the data query device 1200 may include at least one processor 1210, a storage (e.g., non-volatile storage) 1220, a memory 1230, and a communication interface 1240, and the at least one processor 1210, the storage 1220, the memory 1230, and the communication interface 1240 are coupled together via a bus 1260. The at least one processor 1210 executes at least one computer-readable instruction (i.e., the elements described above as being implemented in software) stored or encoded in memory.

In one embodiment, computer-executable instructions are stored in the memory that, when executed, cause the at least one processor 1210 to: responding to a received data query request initiated by a user, performing data query in a memory table of a memory layer of a database system, and performing data storage by the database system according to the data storage method; in response to that the data query result aiming at the memory layer does not completely meet the data query condition in the data query request, executing data query from a top layered storage layer of a storage layer of the database system in the following way until the data query condition is completely met or the query of a bottom layered storage layer is completed: in response to that the data query result aiming at the memory layer or the upper layered storage layer does not completely meet the data query condition, positioning the data storage file in the current layered storage layer according to the common hash value of the key of the target data of the data query request and positioning the offset and the data length of the main data block in the data storage file according to the perfect hash value of the key; reading the main data block from the data storage file based on the offset and the data length of the positioned main data block; analyzing the read main data block to obtain a data query result; in response to that the data query result for the main data block does not completely satisfy the data query condition, acquiring the offset and the data length of the additional data block recorded in the main data block; reading all the additional data blocks from the data storage file based on the acquired offset and data length of the additional data blocks; analyzing each read additional data block to obtain a data query result; and providing all the obtained data query results to the user.

It should be understood that the computer-executable instructions stored in the memory, when executed, cause the at least one processor 1210 to perform the various operations and functions described above in connection with fig. 8 and 10 in the various embodiments of the present description.

According to one embodiment, a program product, such as a machine-readable medium (e.g., a non-transitory machine-readable medium), is provided. A machine-readable medium may have instructions (i.e., elements described above as being implemented in software) that, when executed by a machine, cause the machine to perform various operations and functions described above in connection with fig. 1-10 in the various embodiments of the present specification. Specifically, a system or apparatus may be provided which is provided with a readable storage medium on which software program code implementing the functions of any of the above embodiments is stored, and causes a computer or processor of the system or apparatus to read out and execute instructions stored in the readable storage medium.

In this case, the program code itself read from the readable medium can realize the functions of any of the above-described embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code form part of the present invention.

Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-R, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs), magnetic tapes, nonvolatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or from the cloud via a communications network.

According to one embodiment, a computer program product is provided that includes a computer program that, when executed by a processor, causes the processor to perform the various operations and functions described above in connection with fig. 1-10 in the various embodiments of the present specification.

It will be understood by those skilled in the art that various changes and modifications may be made in the above-disclosed embodiments without departing from the spirit of the invention. Accordingly, the scope of the invention should be limited only by the attached claims.

It should be noted that not all steps and units in the above flows and system structure diagrams are necessary, and some steps or units may be omitted according to actual needs. The execution order of the steps is not fixed, and can be determined as required. The apparatus structures described in the above embodiments may be physical structures or logical structures, that is, some units may be implemented by the same physical entity, or some units may be implemented by a plurality of physical entities, or some units may be implemented by some components in a plurality of independent devices.

In the above embodiments, the hardware units or modules may be implemented mechanically or electrically. For example, a hardware unit, module or processor may comprise permanently dedicated circuitry or logic (such as a dedicated processor, FPGA or ASIC) to perform the corresponding operations. The hardware units or processors may also include programmable logic or circuitry (e.g., a general purpose processor or other programmable processor) that may be temporarily configured by software to perform the corresponding operations. The specific implementation (mechanical, or dedicated permanent, or temporarily set) may be determined based on cost and time considerations.

The detailed description set forth above in connection with the appended drawings describes exemplary embodiments but does not represent all embodiments that may be practiced or fall within the scope of the claims. The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration," and does not mean "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described technology. However, the techniques may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of data storage, comprising:

writing data to be stored into a memory table of a memory layer of a database system, wherein the data to be stored comprises Value in a Key-Value pair;

responding to the memory table being full, starting from a top layered storage layer in storage layers of a data storage medium of the database system, executing data writing on the storage data of the memory table layer by layer according to the following mode until the lower layer data writing is not triggered any more:

reading all storage data in a data storage file of a current layered storage layer;

merging the written upper storage data with the read storage data;

dividing Value corresponding to each Key in the merged storage data into a main data block and a corresponding additional data block;

respectively determining a data storage file in which each main data block is located and an offset of each main data block in the data storage file based on the common hash value and the perfect hash value of the key corresponding to each main data block, wherein each data storage file corresponds to a hash value range; and

and writing data into each main data block and the corresponding additional data block according to the determined data storage file and the determined offset in the data storage file, wherein each written main data block comprises metadata, and the metadata comprises the number of the additional data blocks, the offset of each additional data block in the data storage file and the data length.

2. A data storage method according to claim 1, wherein one or more of said respective main data blocks do not include a corresponding additional data block, and for a main data block that does not include an additional data block, no metadata is included when writing to the data storage file.

3. The data storage method of claim 1, wherein the lower data write is triggered in response to a file size of one or more data storage files in the current hierarchical storage tier after completion of the current data write reaching or exceeding a predetermined threshold, the written storage data comprising storage data in the data storage files in the current hierarchical storage tier that reached or exceeded the predetermined threshold.

4. The data storage method of claim 1, wherein the additional data blocks in the data storage file are stored contiguously or separately.

5. The data storage method of claim 1, wherein the merged storage data performs data slicing according to one of the following data slicing algorithms:

a data segmentation algorithm based on data use frequency;

a data segmentation algorithm based on data columns in the original data table;

a data segmentation algorithm based on a data timestamp range;

graph data segmentation algorithm based on edge type, number of edges or edge index.

6. The data storage method of claim 1, wherein the number of tiered storage layers of the data storage medium comprises a predetermined number of layers, and/or each tiered storage layer comprises a predetermined number of data storage files, respectively.

7. The data storage method of claim 6, wherein the number of data storage files included in each hierarchical storage tier increases from top to bottom tier by tier.

8. A method of data query, comprising:

in response to receiving a data query request initiated by a user, performing data query in a memory table of a memory layer of a database system, wherein the database system performs data storage according to the method of any one of claims 1 to 7;

in response to that the data query result for the memory layer does not completely satisfy the data query condition in the data query request, executing data query from a top layered storage layer of a storage layer of the database system in the following manner until the data query condition is completely satisfied or the query of a bottom layered storage layer is completed:

in response to that the data query result aiming at the memory layer or the upper layered storage layer does not completely meet the data query condition, positioning a data storage file in the current layered storage layer according to the common hash value of the key of the target data of the data query request and positioning the offset and the data length of the main data block in the data storage file according to the perfect hash value of the key;

reading the main data block from the data storage file based on the offset and the data length of the positioned main data block;

analyzing the read main data block to obtain a data query result;

in response to that the data query result for the main data block does not completely satisfy the data query condition, acquiring the offset and the data length of the additional data block recorded in the main data block;

reading all the additional data blocks from the data storage file based on the acquired offset and data length of the additional data blocks;

analyzing each read additional data block to obtain a data query result; and

and providing all the obtained data query results to the user.

9. The data query method of claim 8, wherein the data reading of the respective additional data blocks is performed in parallel.

10. A data storage device, comprising:

the memory writing unit writes data to be stored into a memory table of a memory layer of the database system, wherein the data to be stored comprises Value in a Key-Value pair; and

a data storage medium writing unit which writes the storage data of the memory table into the data storage medium of the database system in response to the memory table being full,

wherein the data storage write unit comprises:

the data reading module is used for reading all storage data in the data storage file of the current layered storage layer;

the data merging module merges the written upper-layer storage data and the read storage data;

the data segmentation module is used for segmenting Value corresponding to each Key in the merged storage data into a main data block and a corresponding additional data block;

the storage position determining module is used for determining a data storage file where each main data block is located and the offset of each main data block in the data storage file, wherein each data storage file corresponds to a hash value range, respectively based on the common hash value and the perfect hash value of the key corresponding to each main data block; and

the data writing module writes data into each main data block and the corresponding additional data block according to the determined data storage file and the offset in the data storage file, each written main data block comprises metadata, and the metadata comprises the number of the additional data blocks, the offset and the data length of each additional data block in the data storage file,

the data storage writing unit starts from a top layered storage layer in storage layers of a data storage medium of the database system, and performs layer-by-layer data writing on the storage data of the memory table until lower-layer data writing is not triggered any more.

11. The data storage device of claim 10, wherein the data storage medium writing unit performs a lower level data write on the storage data in the data storage file in the current hierarchical storage tier that meets or exceeds a predetermined threshold in response to a file size of one or more data storage files in the current hierarchical storage tier meeting or exceeding the predetermined threshold.

12. The data storage device of claim 10, wherein the data slicing module performs data slicing on the merged storage data according to one of the following data slicing algorithms:

a data segmentation algorithm based on data usage frequency;

a data segmentation algorithm based on data columns in the original data table;

a data segmentation algorithm based on a data timestamp range;

13. The data storage device of claim 10, wherein the number of tiered storage layers of the data storage medium comprises a predetermined number of layers, and/or each tiered storage layer comprises a predetermined number of data storage files, respectively.

14. The data storage device of claim 13, wherein the number of data storage files comprised by each of the tiered storage tiers increases from top to bottom tier by tier.

15. A data query apparatus, comprising:

a memory query unit, which is used for performing data query in a memory table of a memory layer of a database system in response to receiving a data query request initiated by a user, wherein the database system performs data storage according to the method of any one of claims 1 to 7;

the data storage medium query unit is used for responding to the data query result aiming at the memory layer and not completely meeting the data query condition in the data query request, and performing data query on the data storage medium of the database system: and

a query result providing unit for providing all the obtained data query results to the user,

wherein the data storage medium querying unit comprises:

the main data block positioning module is used for positioning a data storage file in the current hierarchical storage layer according to a common hash value of a key of target data of the data query request and positioning the offset and the data length of a main data block in the data storage file according to a perfect hash value of the key in response to that a data query result aiming at the memory layer or the upper hierarchical storage layer does not completely meet the data query condition;

the main data block reading module is used for reading the main data block from the data storage file based on the offset and the data length of the positioned main data block;

the main data block query module analyzes the read main data block to obtain a data query result;

the additional data block positioning module is used for responding that the data query result aiming at the main data block does not completely meet the data query condition, and acquiring the offset and the data length of the additional data block recorded in the main data block;

the additional data block reading module is used for reading all the additional data blocks from the data storage file based on the acquired offset and data length of the additional data blocks;

the additional data block query module is used for analyzing each read additional data block to obtain a data query result;

the data storage medium query unit starts to execute data query from a top layered storage layer of a storage layer of the database system until the data query condition is completely met or the query of a bottom layered storage layer is completed.

16. The data query device as claimed in claim 15, wherein the additional data block reading module reads each additional data block from the data storage file in parallel.

17. A database system, comprising:

a data storage device as claimed in any one of claims 10 to 14;

the data querying device according to claim 15 or 16;

a memory layer; and

a data storage medium.

18. A data storage device comprising:

at least one processor for executing a program code for the at least one processor,

a memory coupled to the at least one processor, an

A computer program stored in the memory, the computer program being executable by the at least one processor to implement a data storage method as claimed in any one of claims 1 to 7.

19. A computer readable storage medium storing executable instructions that when executed cause a processor to perform a data storage method as claimed in any one of claims 1 to 7.

20. A data query apparatus, comprising:

at least one processor for processing the received data,

a memory coupled to the at least one processor, an

A computer program stored in the memory, the computer program being executable by the at least one processor to implement the data query method of claim 8 or 9.

21. A computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the data query method of claim 8 or 9.