WO2020034818A1

WO2020034818A1 - Partition merging method and database server

Info

Publication number: WO2020034818A1
Application number: PCT/CN2019/097559
Authority: WO
Inventors: 谢晓芹
Original assignee: 华为技术有限公司
Priority date: 2018-08-14
Filing date: 2019-07-24
Publication date: 2020-02-20

Abstract

A partition merging method and a database server. The method comprises: a first database server obtains the metadata of a first partition according to the identifier of the current file of the first partition, obtains the metadata of a second partition according to the identifier of the current file of the second partition, and generates the metadata of a third partition according to the metadata of the first partition and that of the second partition. The aforementioned technical solution can reduce the amount of data that is read and written, and improve a partition merging speed.

Description

Partition merge method and database server

This application requires Chinese patent applications filed on August 14, 2018 with the State Intellectual Property Office of China, application number 201810919475.9, and application name "Partition Processing Method for Distributed Database System Based on Log Structure-Merge Tree" and September 2018 The priority filed with the State Intellectual Property Office of the People's Republic of China on September 29, application number 201811147298.3, and application name "Division Merging Method and Database Server" is incorporated herein by reference in its entirety.

Technical field

The present application relates to the field of information technology, and more particularly, to a partition consolidation method and a database server.

Background technique

In a distributed database system such as a key-value database (KVDB), in order to meet the growing demand for data entries, partitions are usually used to manage the data entries of the data tables. Each partition is provided by a certain database server, such as storage management. The ownership relationship between the partition and the database server is dynamically specified by the management server of the distributed database system.

Distributed database systems require dynamic load balancing based on cluster size, load, or other policies. Therefore, in some cases, it is necessary to merge two adjacent partitions into a new partition.

At present, the merge scheme of adjacent partitions needs to read out data entries in one of the partitions and then write it to the other partition. The amount of data entries held in a partition is usually large. Therefore, the large number of data entries read and written during the partition merge process will increase the overhead significantly.

Summary of the Invention

This application provides a partition consolidation method and a database server, which can reduce the amount of data read and write.

In a first aspect, an embodiment of the present application provides a method for merging partitions in a distributed database system. The distributed database system includes a first database server, a second database server, and a management server. The first database server runs a first partition. The second database server runs a second partition; the method includes: the first database server receives a merge instruction sent by a management server, the merge instruction being used to implement merging the first partition and the second partition into a third partition, wherein the The first partition and the second partition are adjacent partitions; the merge instruction includes the identifier of the current file of the first partition and the identifier of the current file of the second partition; the current file of the first partition records that the first partition is stored A file identifier of a file of metadata of a partition; a file identifier of a file storing metadata of the second partition is recorded in a current file of the second partition; and the first database server according to the identifier of the current file of the first partition Obtaining metadata of the first partition; the first database server according to the current status of the second partition Identification documents for metadata second partition; the database server combining the first metadata and the second metadata partition first partition metadata is generated in the third partition. The above technical solution can directly merge the first partition and the second partition into the third partition according to the metadata of the first partition and the metadata of the second partition, without reading and merging the data in the first partition and the second partition to the merge. After partitioning, this can reduce the amount of data read and write, and improve the speed of partition consolidation. In addition, during the process of merging the two partitions, the business write operations of the two partitions will be temporarily frozen until the partition merge is completed. The embodiment of the present application also reduces the freeze time of the business write operations of the partitions. Optionally, the load of the first database server is lighter than the load of the second database server. In the embodiment of the present application, the method of combining the metadata of the first partition and the metadata of the second partition to generate metadata of the third partition is to implement combining the first partition and the second partition into the third partition, that is, to access the third partition. The metadata of the partition can access the data entry of the first partition and the data entry of the second partition. That is, the metadata of the third partition is generated by merging the metadata of the first partition and the metadata of the second partition. The metadata of the third partition corresponds to the data entry of the first partition and the data entry of the second partition. The data entry of the second partition and the data entry of the second partition serve as the data entries of the third partition.

With reference to the first aspect, in a possible implementation manner of the first aspect, the metadata of the first partition includes data storage unit information of a secondary column cluster of the first partition, and the metadata of the second partition includes the The data storage unit information of the second-level column cluster of the second partition, and the first database server merges the metadata of the first partition and the metadata of the second partition to generate metadata of the third partition, specifically including: the first The database server merges the data storage unit information of the second-level column cluster of the first partition and the data storage unit information of the second-level column cluster of the second partition to generate the data storage unit information of the target second-level column cluster; according to the target second-level column The data storage unit information of the cluster determines the data storage unit information of the secondary column cluster of the third partition. In the above technical solution, when the first partition and the second partition are merged, only the data storage unit information of the second-level column clusters of the first partition and the data storage unit information of the second-level column clusters of the second partition may be merged. Without copying (also known as reading and writing) the data entries stored in the corresponding data storage unit, which can reduce the amount of data read and write, thereby reducing the time that the partition is frozen, improve the efficiency of partition consolidation, and reduce business writes congestion. Optionally, the distributed database system is a database system using a long-structured merged-tree (LSM-tree) algorithm with a log structure, and the data storage unit information is Sorted String Table (SSTable) information. The data storage unit is an SSTable. The file storing the metadata of the first partition and the file storing the metadata of the second partition are both manifest files.

With reference to the first aspect, in a possible implementation manner of the first aspect, the first database server creates a current file for the third partition, and the current file of the third partition records metadata storing the third partition The file ID of the file.

With reference to the first aspect, in a possible implementation manner of the first aspect, the data storage unit information of the second-level column clusters of the first partition includes P ₁ layer data storage unit information, where P ₁ is greater than or equal to 2 Positive integer; the data storage unit information of the second-level column cluster of the second partition includes P ₂ layer data storage unit information, where P ₂ is a positive integer greater than or equal to 2; the data storage unit information of the target second-level column cluster includes Q layer data storage unit information, the Q layer data storage unit information includes data storage unit information of the second column cluster of the first partition and data storage unit information of the second column cluster of the second partition, wherein the Q layer One layer of data storage unit information of the data storage unit information includes the data storage unit information of the second-level column cluster of the first partition, and _one layer of data storage unit information in the P ₁ layer data storage unit information and the second partition of the second partition. P ₂ layer data hierarchy data information storing unit stage column data storage cluster includes the cell information in the cell information storage, Q-1 Q layer data storage unit of the information layer data storage unit Column two hierarchy data P ₁ layer data storage unit of the information storage layer of the data information of the information unit includes two columns of the first cluster storing data partition includes a cell information storage unit in the information or the second partition The data storage unit information of the cluster includes a layer of data storage unit information in the P ₂ layer data storage unit information, where Q is equal to P ₁ + P ₂ -1.

With reference to the first aspect, in a possible implementation manner of the first aspect, the 0th-level data storage unit information in the Q-layer data storage unit information includes the data storage unit information of the second-level column cluster of the first partition. Level 0 data storage unit information and level 2 data storage unit information of the secondary column cluster data storage unit information of the second partition, and layer 2 × q-1 layer data storage units of the Q layer data storage unit information information includes two columns of the first cluster storing data _{partition. 1} P layer data unit storage information comprises a first information layer data unit q P layer data storage unit of the information storage cell information, the Q layer data units stored information q-th layer data storage unit of the information layer P ₂ 2 × q data of the layer data comprises a storage means information of the second two columns of clusters storing data partition includes a cell information storage unit storing information of the P-layer data unit information, q = 1, ..., P-1, where the value of P is the smallest of P ₁ and P ₂ minus 1.

With reference to the first aspect, in a possible implementation manner of the first aspect, the 0th-level data storage unit information in the Q-layer data storage unit information includes the data storage unit information of the second-level column cluster of the first partition. Tier 0 data storage unit information and Tier 0 data storage unit information of the secondary column cluster data storage unit information of the second partition. Tier 1 data storage unit information in the Q-layer data storage unit information to P P layer data storage unit ₁ in the first layer in the data storage unit of the information P layer data storage unit to store cell information 1 layer data unit information storing two columns of data clusters that each first partition comprises information to P-1 layer data storage unit information; the Q-layer data storage unit information from the P-layer data storage unit information to the Q-1 layer data storage unit information are the data storage of the second-level column clusters of the second partition, respectively The unit information includes the layer ₂ data storage unit information in the layer ₂ data storage unit information and the layer 1 data storage unit information in the layer P data storage unit information, where the value of P is P ₁ and P ₂ Minus one minus one.

With reference to the first aspect, in a possible implementation manner of the first aspect, the method further includes: determining, by the database server, the second column cluster of the third partition according to the data storage unit information of the target second column cluster. Data storage unit information, where the data storage unit information of the second-level column cluster of the third partition includes P-layer data storage unit information, where the data storage unit information of the second-level column cluster of the third partition is P-layer data unit storage information The first-level data storage unit information is the data storage unit information of the data storage unit obtained by merging and rearranging the data storage unit corresponding to the 0th-level data storage unit information of the Q-layer data storage unit information. The third partition Each layer of data storage unit information in the layer 2 to P-1 layer data storage unit information of the data storage unit information of the second-level column cluster of the second-level column cluster is the first layer of the Q-layer data storage unit information. Data storage unit information corresponding to at least two layers of data storage unit information in layer 1 data storage unit information to layer Q-1 layer data storage unit information is merged Data storage units obtained after row information storage means. In the above technical solution, in the process of merging the first partition and the second partition, it is only necessary to read and write the data storage unit for merging and rearranging, and the data read and write volume is small, thereby reducing the time that the partition is frozen To improve the efficiency of partition consolidation and reduce business congestion. Further, the above technical solution can compress the number of information storage unit information layers in the third partition, which facilitates subsequent operations such as searching for data in the third partition.

With reference to the first aspect, in a possible implementation manner of the first aspect, a prefix of an entry key value in each data storage unit information in the data storage unit information of the secondary column cluster is a non-partitioned key value.

With reference to the first aspect, in a possible implementation manner of the first aspect, the metadata of the first partition includes data storage unit information of a secondary column cluster of the first partition, and the metadata of the second partition includes the The data storage unit information of the second-level column cluster of the second partition, and the first database server merges the metadata of the first partition and the metadata of the second partition to generate metadata of the third partition, specifically including: the first The database server merges the data storage unit information of the second-level column clusters of the first partition and the data storage unit information of the second-level column clusters of the second partition to generate the data storage unit information of the second-level column clusters of the third partition. Further, for the manner of generating the data storage unit information of the second-level column clusters of the third partition, reference may be made to the manner of generating the data storage unit information of the target second-level column clusters in the foregoing various implementations of the first aspect.

With reference to the first aspect, in a possible implementation manner of the first aspect, the metadata of the first partition further includes a set of write-ahead log information of the first partition, and the metadata of the second partition further includes the second A partitioned write-ahead log information set, the method further comprising: the database server merging the write-ahead log information set of the first partition and the write-ahead log information set of the second partition to generate a write-ahead log information set of the third partition, The write-ahead log information set of the third partition includes the write-ahead log information in the write-ahead log information set of the first partition and the write-ahead log information in the write-ahead log information set of the second partition, where N is A positive integer greater than or equal to 2, N ₁ and N ₂ are positive integers greater than or equal to 1, and the sum of N ₁ and N ₂ is N. In the above technical solution, when merging the first partition and the second partition, only the write-ahead log information set of the first partition and the write-ahead log information set of the second partition can be merged without copying the corresponding write-ahead Log information, which can reduce the amount of data read and write, thereby reducing the time that the partition is frozen, improve the efficiency of partition consolidation, and reduce business congestion.

With reference to the first aspect, in a possible implementation manner of the first aspect, the metadata of the first partition further includes data storage unit information of a main column cluster of the first partition, and the metadata of the second partition further includes The data storage unit information of the main column cluster of the second partition, the method further comprises: the database server merging the data storage unit information of the main column cluster of the first partition and the data storage unit information of the main column cluster of the second partition Generate data storage unit information of the main column cluster of the third partition. In the above technical solution, when the first partition and the second partition are merged, only the data storage unit information of the main column cluster of the first partition and the data storage unit information of the main column cluster of the second partition may be merged, and There is no need to copy the data entries in the corresponding data storage unit, which can reduce the amount of data read and write, thereby reducing the time that the partition is frozen, improve the efficiency of partition consolidation, and reduce business congestion.

With reference to the first aspect, in a possible implementation manner of the first aspect, the data storage unit information of the main column cluster of the first partition includes K ₁ layer data storage unit information, where K ₁ is a positive value greater than or equal to 1 Integer, the data storage unit information of the main column cluster of the second partition includes K ₂ layer data storage unit information, where K ₂ is a positive integer greater than or equal to 1; the data storage unit information of the main column cluster of the third partition includes K-layer data storage unit information, where the data storage unit information of the main column cluster of the third partition includes the K-th data storage unit information in the K-layer data storage unit information including the K-layer of the K _1- level data storage unit information k-th layer data storage layer data unit information K information data storage unit storing the cell information in the k-th layer data storage unit of the information layer and the data storage unit K ₂ information, wherein K is the minimum value of K ₁ and K ₂ wherein the k-th layer data entry key according to any of the k-th layer data storage layer data K ₁ cell information storage unit storing information in a cell information and data of the data storage layer K ₂ information storage unit A key entry of any data stored in the cell information unit of information do not overlap.

With reference to the first aspect, in a possible implementation manner of the first aspect, a prefix of an entry key value in each data storage unit information in the data storage unit information of the main column cluster is a partition key value.

In a second aspect, a database server is provided, and the database server includes a unit for executing the first aspect or any possible implementation manner of the first aspect.

In a third aspect, a database server is provided. The database server includes a processor and a communication interface. The processor combines the communication interface to implement the first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer storage medium, and the database server runs a computer instruction to implement the first aspect or any possible implementation manner of the first aspect.

In a fifth aspect, the present application provides a computer program product containing instructions, and when the computer instructions in the computer program product are run on a database server, the database server is caused to execute the first aspect or any one of the first aspect above. Implementation.

In a sixth aspect, the present application provides a method for merging partitions in a distributed database system. The distributed database system includes a first database server, a second database server, and a management server. The first database server runs the first partition. The two database servers run a second partition; the method includes: the management server creates a third partition and determines to merge the first partition and the second partition into the third partition; the management server sends a merge instruction to the first database server , The merge instruction is used to implement merging the first partition and the second partition into a third partition, wherein the first partition and the second partition are adjacent partitions; the merge instruction includes a current file of the first partition An identifier and an identifier of a current file of the second partition; a file identifier of a file storing metadata of the first partition is recorded in the current file of the first partition; and a storage of the second is recorded in a current file of the second partition The file ID of the partitioned metadata file.

With reference to the sixth aspect, in a possible implementation manner of the sixth aspect, the management server receives a response message sent by the first database, and the response message includes a current file identifier of the third partition.

With reference to the sixth aspect, in a possible implementation manner of the sixth aspect, the management server establishes a mapping relationship between the third partition and the first database server. After the management server creates the third partition, the partition routing table is updated. The partition routing table includes the mapping relationship between the third partition and the first database server. The specific implementation may be the mapping relationship between the identifier of the third partition and the address of the database server.

With reference to the sixth aspect, in a possible implementation manner of the sixth aspect, the management server determines, according to the load of the first database server and the second database server, that the first database server divides the first partition and the first database server. The two partitions are merged into the third partition; wherein the load of the first database server is lighter than the load of the second database server.

According to a seventh aspect, a management server is provided, and the management server includes a unit for executing the sixth aspect or any one of the possible implementation manners of the sixth aspect.

According to an eighth aspect, a management server is provided. The management server includes a processor and a communication interface. The processor implements the sixth aspect or any possible implementation manner of the sixth aspect in combination with the communication interface.

In a ninth aspect, an embodiment of the present application provides a computer storage medium, and the management server runs a computer instruction to implement the sixth aspect or any possible implementation manner of the sixth aspect.

In a tenth aspect, the present application provides a computer program product containing instructions. When the computer instructions in the computer program product run on a management server, the management server implements any of the foregoing sixth aspect or the sixth aspect. Implementation.

In an eleventh aspect, the present application provides a distributed database system, where the distributed database system includes a first database server, a second database server, and a management server; the first database server is configured to implement the first aspect or the first Any possible implementation manner of one aspect, and the management server is configured to implement the foregoing sixth aspect or any possible implementation manner of the sixth aspect.

In a twelfth aspect, an embodiment of the present application provides a method for merging partitions in a distributed database system. The distributed database system includes a first database server, a second database server, and a management server. The first database server runs a first partition. The second database server runs a second partition; the method includes: the first database server receives a merge instruction sent by a management server, the merge instruction is used to implement merging the first partition and the second partition into a third partition, where The first partition and the second partition are adjacent partitions; the first database server obtains metadata of the first partition and metadata of the second partition according to the first partition, and combines metadata of the first partition and The metadata of the second partition generates metadata of the third partition.

With reference to the twelfth aspect, in a possible implementation manner of the twelfth aspect, the merge instruction includes an identifier of the current file of the first partition and an identifier of the current file of the second partition; A file identifier of a file storing metadata of the first partition is recorded in the file; a file identifier of a file storing metadata of the second partition is recorded in a current file of the second partition; the first database server obtains the first The metadata of a partition and the metadata of the second partition specifically include: obtaining the metadata of the first partition according to the identifier of the current file of the first partition, and obtaining the second metadata according to the identifier of the current file of the second partition Partition metadata.

For other possible implementation manners in the twelfth aspect, reference may be made to any possible implementation manner in the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of a partition.

Figure 2 is a schematic diagram of the KVDB architecture.

FIG. 3 is a schematic flowchart of a method for processing a partition according to an embodiment of the present application.

FIG. 4 is a schematic diagram of a layer merging process.

FIG. 5 is a structural block diagram of a database server according to an embodiment of the present application.

FIG. 6 is a structural block diagram of a database server according to an embodiment of the present invention.

FIG. 7 is a structural block diagram of a management server according to an embodiment of the present application.

FIG. 8 is a structural block diagram of a management server according to an embodiment of the present invention.

detailed description

The technical solutions in this application will be described below with reference to the drawings.

In the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" describes the association relationship of related objects, and indicates that there can be three kinds of relationships, for example, A and / or B can represent: the case where A exists alone, A and B exist simultaneously, and B alone exists, where A, B can be singular or plural. The character "/" generally indicates that the related objects are an "or" relationship. "At least one or more of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one item (a), a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c may be single or multiple. In addition, in the embodiments of the present application, the words “first”, “second” and the like do not limit the number and execution order.

It should be noted that, in this application, words such as "exemplary" or "for example" are used as examples, illustrations, or illustrations. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the words "exemplary" or "for example" is intended to present the relevant concept in a concrete manner.

In order to facilitate those skilled in the art to better understand the embodiments of the present application, some basic concepts related to the distributed database system in the embodiments of the present application are first introduced.

KVDB is a database that uses a key-value store. The data in the database is organized, indexed, and stored in the form of key values. Common KVDB includes: RockesDB, LevelDB, etc.

Partitioning key value: In partitioning technology, the value of a certain column field of the data entry in the data table, or the sequential combination of several column field values is usually used to determine the partition. This value is called the partition key. . Each data entry can uniquely determine the partition in which the data entry is located according to its partition key value.

The partition key values are described in more detail in conjunction with Table 1.

Table 1

bnbn	onon	verver	crtcrt	dlcdlc
AA	dd	11	15012328529861501232852986	2020
A A		dd	22	15012328529861501232852986	2020
BB	xx	11	15067102389831506710238983	2020

Table 1 shows the five column fields {bn, on, ver, crt, dlc} of the three data entries. For example, the values of the five column fields of the first data entry are {A, d, 1, 1501232852986, 20}. The partition key value may be a value of one column field of the above five column fields, or a sequential combination of the values of several column fields. For example, suppose the partition key value is a combined value of the column field {bn, on}. It can be seen that in the above three data entries, the column fields {bn, on} of the first data entry and the second data entry are both {A, d}. In other words, the partition key values for the first and second data entries are both {A, d}. The column field {bn, on} of the third data entry among the above three data entries is {B, x}. In other words, the partition key value for the third data entry is {B, x}.

For ease of description, the value of the column field On is represented by a letter in Table 1. In practical applications, the value of the column field On may be a specific value, such as 0109, 0208, and the like.

The data entries in the partition in the embodiment of the present application are arranged according to the natural order of the partition key values. It is still assumed that the partition key value is the combined value of the column fields {bn, on}. FIG. 1 is a schematic diagram of multiple partitions with a partition key value of {bn, on}. The assignment of points on the boundaries of the five partitions as shown in FIG. 1 can be divided using the principle of left opening, right closing, or right opening, left closing. If the principle of left-open and right-close is adopted, the data entry with partition key values of bn = A and on = w is the data entry in partition 1. If the right open and left closed principle is adopted, the data entry with partition key values of bn = A and on = w is the data entry in partition 2. With reference to the partitioning diagram in Figure 1 and the partition key values of the three data entries shown in Table 1, it can be seen that the first data entry and the second data entry are the data entries in partition 1, and the third data entry is the partition. Data entry in 2. In order to ensure that the data entries in the partition are arranged in the natural order of the partition key values, the data table can be split by using a range partition method. Other data table splitting methods may also cause the data entries in the partition to be arranged in the natural order of the partition key values. Therefore, the embodiment of the present application does not limit the method of splitting the data table, as long as the method of splitting the data table enables the data table to be split, the data entries in the partition are arranged in the natural order of the partition key values. It can be seen from the partitions shown in FIG. 1 that the partition key values in partitions 1 and 2 are arranged in a natural order and are continuous, which are called adjacent partitions. In the embodiment of the present application, KVDB manages the data entries in the data table with partitions as granularity and is managed by different database servers. That is, the database server provides access to partitioned data entries. .

Partition routing table: The partition routing table can include the following information: the partition ID and the address of the database server to which it belongs, and it can also include the partition root index file ID, the left boundary of the partition, the right boundary of the partition, and the partition status. The address of the database server may be an Internet Protocol (IP) address, or an identifier of the database server, etc. This is not limited in this embodiment of the present invention, and the partition routing table is also referred to as a partitioned view. Figure 1 is a schematic diagram of a partition. For example, Table 2 is a partition routing table based on multiple partitions shown in FIG. The partition root index file identifier identifies the file name of the inventory file for each partition. The partition status indicates the current status of the partition, such as normal service status, split status, merge status, isolation status, and so on. Normal shown in Table 2 indicates that the current status of the partition is normal service status.

Table 2

$ Min is infinitely small, and $ Max is infinite. With the partition key value of the data entry as input, querying the partition routing table can obtain information such as the partition to which the data entry belongs and the database server to which it belongs. Taking the third data entry in Table 1 as an example, the partition key value of the data entry is {B, x}. According to the partition routing table shown in Table 2, it can be determined that the partition to which the data entry belongs is partition 2, and the home database server address is 8.11.234.2:27021.

The partition merges referred to in the embodiments of the present application are all merges of adjacent partitions, for example, merge partition 1 and partition 2, merge partition 3 and partition 4, and merge partition 4 and partition 5.

The primary index entry includes the partition key value. In addition to the partition key value, the main index entry can also include values from several other column fields of the data entry. Partition key values and values from one or more column fields from multiple other column fields can make up the primary index entry. Taking Table 1 as an example, the main index entry may include five column fields {bn, on, ver, crt, dlc} as shown in Table 1, where the key value of the main index entry corresponds to the column field {bn, on, ver} . {bn, on} can be called the prefix of the main index entry. It can be seen that {bn, on} is the partition key value. Therefore, the primary index entry is the primary index entry prefixed by the partition key value. An entry key is a unique index into a data entry and can include multiple column fields for that data entry.

Secondary index entries can also be called secondary index entries. In a distributed database system, to meet complex query scenarios, one or more secondary index entries can be created for each data entry. The secondary index entry consists of the secondary index column field and the column field corresponding to the key value of the primary index entry. The format of the secondary index entry can be the secondary index column field + the primary index entry key value column field, or the secondary index column field and the primary index entry key value column field. In this case, the secondary index entry is a secondary index entry that is prefixed with a non-partitioned key value. The format of the secondary index entry can also be the column field corresponding to the key value of the primary index entry + the secondary index column field. In this case, the secondary index entry is a secondary index entry prefixed by the partition key value. When each data entry includes multiple secondary index entries, the secondary index column fields included in different secondary index entries in the multiple secondary index entries are different. For example, suppose each data entry includes two secondary index entries, one of which can be dlc and the other can be crt.

Each partition may include a primary index entry set or one or more secondary index entry sets. The main index entry set consists of all the main index entries under the partition. The primary index entries in the primary index entry set are sorted by the primary index entry key value.

The secondary index entries in the secondary index entry set are sorted by the secondary index entry key value. If each partition includes multiple secondary index entry sets, each secondary index entry set consists of secondary index entries with the same secondary index entry column domain. Assume that each data entry includes two secondary index entries, one of which can be dlc and the other can be crt. This partition can include two secondary index entry sets, one of which includes all secondary index entry key values of the secondary index column domain of the partition under crt, and the other secondary index entry set includes the partition. The secondary index column field is the key value of all secondary index entries of the dlc.

Multiple column domains with the same characteristics can become a column family. As described above, the main index entries included in the main index entry set have the same multiple column fields. Therefore, a main index entry set can be called a column cluster, and the column cluster can be called a main index column cluster. Similarly, the secondary index entries included in a secondary index entry set also have the same multiple column fields. Therefore, a secondary index entry can also be used as a column cluster, which can be called a secondary index column cluster.

Some embodiments of the present application are described by taking a KVDB adopting a log-structured merged-tree (LSM-tree) algorithm as an example. Therefore, the following briefly introduces the LSM-tree.

Write-ahead logging (WAL) file: When a data entry is inserted into the KVDB, the data entry is written into the WAL file first, and then inserted into the memory table (memTable) after the write is successful.

MemTable: MemTable corresponds to a WAL file, which is an ordered organization structure of the contents of the WAL file in memory. MemTable provides a structure for writing, deleting, and reading key-value data (data entries). MemTable internally stores data items in order by entry key value.

Immutable MemTable: When the memory space occupied by MemTable reaches an upper limit, it is necessary to dump the data items stored in memory in order according to the entry key value to the Sorted String Table (SSTable), and correspondingly WAL files no longer write new data entries. At this time, the MemTable will be frozen into an immutable MemTable (Immutable MemTable), and a new MemTable will be generated at the same time. The new data entry is recorded in the new WAL file and the newly generated MemTable. The data entries in the immutable MemTable are immutable. In other words, the data items in the immutable MemTable can only be read, not written or deleted.

SSTable is the unit where KVDB data is stored. The entry key values in each SSTable are ordered. After each immutable MemTable is merged, an SSTable is obtained. The process of merging the immutable MemTable to obtain the SSTable can become a minor compaction.

KVDB divides the storage of the SSTable file into different levels, level 0 to level n, where n is a positive integer greater than or equal to 1. Level 0 includes multiple SSTable files. Among the multiple SSTables, one SStable is obtained by sub-merging an immutable MemTable. In other words, sub-merging multiple immutable MemTables to obtain corresponding SSTables. The entry keys of different SSTables in the multiple SSTables will overlap. After meeting certain conditions, the SSTable in level 0 and SSTable in level 1 are merged, and the SSTable obtained after the merge is the SSTable stored in level 1.

Each level in level 1 to level n maintains the specified number of SSTables, and the entry key values between all SSTables in each level do not overlap. When the SSTable in a level meets certain conditions, the SSTable in that level can be selected as the Level layer corresponding to the next level (that is, the value of level plus 1). For example, the next level of level 1 is level 2. The next level of level 2 is level 3, and so on). After merging, the selected SSTable is deleted. The merge processing of the SSTable between the two levels can be referred to as a major compaction.

Manifest file: The manifest file is used to record WAL file information and SSTable information. More specifically, the WAL file information recorded in the manifest file includes an identification of the WAL file and a time serial number of the WAL file. The SSTable information recorded in the manifest file includes the column cluster to which the SSTable belongs, the level to which the SSTtable belongs, the identity of the SSTable, the time sequence number of the SSTable, the size of the SSTable, the minimum entry key value of the SSTable, and the maximum entry of the SSTable. One or more of the key-values.

The KVDB will be described below with reference to FIG. 2.

The KVDB 200 shown in FIG. 2 includes a management server 210, a database server 221, a database server 222, and a database server 223. The KVDB 200 shown in FIG. 2 may further include a storage server 231, a storage server 232, and a storage server 233.

The database server 221, the database server 222, and the database server 223 may be collectively referred to as a distributed database service cluster. The storage server 231, the storage server 232, and the storage server 233 may provide a distributed shared storage pool for KVDB. In specific implementation, KVDB can also be provided with storage resources by centralized storage, for example, storage arrays can provide storage resources for KVDB.

The management server 210 is responsible for specifying the ownership relationship between the partition and the database server. The partition routing table is also maintained by the management server 210.

The above operations of freezing a MemTable to an immutable MemTable, a secondary merge, and a primary merge can all be performed by a database server.

In a distributed database system, a database server is responsible for the storage management of data items in a partition. Therefore, taking the storage server as an example, the WAL file, SSTable, and manifest file of the corresponding partition generated by the database server can be persisted in the storage server. The database server can access the WAL file, SSTable, and manifest file of the corresponding partition stored in the storage server. MemTable, immutable MemTable are stored in the memory of the database server.

FIG. 3 is a schematic flowchart of a method for processing a partition according to an embodiment of the present application. The method shown in FIG. 3 can be applied to a KVDB of a merge tree based on a log structure. Among them, partition 1 is served by database server 1, and partition 2 is served by database server 2.

301. The management server merges adjacent first partitions and second partitions according to a balancing policy, marks partition 1 and partition 2 as merged states in the partition routing table, and persists them.

The balancing strategy can be the number of partition data entries, access popularity, etc., or the load of the database server running the partition.

302. The management server notifies the first partition of the database server 1 that it is ready to merge.

303. The management server notifies the second partition of the database server 2 that it is ready to merge.

304. The database server 2 stops the merge task and sets the second partition as read-only, suspends the write request, and sends a successful response to the management server.

Specifically, if the database server 2 has a merge task that has not yet started, the database server 2 stops performing the merge task. If the database server 2 is performing a merge task, the database server 2 stops performing the merge task after completing the ongoing merge task. In other words, the database server 2 does not make changes to the content saved in the second partition after receiving the notification of preparation for merging sent by the management server 1.

305. The database server 1 stops the merge task and sets the first partition as read-only, suspends the write request, and sends a successful response to the management server.

Step 305 is similar to step 304, and it is unnecessary to repeat it here.

306. The management server creates a third partition in the distributed database system, and marks the third partition as an initial state in the partition routing table.

The management server creates a third partition in the distributed database system. The specific implementation may be to generate the identifier of the third partition and add the identifier of the third partition to the partition routing table. Exemplarily, in a specific implementation process in the embodiment of the present application, the management server may determine that the third partition is run by the lightly loaded database server 1 based on the load of the database server 1 and the database server 2, for example, the load of the database server 1 is small. The third partition is run on the database server 1, and the database server 1 provides services for the third partition. It can be understood that, optionally, the management server may also instruct other database servers (such as database server 2 or database server 3) to merge the first partition and the second partition into the third partition, that is, in other database servers Run on the third partition. In the specific implementation process in the embodiment of the present application, in another implementation manner, a database server 1 and a database server 2 are randomly selected to run the third partition, or a database is selected between the database server 1 and the database server 2 through a specific algorithm. The server runs the third partition. For example, a database server may perform the third partition based on a partition identifier (such as a partition number) managed by the database server to determine the total number of partitions. Specifically, the management server determines the database server running the third partition according to the foregoing implementation manner.

307. The management server sends a merge instruction to the database server 1, and the merge instruction is used to implement merging the first partition and the second partition into a third partition.

In the embodiment of the present application, the method of combining the metadata of the first partition and the metadata of the second partition to generate metadata of the third partition is to implement combining the first partition and the second partition into the third partition, that is, to access the third partition. The metadata of the partition can access the data entry of the first partition and the data entry of the second partition. Specifically, in an implementation, the merge instruction includes an identifier of a current file of the first partition and an identifier of a current file of the second partition. In the current file of the first partition, a file identifier of a file storing metadata of the first partition is recorded. A file identifier of a file storing metadata of the second partition is recorded in the current file of the second partition. Taking KVDB using the LSM-tree algorithm as an example, the manifest file in the KVDB of the LSM-tree algorithm is used to store partition metadata, so the file identifier of the file in which the metadata of the storage partition is recorded in the current file may be LSM-tree The identification of the manifest file in the algorithm's KVDB.

308. The database server 1 creates a database corresponding to the third partition.

The database server 1 creates a database corresponding to the third partition. One implementation is to start a new database process or database instance, and another implementation may be to use the currently running database process or database instance as the database of the third partition.

309. The database server 1 obtains metadata of the first partition and metadata of the second partition.

Specifically, the database server 1 reads the current file of the first partition to obtain metadata of the first partition; reads the current file of the second partition to obtain metadata of the second partition; and combines the metadata of the first partition with the The metadata of the second partition is loaded into the memory of the database server 1.

In another implementation, the database server 1 obtains the address of the database server 2 running the second partition from the management server, the database server 1 obtains the metadata of the second partition from the database server 2, or obtains the metadata of the second partition from the database server 2. For the information of the data, the database server 1 obtains the metadata of the second partition according to the information of the metadata of the second partition. The information of the metadata of the second partition may be a file identifier of a file storing the metadata of the second partition.

310. The database server 1 merges metadata of the first partition with metadata of the second partition to generate metadata of the third partition.

Specifically, the database server 1 generates metadata of the third partition. Further, the database server 1 creates a current file for the third partition, and the current file of the third partition records a file identifier of a file storing metadata of the third partition.

The metadata of the first partition includes data storage unit information of a secondary column cluster of the first partition. The metadata of the second partition includes data storage unit information of the secondary column clusters of the second partition. Taking KVDB using the LSM-tree algorithm as an example, the data storage unit information is SSTable information, and the data storage unit is SSTable.

In the first partition and the second partition, the index (including the primary index and the secondary index) prefixed by the partition key value and the secondary index entry prefixed by the non-partition key value are stored in different column clusters, respectively. The data storage unit information of the primary column cluster refers to the related information organized and stored by the index (including the primary index and the secondary index) entries prefixed by the partition key value. The data storage unit information of the secondary column cluster refers to related information organized and stored by the secondary index entries whose prefixes are non-partitioned key values.

The database server 1 may combine data storage unit information of the secondary column clusters of the first partition with data storage unit information of the secondary column clusters of the second partition to generate data storage unit information of the target secondary column clusters. Optionally, in some embodiments, the data storage unit information of the target secondary column cluster may be used as the data storage unit information of the secondary column cluster of the third partition. Optionally, in other embodiments, the data server 1 may determine the data storage unit information of the secondary column cluster of the third partition according to the data storage unit information of the target column cluster.

The data storage unit information of the second column cluster of the first partition includes data storage unit information of the P ₁ layer, where P ₁ is a positive integer greater than or equal to 2; the data storage unit information of the second column cluster of the second partition Including the data storage unit information of the P ₂ layer, where P ₂ is a positive integer greater than or equal to 2. The data storage unit information of the target secondary column cluster includes the data storage unit information of the Q layer, and the data storage unit information of the Q layer The data storage unit information of the second-level column cluster of the first partition and the data storage unit information of the second-level column cluster of the second partition, wherein the one-level data storage unit information of the Q-level data storage unit information includes the _One layer of data storage unit information in the data storage unit information of the layer P _{1 and one} layer of data storage unit information in the data storage unit information of the P ₂ layer, and the data of the Q-1 layer of the data storage unit information of the Q layer information storage means storing one data unit information includes the number of data of one layer of the data storage layer P ₁ in the cell information storage unit of the information or data layer P ₂ is stored in the cell information Information storing _{means,. 1} + wherein Q is equal to P ₂ -1 P.

Optionally, in some embodiments, the layer 0 data storage unit information in the layer Q data storage unit information includes the layer 0 data storage unit information of the data storage unit information of the second-level column clusters of the first partition and The layer 0 data storage unit information of the data storage unit information of the second-level column cluster of the second partition, and the layer 2 × q-1 layer data storage unit information in the data storage unit information of the Q layer includes the data storage of the P ₁ layer. The q layer data storage unit information of the P layer data storage unit information in the unit information, and the 2 × q layer data storage unit information in the Q layer data unit storage information includes the P layer data storage in the P ₂ layer data storage unit information. The q-level data of the unit information stores unit information, where q = 1,..., P-1, where the value of P is the minimum of P ₁ and P ₂ minus 1.

Alternatively, in some embodiments, P-layer data of the layer data P ₁ in the cell information storage unit storing information may be the first layer of the P ₁ 0 layer data storage unit to the first information layer P-1 data information storage unit . Similarly, the data storage layer P ₂ P layer data unit information storing means that the information may be stored P ₂ layer data unit information layer 1 to layer data P stored cell information.

Alternatively, in some embodiments, layer data of the P ₁ P layer data unit information storing means storing information may be _a P layer of the data storage unit in the information layer of the reciprocal of the first to the penultimate layer data storage unit of the information P . Similarly, the data storage layer P ₂ P layer data unit information storing means that the information may be the data storage layer P ₂ in the cell information to the inverse of the first layer P layer penultimate data cell information is stored.

Alternatively, in some embodiments, the data storage layer _{P. 1} P layer data unit information storing means information may be an intermediate layer data of the P P _{unit. 1} layer data stored in the storage cell information message. For example, the data storage layer P ₁ P layer data unit information storing unit of the information layer of the P-2 to P + 1 layer ₁ layer data storage means data information storing cell information. Similarly, the data storage layer P ₂ P layer data unit information storing means information may be an intermediate layer data of the P ₂ P layer data stored in the cell information storage unit information. For example, the layer ₂ data storage unit information in the layer ₂ data storage unit information and the layer 2 to P + 1 layer data storage unit information in the layer 2 data storage unit information.

Taking KVDB using the LSM-tree algorithm as an example, as mentioned earlier, the data storage unit information is SSTable information, and the data storage unit is SSTable. A first partition manifest file includes information SSTable layer P ₁ column two clusters, the index entry information SSTable layer P ₁ is non-partition key value for the two index entries prefix. The second partition manifest file information comprises P ₂ layer SSTable two columns cluster, the index entry information SSTable P ₂ layer in a non-partition key value for the two index entries prefix. The target manifest file includes the Q-level SSTable information of the second-level column clusters, and the index entries of the Q-level SSTable information are the second-level index entries prefixed by the non-partition key value. Optionally, in some embodiments, the target manifest file may be a manifest file of the third partition. Optionally, in other embodiments, the target manifest file may be used to determine the manifest file of the third partition.

Assume that P ₁ and P _{2 both} take the value 2. In this case, Q = 2 + 2-1 = 3. In this case, the manifest file of the first partition includes the 2-level SSTable information of the secondary column cluster, which is the SSTable information of the 0th level to the SStable information of the 1st level, respectively. The list file of the second partition includes the second-level SSTable information of the second-level column cluster, which is the SSTable information of the 0th layer to the SStable information of the first layer. The target list file includes the three-level SSTable information of the second-level column clusters, which are the tier-0 SSTable information to the second-tier SStable information.

The layer 0 SSTable information in the target manifest file includes the layer 0 SSTable information in the manifest file of the first partition and the layer 0 SSTable information in the manifest file of the second partition.

The layer 1 SSTable information in the target manifest file includes the layer 1 SSTable information in the manifest file of the first partition.

The layer 2 SSTable information in the target manifest file includes the layer 1 SSTable information in the manifest file of the second partition.

The following describes the combination of data storage unit information of the secondary column clusters with reference to Tables 3, 4 and 5.

table 3

Table 3 shows the two-layer data storage unit information included in the data storage unit information of the second-level column cluster of the first partition. Each layer of data storage unit information in the two-layer data storage unit information includes two data storage unit information. Table 3 is a table obtained by taking KVDB using the LSM-tree algorithm as an example. Therefore, the corresponding data storage unit information can also be called SSTable information.

Table 4

Table 4 shows the two-layer data storage unit information included in the data storage unit information of the second-level column cluster of the second partition. Each layer of data storage unit information in the two-layer data storage unit information includes two data storage unit information. Table 4 is a table obtained by taking KVDB using the LSM-tree algorithm as an example. Therefore, the corresponding data storage unit information can also be called SSTable information.

table 5

Table 5 shows the three-layer data storage unit information included in the data storage unit information of the target second-level column cluster. The 0-level data storage unit information in the three-layer data storage unit information includes 4 data storage unit information. The layer and second layer data storage unit information includes two data storage unit information, respectively. Table 5 is a table obtained by taking KVDB using the LSM-tree algorithm as an example. Therefore, the corresponding data storage unit information can also be called SSTable information. It can be seen that the SSTable information of level 0 shown in Table 5 includes the SSTable information of level 0 shown in Table 3 and Table 4, and the SSTable information of level 1 shown in Table 5 includes the SSTable information of level 1 shown in Table 3. . The SSTable information of level 2 shown in Table 5 includes the SSTable information of level 2 shown in Table 4. In other words, the SSTable information of level 0 in the target manifest file after merging includes the SSTable information of level 0 in the manifest file of the first partition and the SSTable information of level 0 in the manifest file of the second partition; The SSTable information of level 1 in the target manifest file includes the SSTable information of level 1 in the manifest file of the first partition; the SSTable information of level 2 of the target manifest file after the merge includes the SSTable of level 1 in the manifest file of the second partition information.

Optionally, in other embodiments, the 2 × q-1 layer data storage unit information in the Q layer data storage unit information includes the first layer of the P layer data storage unit information in the second-level column cluster of the second partition. q layer data storage unit information. The 2 × q layer data storage unit information in the 2 × P layer data unit storage information includes the q layer data storage of the P layer data storage unit information of the second-level column cluster of the first partition. Unit information.

Assume that P ₁ and P _{2 both} take the value 2. In this case, Q = 2 + 2-1 = 3. In this case, the manifest file of the first partition includes layer 0 SSTable information and layer 1 SStable information. The manifest file of the second partition includes layer 0 SSTable information and layer 1 SStable information. The target list file includes the three-level SSTable information of the second-level column clusters, which are the tier-0 SSTable information to the second-tier SStable information.

The layer 1 SSTable information in the target manifest file includes the layer 1 SSTable information in the manifest file of the second partition.

The layer 2 SSTable file information in the target manifest file includes the layer 1 SSTable information in the manifest file of the first partition.

In the above embodiment, the SSTable information from the first partition and the SSTable information from the second partition are superimposed on each other. In other embodiments, the SSTable information from the first partition and the SStable information from the second partition may be stacked.

Optionally, in some embodiments, the layer 0 data storage unit information in the layer Q data storage unit information includes the layer 0 data storage unit information of the first partition and the layer 0 data of the second partition Storage unit information. The first layer data storage unit information to the P-1 layer data storage unit information in the Q layer data storage unit information are the first layer data storage in the P layer data storage unit information of the first partition. The unit information to the P-1 layer data storage unit information; the Q layer data storage unit information from the P layer data storage unit information to the Q-1 layer data storage unit information are the P layer data storage of the second partition, respectively Among the unit information, layer 1 data stores unit information to layer P-1 layer data stores unit information.

Assume that P ₁ and P _{2 both} take the value 4. In this case Q = 4 + 4-1 = 7. In this case, the manifest file of the first partition includes the 4-level SSTable information of the second-level column cluster, which is the SSTable information of the 0th level to the SStable information of the 3rd level. The manifest file of the second partition includes the 4-level SSTable information of the second-level column clusters, which are the SSTable information of the 0th level to the SStable information of the 3rd level. The target list file includes the 7-level SSTable information of the secondary column cluster, which is the SSTable information of the 0th layer to the SStable information of the 6th layer.

The layer 2 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the first partition.

The layer 3 SSTable information in the target manifest file includes the layer 3 SSTable information in the manifest file of the first partition.

The layer 4 SSTable information in the target manifest file includes the layer 1 SSTable information in the manifest file of the second partition.

The layer 5 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the second partition.

The layer 6 SSTable information in the target manifest file includes the layer 3 SSTable information in the manifest file of the second partition.

Optionally, in other embodiments, the layer 0 data storage unit information in the layer Q data storage unit information includes the layer 0 data storage unit information of the first partition and the layer 0 data of the second partition. Storage unit information, the first layer data storage unit information to the P-1 layer data storage unit information in the Q layer data storage unit information are the first layer data in the second layer P layer data storage unit information The storage unit information to the P-1 layer data storage unit information; the P-layer data storage unit information to the Q-1 layer data storage unit information in the Q-layer data storage unit information are the P-layer data of the first partition, respectively Among the storage unit information, layer 1 data storage unit information to layer P-1 layer data storage unit information.

Assume that P ₁ and P _{2 both} take the value 4. In this case Q = 4 + 4-1 = 7. In this case, the manifest file of the first partition includes layer 0 SSTable information to layer 3 SStable information. The manifest file of the second partition includes layer 0 SSTable information to layer 3 SStable information. The target list file includes the 7-level SSTable information of the secondary column cluster, which is the SSTable information of the 0th layer to the SStable information of the 6th layer.

The layer 2 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the second partition.

The layer 3 SSTable information in the target manifest file includes the layer 3 SSTable information in the manifest file of the second partition.

The layer 4 SSTable information in the target manifest file includes the layer 1 SSTable information in the manifest file of the first partition.

The layer 5 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the first partition.

The layer 6 SSTable information in the target manifest file includes layer 3 SSTable information in the manifest file of the first partition.

It can be seen that in the process of merging the data storage unit information of the secondary column clusters of adjacent partitions, the tier 0 data storage unit information is merged by directly merging the data storage unit information of the two-level column clusters of the two partitions in the 0th layer. The data storage unit information is merged into the data storage unit information of the same second-level column cluster. In other words, the merging method is to directly add the layer 0 data storage unit information in the second partition directly to the layer 0 data storage unit information in the first partition. This merge method can be called append merge. In the process of merging the data storage unit information of the secondary column clusters of adjacent partitions, the data storage unit information other than the layer 0 data storage unit information is merged in a superimposed manner. The following merge method can be Overlay merge.

In the above example, P ₁ and P _{2 have the} same value. In some cases, the values of P ₁ and P ₂ may be different. In this case, the Q layer data storage unit 2 × P layer data storage unit information may include the _{P. 1} layer data store P layer data unit information storing cell information and P ₂ layer data stored in the cell information P-layer data stores unit information. The Q-2 × P layer data storage unit information of the Q layer data storage unit information may include P ′ layer data storage unit information, where the value of P ′ is max (P ₁ , P ₂ ) -min (P ₁ , P ₂ ), where max (P ₁ , P ₂ ) represents the maximum value of P ₁ and P ₂ , and min (P ₁ , P ₂ ) represents the minimum value of P ₁ and P ₂ . In other words, P 'is equal to ₁ and the maximum value of P ₂ P ₁ and subtracting the minimum value P of the P _2. Or the value of P 'is | P ₁ -P ₂ |, that is, the value of P' is the absolute value of the difference between P ₁ and P ₂ .

For example, suppose P ₁ = 5 and P ₂ = 3, in which case Q = 5 + 3-1 = 7. In this case, the manifest file of the first partition includes the 5-level SSTable information of the second-level column cluster, which is the SSTable information of the 0th level to the SStable information of the 4th level. The list file of the second partition includes the three-level SSTable information of the second-level column clusters, which are the tier-0 SSTable information to the second-tier SStable information, respectively. The target list file includes the 7-level SSTable information of the secondary column cluster, which is the SSTable information of the 0th layer to the SStable information of the 6th layer.

The layer 0 SSTable file information in the target manifest file includes the layer 0 SSTable information in the manifest file of the first partition and the layer 0 SSTable information in the manifest file of the second partition.

The layer 3 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the first partition.

The layer 4 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the second partition.

The layer 5 SSTable information in the target manifest file includes layer 3 SSTable information in the manifest file of the first partition.

The layer 6 SSTable information in the target manifest file includes the layer 4 SSTable information in the manifest file of the first partition.

In other words, the P 'layer data storage unit information may be the last P' layer data storage unit information in the Q layer data storage unit information.

Optionally, in other embodiments, the P'-layer data storage unit information may also be the pre-P'-layer data storage unit information in the Q-layer data storage unit information.

The layer 5 SSTable information in the target manifest file includes the layer 4 SSTable information in the manifest file of the first partition.

The layer 6 SSTable information in the target manifest file includes the layer 2 SSTable information in the manifest file of the second partition.

The metadata of the first partition further includes data storage unit information of a main column cluster of the first partition. The metadata of the second partition includes data storage unit information of a main column cluster of the second partition.

The database server 1 may combine data storage unit information of the main column cluster of the first partition with data storage unit information of the main column cluster of the second partition to generate data storage unit information of the main column cluster of the third partition.

The first column of the primary partition cluster data storage means K ₁ information includes cell information storage layer data, wherein K ₁ is greater than or equal to a positive integer. The data storage unit information of the main column cluster of the second partition includes K ₂ layer data storage unit information, where K ₂ is a positive integer greater than or equal to 1. The metadata of the third partition includes data storage unit information of the main column cluster of the third partition, and the data storage unit information of the main column cluster of the third partition includes K-layer data storage unit information, where the main column of the third partition The k-th data storage unit information of the data storage unit information of the cluster includes the k-th data storage unit information in the K-level data storage unit information of the K ₁ -level data storage unit information and the K of the K _2- level data storage unit information. The k-th data storage unit information in the layer data storage unit information, where K is the minimum value of K ₁ and K ₂ , and any one of the k-th data storage unit information in the K _1- level data storage unit information is stored. The entry key value of the unit information does not overlap with the entry key value of any data storage unit information in the k-th data storage unit information of the K ₂ layer data storage unit information.

M _k1 represents the number of data storage unit information included in the k-th data storage unit information in the K-layer data storage unit information in the data storage unit information of the main column cluster of the first partition. For example, M ₀₁ = 2 indicates that level 0 of the data storage unit information of the main column cluster of the first partition includes 2 data storage unit information. M _k2 indicates the number of data storage unit information included in the k-th data storage unit information in the K-layer data storage unit information in the data storage unit information of the main column cluster of the second partition. For example, M ₀₂ = 2 indicates that level 0 of the data storage unit information of the main column cluster of the second partition includes 2 data storage unit information. The number of data storage unit information included in the k-th data storage unit information in the data storage unit information of the main column cluster of the third partition is the sum of M _k1 and M _k2 . For example, assuming M ₀₁ = 2 and M ₀₂ = 2, level 0 in the data storage unit information of the main column cluster of the third partition includes 4 data storage unit information.

Taking KVDB using the LSM-tree algorithm as an example, the SSTable information stored in the manifest file corresponds to the K-layer SSTable information. The following describes the combination of the data storage unit information of the main column cluster with reference to Table 6, Table 7, and Table 8.

Table 6

Table 6 shows that the data storage unit information of the main column cluster of the first partition includes two layers of data storage unit information, and each layer of data storage unit information in the two-layer data storage unit information includes 2 data storage unit information. Table 6 is a table obtained by taking KVDB using the LSM-tree algorithm as an example. Therefore, the corresponding data storage unit information can also be called SSTable information.

Table 7

Table 7 shows that the data storage unit information of the main column cluster of the second partition includes two layers of data storage unit information, and each layer of data storage unit information in the two-layer data storage unit information includes 2 data storage unit information. Table 7 is a table obtained by taking KVDB using the LSM-tree algorithm as an example. Therefore, the corresponding data storage unit information can also be called SSTable information.

Table 8

Table 8 shows that the data storage unit information of the main column cluster of the third partition includes two layers of data storage unit information, and each layer of data storage unit information in the two-layer data storage unit information includes 4 data storage unit information. Table 8 is a table obtained by taking KVDB using the LSM-tree algorithm as an example. Therefore, the corresponding data storage unit information can also be called SSTable information. It can be seen that the SSTable information of level 0 shown in Table 8 includes the SSTable information of level 0 shown in Table 6 and Table 7, and the SSTable information of level 1 shown in Table 8 includes level 1 shown in Table 6 and Table 7. SSTable information.

The data storage unit information of the main column cluster of the first partition and the data storage unit information of the main column cluster of the second partition are both an ordered sequence prefixed by the partition key. Because the partition is divided based on the range partition, the main column clusters of the two partitions store all the key values between the layers in the unit information without overlapping the ranges. Therefore, the data storage unit information of each layer of the second partition can be directly added to the data storage unit information of the same layer of the first partition, thereby forming the data storage unit information of the main column cluster of a partition. For the convenience of description, the following method of merging information of a layer of data storage unit in one partition directly to data storage unit information of the same layer in another partition is called append merge.

Similar to the data storage unit information of the second column cluster, the value of K _{1 and} the value of K ₂ may be different. In the K values ranging from ₁ and K ₂ are different, the third data storage unit includes a partition information storage unit in addition to K information data outer layer may further include K 'layer data storage unit, wherein K' is The value is max (K ₁ , K ₂ ) -min (K ₁ , K ₂ ), where max (K ₁ , K ₂ ) represents the maximum value of K ₁ and K ₂ , and min (K ₁ , K ₂ ) represents The minimum of K ₁ and K ₂ . In other words, K 'equals K ₁ subtracting the minimum value of K ₁ and K ₂ and the maximum value of K _2. Or the value of K 'is | K ₁ -K ₂ |, that is, the value of K' is the absolute value of the difference between K ₁ and K ₂ . If K _{1 is} greater than K ₂ , the K ′ layer data storage unit is the last K ′ layer data storage unit information of the K ₁ layer data storage unit information of the first partition. If K _{2 is} greater than K ₁ , the K ′ layer data storage unit is the last K ′ layer data storage unit information of the K ₂ layer data storage unit information of the first partition.

For example, suppose K ₁ = 3 and K ₂ = 2, in which case K = 2 and K ′ = 1. In this case, the manifest file of the first partition includes the three-layer SSTable information of the main column cluster, which are the tier-0 SSTable information to the tier-2 SStable information, respectively. The manifest file of the second partition includes the two-layer SSTable information of the main column cluster, which are the tier-0 SSTable information and the tier-1 SStable information, respectively. The target list file includes 3 layers of SSTable information of the main column cluster, which are layer 0 SSTable information to layer 2 SStable information.

The level 0 SSTable information in the target manifest file includes the level 0 SSTable information of the main column cluster in the manifest file of the first partition and the level 0 SSTable information of the main column cluster in the manifest file of the second partition.

The first-level SSTable information in the target manifest file includes the first-level SSTable information of the main column cluster in the first partition's manifest file and the first-level SSTable information of the main column cluster in the second partition's manifest file.

The layer 2 SSTable information in the target manifest file includes the layer 2 SSTable information of the main column cluster in the manifest file of the first partition.

The order of the data storage unit information of the primary / secondary column clusters from the same partition in the combined data storage unit information of the primary / secondary column clusters does not change.

Take the data storage unit information of the main column cluster as an example. As shown in Tables 6, 7, and 8, SSTable f1.1.1 precedes SSTable f1.1.2 before merging. After the merger, SSTable f1.1.1 is still before SSTable f1.1.2.

As shown in Table 8, in the data storage unit information of the main column cluster of the third partition, the SSTable at level 1 of the data storage unit information of the main column cluster of the first partition and the data storage of the main column cluster of the second partition The order of the SSTable of the level 1 of the unit information is: SSTable f1.1.1, SSTable f1.1.2, SSTable f2.1.1, SSTable f2.1.2. In some embodiments, as long as the sequence of data storage unit information of the main column cluster of the same partition does not change. In other words, the merged data storage unit information from the main column cluster of another partition may be located before the data storage unit information of the main column cluster of the same partition. For example, the data storage unit information of the main column cluster of the third partition may also be shown in Table 9.

Table 9

As shown in Table 9, in the data storage unit information of the main column cluster of the third partition, the data storage unit information of the main column cluster of the first partition from level 1 SSTable and the data storage unit of the main column cluster of the second partition The order of the SSTable for the level of information is: SSTable f1.1.1, SSTable f2.1.1, SSTable f1.1.2, SSTable f2.1.2. It can be seen that although SSTable f2.1.1 is located between SSTable f1.1.1 and SSTable f1.1.2, SSTable f1.1.1 is still before SSTable f1.1.2.

The sequence of the data storage unit information of the secondary column cluster is similar to the sequence of the data storage unit information of the main column cluster, and it is unnecessary to repeat them here.

The metadata of the first partition further includes a set of write-ahead log information of the first partition. The metadata of the second partition includes a set of write-ahead log information of the second partition.

The database server 1 may combine the write-ahead log information set of the first partition and the write-ahead log information set of the second partition to generate a write-ahead log information set of the third partition.

The third pre-write log partition information set includes N write-ahead log information, the first pre-write log partition information set includes the N ₁ write-ahead log information of the N write-ahead log information, the second partition The set of write-ahead log information includes N ₂ pieces of write-ahead log information in the N file instructions, where N is a positive integer greater than or equal to 2, N ₁ and N ₂ are positive integers greater than or equal to 1 and N ₁ The sum with N ₂ is N.

In other words, in the above embodiment, the database server 1 only compares the pre-write log information included in the first pre-write log information set in the metadata of the first partition with the second pre-write log information of the second partition metadata. The write-ahead log information included in the set is combined into a third write-ahead log information set within the third metadata information. The database server 1 does not read the N ₁ write-ahead logs indicated by the N ₁ write-ahead log information in the first write-ahead log information set from the first partition, and writes the N ₁ write-ahead logs Into the merged partition. In other words, during the partition merging process, only the write-ahead log information in the metadata information needs to be merged, and there is no need to perform read and write operations on the write-ahead log indicated by the write-ahead log information. The write-ahead log information may include an identifier of the write-ahead log and a time serial number of the write-ahead log. Therefore, the size of the write-ahead log information is usually in the order of KB. The size of the write-ahead log is usually in the order of MB. Therefore, compared to reading and writing the write-ahead log, reading and writing the read-ahead log information can reduce the amount of read and write data, thereby reducing the overhead of the distributed database system.

Taking KVDB using the LSM-tree algorithm as an example, the write-ahead log information can be the WAL file information stored in the manifest file. As described above, the WAL file information includes the identification of the WAL file. Optionally, the time serial number of the WAL file may also be included. WAL file to a serial number identification information includes time and WAL WAL document files as an example, the first pre-write log partition information set may comprise identifying each of the N ₁ WAL files and the N ₁ WAL file time WAL document sequence number, write-ahead log information of the second set may include a time partition identifier SEQ ID N ₂ th WAL WAL files for each file, and the N ₂ th WAL file. Write-ahead log information of the third partition comprises a set of time series of numbers identifying the N ₁ WAL WAL files for each file, and the files in the N ₁ WAL, and the third partition of the pre-write log information set may further include the time sequence number identifies the file and for each WAL WAL th N ₂ N ₂ th file WAL file.

In some embodiments, the order of the first write-ahead log information and the second write-ahead log information in the first write-ahead log information set is the same as the order of the first write-ahead log information and the second write-ahead log information. The sequence in the third write-ahead log information set is the same. The first write-ahead log information and the second write-ahead log information are any two write-ahead log information of N ₁ write-ahead logs included in the write-ahead log information set of the first partition. In other words, in the write-ahead log information set of the first partition, if the first write-ahead log information precedes the second write-ahead log information, then in the write-ahead log information set of the third partition, the The first write-ahead log information is still before the second write-ahead log information.

Similarly, the order of the third write-ahead log information and the fourth write-ahead log information in the write-ahead log information set of the second partition is the same as that of the third write-ahead log information and the fourth write-ahead log information in the first The sequence of the three-part write-ahead log information set is the same. The third write-ahead log information and the fourth write-ahead log information are any two write-ahead log information of the N ₂ write-ahead logs included in the write-ahead log information set of the second partition. In other words, in the write-ahead log information set of the second partition, if the third write-ahead log information precedes the fourth write-ahead log information, then in the write-ahead log information set of the third partition, the The third write-ahead log information is still before the fourth write-ahead log information.

Table 10 shows the combination of WAL file information in the two manifest files.

Table 10

第一分区First partition	第二分区Second partition	第三分区Third partition
f1.w.9，f1.w.8f1.w.9, f1.w.8	f2.w.11，f2.w.7f2.w.11, f2.w.7	f1.w.9，f2.w.11，f1.w.8，f2.w.7f1.w.9, f2.w.11, f1.w.8, f2.w.7

In Table 10, the identification of the WAL file represents the WAL file information, and the sequence of the identification of the WAL file identifies the time serial number of the WAL file. The order of the identification of the WAL file shown in Table 10 is arranged according to the time serial number of the WAL file. The time serial number of the WAL file on the right is lower than the time serial number of the WAL file on the left.

As shown in Table 10, the manifest file 1 of the first partition includes two WAL file information, and the identifiers of the two WAL files are: f1.w.8 and f1.w.9, and f1.w.8's The time serial number is lower than the time serial number of f1.w.9. The manifest file 2 of the second partition includes two WAL file information, the identifiers of the two WAL files are: f2.w.7 and f1.w.11, and the time serial number of f2.w.7 is lower than f1. w.11 time serial number.

As can be seen from Table 10, after the two WAL file information of the first partition and the two WAL file information of the second partition are combined, the WAL file information in the manifest file 3 of the third partition is obtained. The manifest file 3 includes 4 WAL file information. The four WAL file information comes from manifest file 1 and manifest file 2, respectively. In addition, it can be seen from Table 3 that the sequence of the two WAL file information from the manifest file 1 has not changed, and f1.w.8 precedes f1.w.9. Similarly, the sequence of the two WAL file information from manifest file 2 has not changed, f2.w.7 precedes f1.w.11.

311. The database server 1 sends response information to the management server, where the response information includes the current file identifier of the third partition.

312. The management server updates the partition routing table.

Specifically, the management server marks the first partition and the second partition as deleted, and records the left and right boundaries of the third partition and the current file identifier, and the status of the third partition is modified to be normal. The partition routing table includes the mapping relationship between the third partition and the database server 1. The specific implementation may be the mapping relationship between the identifier of the third partition and the address of the database server.

313. The management server sends a partition completion message to the database server 1 and the database server 2. The partition completion message is used to instruct the first partition and the second partition to complete the merge.

The metadata of the third partition includes metadata of the first partition and metadata of the second partition. Therefore, taking the KVDB of the LSM-tree algorithm as an example, because the storage server stores the WAL files, SSTables, and manifest files of the corresponding partitions, the partition consolidation scheme provided by the embodiment of this application, the database server 1 can access the storage according to the metadata of the third partition The WAL files, SSTables, and manifest files of the corresponding partitions of the first and second partitions stored in the server can be implemented without reading and writing data entries of the first and second partitions. The first partition and the second partition are merged, thereby reducing the amount of data read and written during the partition merge and improving the speed of partition merge. In addition, during the process of merging the two partitions, the business write operations of the two partitions will be temporarily frozen until the partition merge is completed. The embodiment of the present application also reduces the freeze time of the business write operations of the partitions.

314. The database server 1 closes the database of the first partition, deletes the current file of the first partition and the metadata of the first partition, sends a successful response to the management server, and returns the pending write request to the rerouted error response. After receiving the reroute error response message, the client sends a request to the management server to update the partition routing table, and then sends the request to the new home database server.

315. The database server 2 closes the database of the second partition, deletes the current file of the second partition and the metadata of the second partition, sends a successful response to the management server, and returns the pending write request to the rerouted error response. After receiving the reroute error response message, the client sends a request to the management server to update the partition routing table, and then sends the request to the new home database server.

316. The management server updates the partition routing table, and deletes records of the first partition and the second partition.

317. The database server 1 starts a layer merging task in the background to reduce the number of layers of metadata of the third partition. Until the merge is completed, the third partition will not participate in the new partition merge.

As mentioned above, in some embodiments, the data server 1 may determine the data storage unit information of the secondary column cluster of the third partition according to the data storage unit information of the target column cluster. The data storage unit information of the second-level column clusters of the third partition includes P-layer data storage unit information, and the first-level data storage of the P-level data unit storage information of the data storage unit information of the second-level column clusters of the third partition The unit information is the data storage unit information of the data storage unit obtained by merging and rearranging the data storage unit corresponding to the 0th-level data storage unit information of the Q-level data storage unit information. Each layer of data storage unit information in the P-layer data unit storage information in the P-layer data unit storage information in the data storage unit information is the first-layer data storage unit information in the Q-layer data storage unit information. The data storage unit information of the data storage unit obtained by merging and rearranging the data storage units corresponding to at least two layers of data storage unit information in the Q-1 layer data storage unit information.

The process in which the data server 1 determines the data storage unit information of the second-level column clusters of the third partition according to the data storage unit information of the target column cluster may be referred to as a level compaction process.

The following describes the layer merging process with reference to FIG. 4.

As shown in FIG. 4, PT1 layer 0 indicates the data storage unit information of the 0th layer in the data storage unit information of the second column cluster of the first partition, and PT2 layer 0 indicates the data storage unit information of the second column of the second partition. In the unit information, the layer 0 data stores unit information. PT3 layer 0 represents the layer 0 data storage unit information in the data storage unit information of the secondary column cluster of the third partition, and so on. Layer 0 indicates that the data storage unit information of the second column cluster of the third partition is obtained according to the data storage unit information of the second column cluster of the first partition and the data storage unit information of the second column cluster of the second partition. The layer 0 data storage unit information in the process, and the layer 1 indicates that the third partition is obtained according to the data storage unit information of the second column cluster of the first partition and the data storage unit information of the second column cluster of the second partition. In the process of storing unit information in the second-level column cluster, the first-level data stores unit information, and so on. real: 30MB, which means that the actual amount of data in the current level is 30MB, max: 400MB, which means that the maximum data amount set for this layer is 400MB.

The layer 0 data storage unit information of the data storage unit information of the second-level column clusters of the first partition and the layer 0 data storage unit information of the data storage unit information of the second-level column clusters of the second partition are added and merged to obtain the target second level. The layer 0 data of the column cluster stores unit information. The layer 1 to layer 3 data storage unit information of the second column cluster data storage unit information of the first partition and the layer 1 to layer 3 data storage unit information of the data storage unit information of the second column cluster of the second partition pass The data storage unit information of layers 1 to 6 of the data storage unit of the target secondary column cluster is superimposed and merged.

As shown in FIG. 4, the layer 0 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 0 data storage unit information of the data storage unit information of the secondary column cluster of the first partition and the second partition. The layer 0 data of the second-level column cluster stores unit information.

The layer 1 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 1 data storage unit information of the data storage unit information of the secondary column cluster of the first partition.

The layer 2 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 1 data storage unit information of the data storage unit information of the secondary column cluster of the second partition.

The layer 3 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 2 data storage unit information of the data storage unit information of the secondary column cluster of the first partition.

The layer 4 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 2 data storage unit information of the data storage unit information of the secondary column cluster of the second partition.

The layer 5 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 3 data storage unit information of the data storage unit information of the secondary column cluster of the first partition.

The layer 6 data storage unit information of the data storage unit information of the target secondary column cluster includes the layer 3 data storage unit information of the data storage unit information of the secondary column cluster of the second partition.

For ease of description, unless otherwise specified, the layer 1 data storage unit information referred to below refers to the layer 1 data storage unit information of the target secondary column cluster data storage unit information. Similarly, the layer 2 data storage unit information refers to the target The layer 2 data of the secondary column cluster stores the unit information, and so on.

For ease of description, the KVDB using the LSM-tree algorithm is also described as an example. Correspondingly, the data storage unit information is SSTable information.

First, the merge sort order is described. Assume that the following content is stored in SSTable1: {Key01-Value, Key10-Value, Key19-Value,}, and that the following content is stored in SSTable2: {Key08-Value, Key12-Value, Key17-Value,}. It is not important whether the data in Value is the same, because it is sorted by Key.

The merge sort process is as follows:

1) Apply for the first space and read the contents of SSTable 1 and SSTable 2 into the first space;

2) Apply for a second space. The size of the second space is the sum of SSTable1 and SSTable2. This second space is used to store the combined SSTable content.

3) Set two pointers, the initial positions are the first data entries of SSTable1 and SSTable2;

4) Compare the data entries in SSTable1 and SSTable2 pointed by the two pointers, select the data entry with a relatively small entry key value and place it in the second space, and move the pointer to the next position, which is stored in the second space The data entry is the merged SSTable;

5) Repeat step 4 until a pointer reaches one of the last data entries in SSTable 1 and SSTable 2;

6) Copy all remaining data entries of another SSTable directly to the end of the merged SSTable;

7), get the sorted content as follows: {Key01-Value, Key08-Value, Key10-Value, Key12-Value, Key17-Value, Key19-Value,};

8) Write the above contents into two new SSTables, SSTable3 and SSTable4:

SSTable 3: {Key01-Value, Key08-Value, Key10-Value}

SSTable 4: {Key12-Value, Key17-Value, Key19-Value}.

The layer merge process includes the following steps:

The first step is to merge and sort all the SSTables indicated by the layer 1 SSTable information and all the SSTables indicated by the layer 2 sstable information, and then rewrite them into the new SSTable, and record the SSTable information corresponding to the newly added SSTable in the layer 2 At the same time, the original SSTable information of layer 1 and layer 2 and the SSTable tag indicated by the original SSTable information are deleted. Assume that all the SSTables indicated by the layer 1 SSTable information and all the SSTables indicated by the layer 2 SSTable information need to be read and written as shown in FIG. 4, the total read and write data amount is 200 MB × 2.

Specifically, it is assumed that Table 11 is an illustration of layer 1 SSTable information and layer 2 SSTable information.

Table 11

It should be noted that, since time series numbers are meaningless for layers 1 and higher, the time series numbers are not included in Tables 11 and 12.

The SSTables corresponding to the layer 1 SSTable information shown in Table 11 are f1a.1.1 and f1a.1.2, and the SSTables corresponding to the layer 2 SSTable information are f2a.1.1 and f2a.1.2. Create new SSTables f3a.1.1 and f3a.1.2, merge and sort the contents of SSTables f1a.1.1 and f2a.1.1 and write them to the newly created SSTables f3a.1.1 and f3a.1.2 (see the previous example of merge and sort). Similarly, create new SSTables f3a.1.3 and f3a.1.4, merge and sort the contents of SSTables f1a.1.2 and f2a.2.2, and write them to the newly created SSTables f3a.1.3 and f3a.1.4. Record the SSTable information corresponding to the newly added SSTable in the manifest file. In this case, the SSTable information recorded in the manifest file can be shown in Table 12.

Table 12

As shown in Table 12, the manifest file only includes newly created SSTable information, and the original SSTable information of Layer 1 and Layer 2 has been deleted. Correspondingly, the SSTables indicated by the SSTable information of the original layer 1 and layer 2 are also deleted.

Through the above merging process, there is no data in layer 1 and the data in layer 2 is doubled.

In the above example, all the SSTables indicated by the layer 1 SSTable information overlap with all the SSTables indicated by the layer 2 SSTable information, so the new SSTable needs to be reordered. When the two SSTables to be merged do not overlap, the SSTable content does not need to be rewritten.

In the second step, after merging and sorting the SSTable at layer 0, rewrite the new SSTable, and record the SSTable information corresponding to the newly added SSTable in the manifest file, and the layer identifier in the SSTable information corresponding to the new SSTable. Is 1. The specific merging and sorting process is similar to step 1, and it is unnecessary to repeat them here. Assume that all SSTables in layer 0 shown in Figure 4 need to read and write, and the total read and write data volume is 30MB × 4.

The third step is to merge and sort all the SSTables indicated by the layer 3SSTable information and all the SSTables indicated by the layer 4SSTable information, and rewrite them into the new SSTable, and record the SSTable information corresponding to the SSTable added by the layer 4 in the manifest file. At the same time, the original SSTable information of layer 3 and layer 4 and the SSTable tag indicated by the original SSTable information are deleted. Layer 3 adds a read-only flag and no longer writes a new SSTable. Assume that all SSTables indicated by the layer 3SSTable information and all SSTables indicated by the layer 4SSTable information shown in Figure 4 need to read and write, and the total read and write data amount is 2GB × 2.

The fourth step is to merge and sort all the SSTables indicated by the layer 5SSTable information and all the SSTables indicated by the layer 6SSTable information, and then rewrite them into the new SSTable, and record the SSTable information corresponding to the SSTable added by the layer 6 in the manifest file At the same time, the original SSTable information of layers 5 and 6 and the SSTable tag indicated by the original SSTable information are deleted. Layer 5 adds a read-only flag and no longer writes a new SSTable. Assume that all the SSTables indicated by the layer 5SSTable information and all the SSTables indicated by the layer 6SSTable information shown in FIG. 4 need to read and write, and the total read and write data amount is 2GB × 2.

The fifth step is to delete layers 3 and 5. The specific operation is as follows: the layer ID of the SSTable information with the layer ID of 4 in the manifest file is changed to 3, and the layer ID of the SSTable information with the layer ID of 6 in the manifest file is changed to 4, so as to delete the original layer 3 and the original layer 5. purpose

In the sixth step, all the SSTables indicated by the SSTable information in the new layer 2 (that is, layer 2 obtained in the first step) and all the SSTables indicated by the SSTable information in the new layer 3 (that is, layer 2 obtained in the fifth step) are sorted. Then, re-write the new SSTable, and record the SSTable information corresponding to the new SSTable in the new layer 3 in the manifest file. At the same time, the original SSTable information of the new layer 2 and new layer 3 and the original SSTable information The indicated SSTable tag is deleted. The new layer 2 is deleted, the layer identifier of the SSTable information with the layer identifier 3 in the manifest file is modified to 2, and the layer identifier of the SSTable information with the layer identifier 4 in the manifest file is modified to 3. Assume that all the SSTables indicated by the SSTable information in the new layer 2 (that is, layer 2 obtained in the first step) and the SSTable information in the new layer 3 (that is, layer 2 obtained in the fifth step) need to be read and written as shown in Figure 4. , The total amount of read and write data is 400MB + 4GB.

Through the above steps 1 to 6, the data storage unit information of the third-level and second-level column clusters and the data storage unit corresponding to the data storage unit information of the third-level and second-level column clusters are obtained.

It can be understood that the expressions of the first step, the second step, the third step, the fourth step, the fifth step, and the sixth step are only for the convenience of distinguishing different steps, and are not a limitation on the order of the steps. From the description of the above six steps, it can be seen that the order of the first step, the third step, and the fourth step can be changed.

Through the layer merging process, the number of layers of the data storage unit can be reduced, which can facilitate querying the data entries stored in the database.

Of course, in some embodiments, the database server 1 may not perform layer merge processing.

The distributed database system provided in the embodiments of the present application may be used to store metadata of a distributed object storage system, metadata of a distributed file system, or metadata of a distributed block storage system.

FIG. 5 is a structural block diagram of a database server according to an embodiment of the present application. As shown in FIG. 5, the database server 500 includes a communication unit 501 and a processing unit 502.

The communication unit 501 is configured to receive a merge instruction sent by a management server, where the merge instruction is used to implement merging a first partition and a second partition into a third partition, wherein the first partition and the second partition are adjacent partitions; The merge instruction includes the identifier of the current file of the first partition and the identifier of the current file of the second partition; the current file of the first partition records the file identifier of the file storing the metadata of the first partition; the second A file identifier of a file storing metadata of the second partition is recorded in a current file of the partition, the first partition runs on the database server, and the second partition runs on another database server.

The processing unit 502 is configured to:

Obtain metadata of the first partition according to the identifier of the current file of the first partition.

Obtaining metadata of the second partition according to the identifier of the current file of the second partition;

Merging metadata of the first partition and metadata of the second partition to generate metadata of the third partition.

In another implementation of the database server 500 shown in FIG. 5, the communication unit 501 is configured to receive a merge instruction sent by the management server, and the merge instruction is used to implement merging the first partition and the second partition into a third partition. Wherein the first partition and the second partition are adjacent partitions; the processing unit 502 is configured to: obtain metadata of the first partition and metadata of the second partition according to the first partition, and merge the first partition and the second partition; The metadata and the metadata of the second partition generate metadata of the third partition.

The database server 500 shown in FIG. 5 can perform various steps performed by the database server 1 shown in FIG. 3. For specific functions and beneficial effects of each unit in the database server 500 shown in FIG. 5, reference may be made to the method shown in FIG. 3, and it is unnecessary to repeat them here.

In a possible implementation manner, the processing unit 502 may be implemented by a processor, and the communication unit 501 may be implemented by a network interface card. In other embodiments, the communication unit 501 may also be implemented by a bus adapter. The specific implementation of the communication unit 501 may support one or more access protocols, for example, an Ethernet message protocol, an Infiniband protocol, and the like, which are not limited in the embodiment of the present invention. In another implementation, the communication unit 501 and the processing unit 502 may also be implemented by software, or both software and hardware.

FIG. 6 is a structural block diagram of a database server according to an embodiment of the present invention. As shown in FIG. 6, the database server 600 includes a processor 601 and a communication interface 602. The processor 601 may be used to process data, control a database server, execute a software program, process data of the software program, and the like. The communication interface 602 is mainly used for communication, for example, communication with a management server in a distributed database system.

In the embodiment of the present application, a circuit having a transmitting and receiving function may be regarded as a communication interface 602 of a database server, and a processor having a processing function may be regarded as a processor 601 of the database server 600. In some embodiments, the communication interface 602 may be implemented by a network interface card. In other embodiments, the communication interface 602 may also be implemented by a bus adapter. The specific implementation of the communication interface 602 may support one or more access protocols, for example, an Ethernet message protocol, an Infiniband protocol, and the like, which are not limited in the embodiment of the present invention. The processing unit may also be called a processor, a processing single board, a processing module, a processing device, and the like.

The processor 601 and the communication interface 602 communicate with each other through an internal connection path, and transfer control and / or data signals

The method disclosed in the foregoing embodiment of the present invention may be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip and has a signal processing capability. In the implementation process, each step of the above method may be completed by using an integrated logic circuit of hardware in the processor 601 or an instruction in the form of software.

The processor described in the embodiments of the present application may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), and a ready-made programmable gate array (field programmable gate array). , FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention may be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or may be performed by using a combination of hardware and software modules in the decoding processor. Software modules can be located in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc. Storage media. The storage medium is located in the memory, and the processor reads the instructions in the memory and completes the steps of the above method in combination with its hardware.

Optionally, in some embodiments, the processor 601 may be a combination of a central processing unit (CPU) and a memory, where the memory may store a method for executing the method executed by the database server 1 in the method shown in FIG. 3 Instructions. The CPU may execute the instructions stored in the memory in combination with other hardware (for example, the communication interface 602) to complete the steps performed by the database server 1 in the method shown in FIG.

An embodiment of the present application further provides a chip, and the chip includes a transceiver unit and a processing unit. The transceiver unit may be an input / output circuit or a communication interface; the processing unit is a processor or a microprocessor or an integrated circuit integrated on the chip. The chip can execute the method executed by the database server 1 in the foregoing method embodiment.

The embodiment of the present application further provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed, the method executed by the database server 1 in the foregoing method embodiment is executed.

The embodiment of the present application further provides a computer program product containing computer instructions, and when the computer instructions are executed, the method performed by the database server 1 in the foregoing method embodiment is executed.

FIG. 7 is a structural block diagram of a management server according to an embodiment of the present application. As shown in FIG. 7, the management server 700 includes a communication unit 701 and a processing unit 702.

The processing unit 702 is configured to create a third partition, and determine to merge the first partition and the second partition into the third partition.

The communication unit 701 is configured to send a merge instruction to the first database server, where the merge instruction is used to implement merging the first partition and the second partition into the third partition, where the first partition and the second partition are Adjacent partitions; the merge instruction includes the identifier of the current file of the first partition and the identifier of the current file of the second partition; the current file of the first partition records a file storing a file storing metadata of the first partition Identification; the current file of the second partition records a file identification of a file storing metadata of the second partition, the first partition runs on the first database server, and the second partition runs on the second database server.

In another implementation of the management server 700, the processing unit 702 is configured to create a third partition, and determine to merge the first partition and the second partition into the third partition.

The communication unit 701 is configured to send a merge instruction to the first database server, where the merge instruction is used to implement merging the first partition and the second partition into the third partition, where the first partition and the second partition are Adjacent partitions; the first partition runs on the first database server, and the second partition runs on the second database server.

The management server 700 shown in FIG. 7 may perform various steps performed by the management server shown in FIG. 3. For specific functions and beneficial effects of each unit in the management server 700 shown in FIG. 7, reference may be made to the method shown in FIG. 3, and it is unnecessary to repeat them here.

In a possible implementation manner, the processing unit 702 may be implemented by a processor, and the communication unit 701 may be implemented by a network interface card. In other embodiments, the communication unit 701 may also be implemented by a bus adapter. The specific implementation of the communication unit 701 may support one or more access protocols, for example, an Ethernet message protocol, an Infiniband protocol, and the like, which are not limited in the embodiment of the present invention.

FIG. 8 is a structural block diagram of a management server according to an embodiment of the present invention. As shown in FIG. 8, the management server 800 includes a processor 801 and a communication interface 802. The processor 801 may be used to process data, control the management server 800, execute software programs, process data of the software programs, and the like. In the embodiment of the present application, a circuit having a transmitting / receiving function can be regarded as the communication interface 802 of the database server, and a processor having a processing function can be regarded as the processor 801 of the database server. For a specific description of the management server 800, reference may be made to the description of the database server 600, and details are not described herein again.

An embodiment of the present application further provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed, the method executed by the management server in the foregoing method embodiment is executed.

An embodiment of the present application further provides a computer program product including computer instructions, and when the computer instructions are executed, the method performed by the management server in the foregoing method embodiment is executed.

Those skilled in the art may realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application is essentially a part that contributes to the existing technology or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several computer instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. The foregoing storage media include: U disks, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or compact discs, and other media that can store computer instructions .

Claims

A partition consolidation method in a distributed database system, characterized in that the distributed database system includes a first database server, a second database server, and a management server, the first database server runs a first partition, and the second The database server runs a second partition, wherein the first partition and the second partition are adjacent partitions; the method includes:

Receiving, by the first database server, a merge instruction sent by a management server, where the merge instruction is used to implement merging the first partition and the second partition into a third partition; the merge instruction includes the first partition An identifier of a current file and an identifier of a current file of the second partition; a file identifier of a file storing metadata of the first partition recorded in the current file of the first partition; and a current file of the second partition A file identifier of a file in which the metadata of the second partition is stored;

Acquiring, by the first database server, metadata of the first partition according to an identifier of a current file of the first partition;

Acquiring, by the first database server, metadata of the second partition according to an identifier of a current file of the second partition;

The first database server combines metadata of the first partition and metadata of the second partition to generate metadata of the third partition.
The method according to claim 1, wherein the metadata of the first partition includes data storage unit information of a secondary column cluster of the first partition, and the metadata of the second partition includes the first partition The data storage unit information of the secondary column cluster of the second partition,

The first database server combining metadata of the first partition and metadata of the second partition to generate metadata of the third partition specifically includes:

Combining, by the first database server, data storage unit information of the secondary column clusters of the first partition and data storage unit information of the secondary column clusters of the second partition to generate data storage unit information of the target secondary column clusters;

The data storage unit information of the second-level column cluster of the third partition is determined according to the data storage unit information of the target second-level column cluster.
The method according to claim 2, characterized in that the first two columns of the tufts of the partitioned data storage means includes information of layer data P 1 storing cell information, wherein P 1 is a positive integer equal to or greater than 2;

The data storage unit information of the second-level column clusters of the second partition includes P 2 layer data storage unit information, where P 2 is a positive integer greater than or equal to 2;

The data storage unit information of the target secondary column cluster includes Q layer data storage unit information, and the Q layer data storage unit information includes data storage unit information of the second column cluster of the first partition and the second partition. The data of the second-level column cluster stores unit information, where:

The one-level data storage unit information of the Q-layer data storage unit information includes the one- level data storage unit information and the information in the P-level one-level data storage unit information included in the data storage unit information of the second-level column cluster of the first partition. two columns of clusters storing data of said second partition layer data units P 2 information comprises the data storage layer of the cell information storage unit information,

Q-1 Q-layer data of the layer data storage unit of the information storage layer of the data storage means includes information of the cell information storage unit of the information data of the first two columns of the cluster comprises a partition layer P 1 of the data storage unit of the information P 2 layer hierarchy data storage unit of the information data in two columns one cluster data storage means of the second partition information or the data stored in the cell information includes a cell information storage, wherein Q is equal to P 1 + P 2 -1.
The method according to claim 3, wherein the 0th layer data storage unit information in the Q layer data storage unit information includes the 0th layer of data storage unit information of the second-level column cluster of the first partition. Data storage unit information and layer 0 data storage unit information of the data storage unit information of the secondary column clusters of the second partition,

The 2 × q-1 layer data storage unit information in the Q layer data storage unit information includes the P layer data in the P 1 layer data storage unit information included in the data storage unit information of the secondary column cluster of the first partition. The q-level data storage unit information storing unit information, and the 2 × q-level data storage unit information in the Q-layer data unit storage information includes the data storage unit information of the second-level column cluster of the second partition. q-th layer data storage unit of the information layer 2 data is stored in the cell information storage unit P layer data information, q = 1, ......, P -1, wherein the value of P is P 1 P 2 and the minimum value of minus 1 .
The method according to claim 3, wherein the 0th layer data storage unit information in the Q layer data storage unit information includes the 0th layer of data storage unit information of the second-level column cluster of the first partition. Data storage unit information and layer 0 data storage unit information of the data storage unit information of the secondary column clusters of the second partition,

The layer Q data storing cell information in the first layer to the second data storage unit of the information layer P-1 data storage unit of the information layer, respectively P 1 data information storage unit column two clusters of the first partition comprises The layer 1 data storage unit information to the layer P-1 data storage unit information in the storage unit information of the P layer data storage unit information;

The P layer data storage unit information to the Q-1 layer data storage unit information in the Q layer data storage unit information are the P 2 layer data included in the data storage unit information of the second column cluster of the second partition. Among the storage unit information, the layer 1 data storage unit information to the layer P-1 data storage unit information in the P layer data storage unit information, where the value of P is the minimum value of P 1 and P 2 minus 1.
The method according to claim 4 or 5, further comprising: determining, by the database server, the secondary column cluster of the third partition according to the data storage unit information of the target secondary column cluster. Data storage unit information, wherein the data storage unit information of the second-level column cluster of the third partition includes P-layer data storage unit information, wherein the data storage unit information of the second-level column cluster of the third partition is P-layer data The first layer data storage unit information in the unit storage information is the data storage unit information of the data storage unit obtained by merging and rearranging the data storage unit corresponding to the 0th layer data storage unit information of the Q layer data storage unit information, Each layer of data storage unit information in the P-layer data unit storage information in the data storage unit information of the second-level column cluster of the third partition is the Q layer The data storage unit corresponding to at least two layers of the data storage unit information in the data storage unit information from the layer 1 data storage unit information to the layer Q-1 layer data storage unit information Data storage unit and rearranging obtained after normalization row information storage means.
The method according to any one of claims 2 to 6, wherein a prefix of an entry key value in each data storage unit information in the data storage unit information of the secondary column cluster is a non-partitioned key value.
The method according to any one of claims 1 to 7, wherein the metadata of the first partition further comprises a set of write-ahead log information of the first partition, and the metadata of the second partition further Including the write-ahead log information set of the second partition, the method further includes:

The database server merges the write-ahead log information set of the first partition and the write-ahead log information set of the second partition to generate the write-ahead log information set of the third partition, wherein the The write log information set includes pre-write log information in the pre-write log information set of the first partition and pre-write log information in the pre-write log information set of the second partition, where N is a positive number greater than or equal to 2. Integer, N 1 and N 2 are positive integers greater than or equal to 1 and the sum of N 1 and N 2 is N.
The method according to any one of claims 1 to 8, wherein the metadata of the first partition further comprises data storage unit information of a main column cluster of the first partition, and The metadata further includes data storage unit information of the main column cluster of the second partition, and the method further includes: the database server merging the data storage unit information of the main column cluster of the first partition and the second partition The data storage unit information of the main column cluster of the third column generates the data storage unit information of the main column cluster of the third partition.
The method according to claim 9, characterized in that the main column of the first cluster storing data partition includes a cell information storage layer data K 1 information units, wherein K 1 is a positive integer equal to or greater than 1,

The data storage unit information of the main column cluster of the second partition includes K 2 layer data storage unit information, where K 2 is a positive integer greater than or equal to 1.

The data storage unit information of the main column cluster of the third partition includes K-layer data storage unit information, wherein the data storage unit information of the main column cluster of the third partition includes k-th layer of K-layer data storage unit information. The data storage unit information includes the k-th data storage unit information in the K-level data storage unit information of the K 1- level data storage unit information and the K-level data storage unit information in the K 2- level data storage unit information. k-level data storage unit information, where K is the minimum value of K 1 and K 2 , where any one of the k-level data storage unit information in the k-th data storage unit information of the K 1- level data storage unit information has an entry key value and the k-th layer data storage layer data K 2 information storage unit according to any one data unit key entry information storage unit of the information do not overlap.
The method according to claim 9 or 10, wherein a prefix of an entry key value in each data storage unit information in the data storage unit information of the main column cluster is a partition key value.
A database server, characterized in that the database server includes:

A communication unit, configured to receive a merge instruction sent by a management server, the merge instruction being used to implement merging a first partition and a second partition into a third partition, wherein the first partition and the second partition are adjacent partitions The merge instruction includes an identifier of a current file of the first partition and an identifier of a current file of the second partition; a file storing metadata of the first partition is recorded in the current file of the first partition The file identifier of the file storing the metadata of the second partition is recorded in the current file of the second partition, the first partition runs on the database server, and the second partition runs on another A database server;

The processing unit is used for:

Obtaining metadata of the first partition according to an identifier of a current file of the first partition;

Obtaining metadata of the second partition according to the identifier of the current file of the second partition;

Merging metadata of the first partition and metadata of the second partition to generate metadata of the third partition.
The database server according to claim 12, wherein the metadata of the first partition includes data storage unit information of a secondary column cluster of the first partition, and the metadata of the second partition includes the The data of the secondary column cluster of the second partition stores unit information,

The processing unit is specifically configured to combine data storage unit information of the secondary column clusters of the first partition and data storage unit information of the secondary column clusters of the second partition to generate a data storage unit of the target secondary column cluster information;

The data storage unit information of the second-level column cluster of the third partition is determined according to the data storage unit information of the target second-level column cluster.
The database server as claimed in claim 13, characterized in that the first two columns of the tufts of the partitioned data storage means includes information of layer data P 1 storing cell information, wherein P 1 is a positive integer equal to or greater than 2;

The data storage unit information of the second-level column clusters of the second partition includes P 2 layer data storage unit information, where P 2 is a positive integer greater than or equal to 2;

The data storage unit information of the target secondary column cluster includes Q layer data storage unit information, and the Q layer data storage unit information includes data storage unit information of the second column cluster of the first partition and the second partition. The data of the second-level column cluster stores unit information, where:

The one-level data storage unit information of the Q-layer data storage unit information includes the one- level data storage unit information and the information in the P-level one-level data storage unit information included in the data storage unit information of the second-level column cluster of the first partition. two columns of clusters storing data of said second partition layer data units P 2 information comprises the data storage layer of the cell information storage unit information,

Q-1 Q-layer data of the layer data storage unit of the information storage layer of the data storage means includes information of the cell information storage unit of the information data of the first two columns of the cluster comprises a partition layer P 1 of the data storage unit of the information P 2 layer hierarchy data storage unit of the information data in two columns one cluster data storage means of the second partition information or the data stored in the cell information includes a cell information storage, wherein Q is equal to P 1 + P 2 -1.
The database server according to claim 14, wherein the 0th-level data storage unit information in the Q-level data storage unit information includes the 0th-level data storage unit information of the second-level column cluster of the first partition. Layer data storage unit information and layer 0 data storage unit information of the second column cluster data storage unit information of the second partition,

The 2 × q-1 layer data storage unit information in the Q layer data storage unit information includes the P layer data in the P 1 layer data storage unit information included in the data storage unit information of the secondary column cluster of the first partition. The q-level data storage unit information storing unit information, and the 2 × q-level data storage unit information in the Q-layer data unit storage information includes the data storage unit information of the second-level column cluster of the second partition. q-th layer data storage unit of the information layer 2 data is stored in the cell information storage unit P layer data information, q = 1, ......, P -1, wherein the value of P is P 1 P 2 and the minimum value of minus 1 .
The database server according to claim 14, wherein the 0th-level data storage unit information in the Q-level data storage unit information includes the 0th-level data storage unit information of the second-level column cluster of the first partition. Layer data storage unit information and layer 0 data storage unit information of the second column cluster data storage unit information of the second partition,

The layer Q data storing cell information in the first layer to the second data storage unit of the information layer P-1 data storage unit of the information layer, respectively P 1 data information storage unit column two clusters of the first partition comprises The layer 1 data storage unit information to the layer P-1 data storage unit information in the storage unit information of the P layer data storage unit information;

The P layer data storage unit information to the Q-1 layer data storage unit information in the Q layer data storage unit information are the P 2 layer data included in the data storage unit information of the second column cluster of the second partition. Among the storage unit information, the layer 1 data storage unit information to the layer P-1 data storage unit information in the P layer data storage unit information, where the value of P is the minimum value of P 1 and P 2 minus 1.
The database server according to claim 15 or 16, wherein the processing unit is further configured to determine the secondary column cluster of the third partition according to the data storage unit information of the target secondary column cluster. Data storage unit information, wherein the data storage unit information of the second-level column cluster of the third partition includes P-level data storage unit information, wherein the data storage unit information of the second-level column cluster of the third partition is P-level data unit The first-level data storage unit information in the storage information is the data storage unit information of the data storage unit obtained by merging and rearranging the data storage unit corresponding to the 0th-level data storage unit information of the Q-level data storage unit information. The layer 2 data storage unit information of the second-level column cluster of the third partition is the layer 2 to layer P-1 data storage unit information. Each layer of data storage unit information is the Q layer data. The data storage unit corresponding to at least two layers of data storage unit information in the layer 1 data storage unit information to the layer Q-1 layer data storage unit information Data storage units obtained after normalization and rearranging information storage unit.
The database server according to any one of claims 13 to 17, wherein a prefix of an entry key value in each data storage unit information in the data storage unit information of the secondary column cluster is a non-partitioned key value .
The database server according to any one of claims 12 to 18, wherein the metadata of the first partition further comprises a set of write-ahead log information of the first partition, and the metadata of the second partition It further includes a pre-write log information set of the second partition, and the processing unit is further configured to combine the pre-write log information set of the first partition and the pre-write log information set of the second partition to generate the first The three-part write-ahead log information set, wherein the third-part write-ahead log information set includes the pre-write log information in the pre-write log information set of the first partition and the pre-write log of the second partition The write-ahead log information in the information set, where N is a positive integer greater than or equal to 2, N 1 and N 2 are positive integers greater than or equal to 1, and the sum of N 1 and N 2 is N.
The database server according to any one of claims 12 to 19, wherein the metadata of the first partition further comprises data storage unit information of a main column cluster of the first partition, and the second partition The metadata further includes data storage unit information of the main column clusters of the second partition, and the processing unit is further configured to merge the data storage unit information of the main column clusters of the first partition and the second partition's The data storage unit information of the main column cluster generates data storage unit information of the main column cluster of the third partition.
The database server as claimed in claim 20, characterized in that the main column of the first cluster storing data partition includes a cell information storage layer data K 1 information units, wherein K 1 is a positive integer equal to or greater than 1,

The data storage unit information of the main column cluster of the second partition includes K 2 layer data storage unit information, where K 2 is a positive integer greater than or equal to 1.

The data storage unit information of the main column cluster of the third partition includes K-layer data storage unit information, wherein the data storage unit information of the main column cluster of the third partition includes k-th layer of K-layer data storage unit information. The data storage unit information includes the k-th data storage unit information in the K-level data storage unit information of the K 1- level data storage unit information and the K-level data storage unit information in the K 2- level data storage unit information. k-level data storage unit information, where K is the minimum value of K 1 and K 2 , where any one of the k-level data storage unit information in the k-th data storage unit information of the K 1- level data storage unit information has an entry key value and the k-th layer data storage layer data K 2 information storage unit according to any one data unit key entry information storage unit of the information do not overlap.
The database server according to claim 20 or 21, wherein a prefix of an entry key value in each data storage unit information in the data storage unit information of the main column cluster is a partition key value.
A database server, characterized in that the database server includes a communication interface and a processor, and the communication interface communicates with the processor; wherein,

The communication interface is configured to receive a merge instruction sent by a management server, where the merge instruction is used to implement merging a first partition and a second partition into a third partition, wherein the first partition and the second partition are related to each other. Adjacent partitions; the merge instruction includes an identifier of the current file of the first partition and an identifier of the current file of the second partition; and the metadata of the first partition is recorded in the current file of the first partition The file identifier of the file; the file identifier of the file storing the metadata of the second partition is recorded in the current file of the second partition, the first partition runs on the database server, and the second partition runs On another database server;

The processor is configured to:

Obtaining metadata of the first partition according to an identifier of a current file of the first partition;

Obtaining metadata of the second partition according to the identifier of the current file of the second partition;

Merging metadata of the first partition and metadata of the second partition to generate metadata of the third partition.
The database server according to claim 23, wherein the metadata of the first partition includes data storage unit information of a secondary column cluster of the first partition, and the metadata of the second partition includes the The data of the secondary column cluster of the second partition stores unit information,

The processor is specifically configured to merge data storage unit information of the second-level column clusters of the first partition and data storage unit information of the second-level column clusters of the second partition to generate a data storage unit of the target second-level column cluster. information;

The data storage unit information of the second-level column cluster of the third partition is determined according to the data storage unit information of the target second-level column cluster.
The database server as claimed in claim 24, characterized in that the first two columns of the tufts of the partitioned data storage means includes information of layer data P 1 storing cell information, wherein P 1 is a positive integer equal to or greater than 2;

The data storage unit information of the second-level column clusters of the second partition includes P 2 layer data storage unit information, where P 2 is a positive integer greater than or equal to 2;

The data storage unit information of the target secondary column cluster includes Q layer data storage unit information, and the Q layer data storage unit information includes data storage unit information of the second column cluster of the first partition and the second partition. The data of the second-level column cluster stores unit information, where:

The one-level data storage unit information of the Q-layer data storage unit information includes the one- level data storage unit information and the information in the P-level one-level data storage unit information included in the data storage unit information of the second-level column cluster of the first partition. two columns of clusters storing data of said second partition layer data units P 2 information comprises the data storage layer of the cell information storage unit information,

Q-1 Q-layer data of the layer data storage unit of the information storage layer of the data storage means includes information of the cell information storage unit of the information data of the first two columns of the cluster comprises a partition layer P 1 of the data storage unit of the information P 2 layer hierarchy data storage unit of the information data in two columns one cluster data storage means of the second partition information or the data stored in the cell information includes a cell information storage, wherein Q is equal to P 1 + P 2 -1.
The database server according to claim 25, wherein the 0th-level data storage unit information in the Q-level data storage unit information includes the 0th-level data storage unit information of the second-level column cluster of the first partition. Layer data storage unit information and layer 0 data storage unit information of the second column cluster data storage unit information of the second partition,

The 2 × q-1 layer data storage unit information in the Q layer data storage unit information includes the P layer data in the P 1 layer data storage unit information included in the data storage unit information of the secondary column cluster of the first partition. The q-level data storage unit information storing unit information, and the 2 × q-level data storage unit information in the Q-layer data unit storage information includes the data storage unit information of the second-level column cluster of the second partition. q-th layer data storage unit of the information layer 2 data is stored in the cell information storage unit P layer data information, q = 1, ......, P -1, wherein the value of P is P 1 P 2 and the minimum value of minus 1 .
The database server according to claim 25, wherein the 0th-level data storage unit information in the Q-level data storage unit information includes the 0th-level data storage unit information of the second-level column cluster of the first partition. Layer data storage unit information and layer 0 data storage unit information of the second column cluster data storage unit information of the second partition,

The layer Q data storing cell information in the first layer to the second data storage unit of the information layer P-1 data storage unit of the information layer, respectively P 1 data information storage unit column two clusters of the first partition comprises The layer 1 data storage unit information to the layer P-1 data storage unit information in the storage unit information of the P layer data storage unit information;

The P layer data storage unit information to the Q-1 layer data storage unit information in the Q layer data storage unit information are the P 2 layer data included in the data storage unit information of the second column cluster of the second partition. Among the storage unit information, the layer 1 data storage unit information to the layer P-1 data storage unit information in the P layer data storage unit information, where the value of P is the minimum value of P 1 and P 2 minus 1.
The database server according to claim 26 or 27, wherein the processor is further configured to determine the secondary column cluster of the third partition according to the data storage unit information of the target secondary column cluster. Data storage unit information, wherein the data storage unit information of the second-level column cluster of the third partition includes P-level data storage unit information, wherein the data storage unit information of the second-level column cluster of the third partition is P-level data unit The first-level data storage unit information in the storage information is the data storage unit information of the data storage unit obtained by merging and rearranging the data storage unit corresponding to the 0th-level data storage unit information of the Q-level data storage unit information. The layer 2 data storage unit information of the second-level column cluster of the third partition is the layer 2 to layer P-1 data storage unit information. Each layer of data storage unit information is the Q layer data. The data storage unit corresponding to at least two layers of the data storage unit information from the layer 1 data storage unit information to the layer Q-1 layer data storage unit information is stored. The data storage unit information of the data storage unit obtained after the rearrangement.
The database server according to any one of claims 24 to 28, wherein a prefix of an entry key value in each data storage unit information in the data storage unit information of the secondary column cluster is a non-partitioned key value .
The database server according to any one of claims 23 to 29, wherein the metadata of the first partition further comprises a set of write-ahead log information of the first partition, and the metadata of the second partition The processor further includes a write-ahead log information set of the second partition, and the processor is further configured to merge the write-ahead log information set of the first partition and the write-ahead log information set of the second partition to generate the first The three-part write-ahead log information set, wherein the third-part write-ahead log information set includes the pre-write log information in the pre-write log information set of the first partition and the pre-write log of the second partition The write-ahead log information in the information set, where N is a positive integer greater than or equal to 2, N 1 and N 2 are positive integers greater than or equal to 1, and the sum of N 1 and N 2 is N.
The database server according to any one of claims 23 to 30, wherein the metadata of the first partition further comprises data storage unit information of a main column cluster of the first partition, and the second partition The metadata further includes data storage unit information of the main column cluster of the second partition, and the processor is further configured to merge the data storage unit information of the main column cluster of the first partition and the second partition's The data storage unit information of the main column cluster generates data storage unit information of the main column cluster of the third partition.
The database server as claimed in claim 31, characterized in that the main column of the first cluster storing data partition includes a cell information storage layer data K 1 information units, wherein K 1 is a positive integer equal to or greater than 1,

The data storage unit information of the main column cluster of the second partition includes K 2 layer data storage unit information, where K 2 is a positive integer greater than or equal to 1.

The data storage unit information of the main column cluster of the third partition includes K-layer data storage unit information, wherein the data storage unit information of the main column cluster of the third partition includes k-th layer of K-layer data storage unit information. The data storage unit information includes the k-th data storage unit information in the K-level data storage unit information of the K 1- level data storage unit information and the K-level data storage unit information in the K 2- level data storage unit information. k-level data storage unit information, where K is the minimum value of K 1 and K 2 , where any one of the k-level data storage unit information in the k-th data storage unit information of the K 1- level data storage unit information has an entry key value and the k-th layer data storage layer data K 2 information storage unit according to any one data unit key entry information storage unit of the information do not overlap.
The database server according to claim 31 or 32, wherein a prefix of an entry key value in each data storage unit information in the data storage unit information of the main column cluster is a partition key value.
A computer-readable storage medium, characterized in that the computer-readable storage medium includes computer instructions, and when a database server runs the computer instructions, the database server executes any one of claims 1 to 11 The method of claim.

.