CN118193541A

CN118193541A - Partition table creation and partition automatic expansion method based on distributed database

Info

Publication number: CN118193541A
Application number: CN202410598259.4A
Authority: CN
Inventors: 汪伟; 高波; 许玲; 金戈; 李兆伟; 徐侃如; 谷帅
Original assignee: Shuyi Technology Beijing Co ltd Wuhan Branch; Shuyi Technology Beijing Co ltd
Current assignee: Shuyi Technology Beijing Co ltd Wuhan Branch; Shuyi Technology Beijing Co ltd
Priority date: 2024-05-15
Filing date: 2024-05-15
Publication date: 2024-06-14

Abstract

The application discloses a partition table creation and partition automatic expansion method based on a distributed database, wherein the distributed database comprises a coordination node CN, a plurality of data nodes DN and a global transaction manager GTM; the CN receives the SQL statement which is sent by the client and used for establishing the partition table, and sends partition information of the partition table to the GTM; the GTM allocates corresponding GOID for the partition table based on the partition information of the partition table, performs local persistent storage on GOID and the partition information of the partition table, and returns GOID to the CN; GOID is the identifier of the partition table common to CN, DN and GTM; the CN sends GOID to each DN so that each DN requests partition information to the GTM to establish a partition table and stores GOID locally; after DN is executed, CN builds partition table and stores GOID locally so as to realize storage of global consistency of partition information of partition table under distributed architecture.

Description

Partition table creation and partition automatic expansion method based on distributed database

Technical Field

The application belongs to the technical field of databases, and in particular relates to a partition table creation and partition automatic expansion method based on a distributed database.

Background

A conventional centralized database (illustrated as Oracle) supports automatic partition features, which automatically creates new partitions when a data node detects that user-inserted data falls outside the partition.

For example, the following table is a definition of a partition table, partition column is col2, the table is built with two initial partitions p1, p2, when the newly inserted data col2 is 2023-02-15, the value is not within the partitions p1, p2, so the storage engine automatically creates a new partition such as p3 less than ('2023-03-02') upon detecting this condition

create table tb(

col1 int,

col2timestamp

) partion by range(col2)

(

partition p1 values less than(‘2023-01-02’),

partition p2 values less than(‘2023-02-02’)

);

The method directly applies a scheme similar to Oracle centralized database partition self-increasing to a distributed database, which can cause the problem that partition metadata of a CN node and a DN node of the distributed database are possibly inconsistent, while the current distributed database partition self-increasing scheme in the industry adopts a scheme that whether to create a new partition is determined according to a currently created partition and a partition in which data is inserted by introducing an additional timing task outside a database SQL engine, and the scheme can cause the complexity of a system to be high. Moreover, the partitions of each data node are different, the partition information CN of each node needs to be recorded, and metadata on the CN expands along with the number of DNs, so that the performance of the CN is reduced.

Disclosure of Invention

Aiming at the defects of the prior art, the application aims to provide a method for creating a partition table and automatically expanding partitions based on a distributed database, which aims to solve the problem that the partition information of a distributed database coordination node and each data node is inconsistent in the prior art.

To achieve the above object, in a first aspect, the present application provides a method for creating a partition table based on a distributed database, where the distributed database includes a coordination node CN, a plurality of data nodes DN, and a global transaction manager GTM;

the partition table creating method comprises the following steps:

step S101, the CN receives an SQL statement for establishing a partition table sent by the client, and sends partition information of the partition table to the GTM based on the SQL statement for establishing the partition table;

Step S102, GTM allocates corresponding GOID for the partition table based on the received partition information of the partition table, performs local persistent storage and management on GOID and the partition information of the corresponding partition table, ensures global consistency of the partition information, and returns GOID to the CN; said GOID is an identifier of said partition table shared by CN, DN and GTM;

Step S103, after receiving GOID, the CN sends GOID and SQL sentences for establishing the partition table to each DN;

step S104, after receiving GOID and SQL sentences, each DN requests GOID corresponding partition information from the GTM, establishes a partition table based on the partition information corresponding to GOID returned by the GTM and the SQL sentences, and simultaneously stores GOID locally;

In step S105, after each DN creates a partition table, the CN creates the partition table based on the partition information corresponding to GOID and simultaneously stores GOID locally.

In an optional example, the CN specifically stores GOID locally by storing a correspondence between GOID and OID in a system table; the OID is an identifier of the partition table which is independently allocated and used by the CN, so that the CN obtains column information of the corresponding partition table based on the OID.

In an optional example, the GTM specifically stores GOID the partition information of the partition table corresponding to the key value storage manner, where GOID is a key, and the partition information of the partition table is a value corresponding to the key.

In an optional example, step S104 further includes locally caching the partition information corresponding to GOID by each DN; step S105 further includes the CN locally caching the partition information corresponding to GOID.

In an alternative example, the distributed database specifically employs PGXC distributed databases.

In a second aspect, the present application provides a partition automatic expansion method based on a distributed database, where the distributed database includes a coordination node CN, a plurality of data nodes DN, and a global transaction manager GTM;

The partition automatic expansion method comprises the following steps:

Step S201, the CN receives a first SQL sentence used for inserting data and sent by a client, so as to convert the first SQL sentence into a second SQL sentence corresponding to the DN, and sends the second SQL sentence to the DN corresponding to the data;

Step S202, analyzing the received second SQL statement by the DN corresponding to the data, searching the corresponding partition according to the value of the data, if not, acquiring the GOID of the partition table corresponding to the data and the partition information corresponding to GOID, determining the information of the newly added partition according to the partition information and the data, and packaging GOID and the information of the newly added partition into a message to be sent to the GTM; the partition table is established based on the partition table creation method based on the distributed database according to the first aspect;

Step S203, after the GTM receives the message, modifying partition information corresponding to GOID in the local persistent storage, and sending GOID and modified partition information corresponding to GOID to CN and each DN so as to update the partition information corresponding to GOID by the CN and each DN;

in step S204, after the DN corresponding to the data updates GOID the partition information corresponding to the data, the data is inserted into the newly added partition.

In an alternative example, the partition information includes the interval of the partition self-increment, and the name, start value, and end value of each partition;

The information of the newly added partition includes the number, name, start value, and end value of the newly added partition.

In a third aspect, the present application provides a distributed database comprising a coordinator node CN, a plurality of data nodes DN and a global transaction manager GTM, wherein:

The CN is used for receiving the SQL statement which is sent by the client and used for establishing the partition table, and sending partition information of the partition table to the GTM based on the SQL statement which is used for establishing the partition table;

The GTM is used for distributing corresponding GOID for the partition table based on the received partition information of the partition table, carrying out local persistent storage and management on GOID and the partition information of the corresponding partition table, guaranteeing the global consistency of the partition information, and returning GOID to the CN; said GOID is an identifier of said partition table shared by CN, DN and GTM;

The CN is used for sending GOID and SQL sentences for establishing the partition table to each DN after receiving GOID;

Each DN is used for requesting GOID corresponding partition information from the GTM after receiving GOID and SQL sentences, establishing a partition table based on the partition information corresponding to GOID returned by the GTM and the SQL sentences, and simultaneously carrying out local storage on GOID;

The CN is configured to, after each DN establishes a partition table, establish the partition table based on the partition information corresponding to GOID and simultaneously store GOID locally.

In an optional example, the CN is further configured to receive a first SQL statement sent by the client and used for inserting data, to convert the first SQL statement into a second SQL statement corresponding to the DN, and to send the second SQL statement to the DN corresponding to the data;

the DN corresponding to the data is also used for analyzing the received second SQL sentence, searching the corresponding partition according to the value of the data, if not, acquiring the GOID of the partition table corresponding to the data and the partition information corresponding to GOID, determining the information of the newly added partition according to the partition information and the data, and packaging the GOID and the information of the newly added partition into a message to be sent to the GTM;

The GTM is further configured to modify partition information corresponding to GOID in the local persistent storage after receiving the message, and send GOID and partition information corresponding to the modified GOID to the CN and each DN, so that the CN and each DN update partition information corresponding to GOID;

the DN corresponding to the data is further used to insert the data into the newly added partition after updating GOID the partition information corresponding to the data.

In a fourth aspect, the present application provides an electronic device comprising: at least one memory for storing a program; at least one processor for executing a memory-stored program, which when executed is adapted to carry out the method described in any one of the possible implementations of the first or second aspect.

In a fifth aspect, the present application provides a computer readable storage medium storing a computer program which, when run on a processor, causes the processor to perform the method described in any one of the possible implementations of the first or second aspects.

In a sixth aspect, the application provides a computer program product which, when run on a processor, causes the processor to perform the method described in any one of the possible implementations of the first or second aspects.

It will be appreciated that the advantages of the third to sixth aspects may be found in the relevant description of the first or second aspects, and are not described here again.

In general, the above technical solutions conceived by the present application have the following beneficial effects compared with the prior art:

The application provides a partition table creation and partition automatic expansion method based on a distributed database, which provides a GOID concept of global object identification on the basis that the distributed database creates a table through a distributed transaction, a GTM is used as a global transaction manager to allocate global GOID for partitions of the partition table, partition information is stored and managed by the GTM, the way of combining GOID with GTM storage partition information is adopted to realize the storage of the global consistency of the partition information of the partition table under the distributed architecture, the accuracy of SQL execution and the high performance of CN are ensured, and further, the global consistency of the partition information can be ensured for a newly added partition.

Drawings

FIG. 1 is a flow chart of a method for creating a partition table based on a distributed database according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a partition automatic expansion method according to an embodiment of the present application;

FIG. 3 is a flow chart of a distributed database implementation provided by the prior art for automatically creating partitions;

FIG. 4 is a flowchart of a client connection CN sending SQL for creating partition tables according to an embodiment of the application;

fig. 5 is an exemplary diagram of a correspondence between GOID of a system table on CN and partition information on GTM provided in an embodiment of the present application;

FIG. 6 is a flow chart of a distributed database implementation for automatically creating partitions provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a distributed database provided by an embodiment of the present application;

Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In the description of the embodiments of the present application, unless otherwise specified, the meaning of "plurality" means two or more, for example, a plurality of data nodes means two or more data nodes, or the like.

First, technical terms involved in the embodiments of the present application will be described.

Partition (P): partitioning. The data of a table is divided into a plurality of blocks, and finally only the table is logically seen, but the bottom layer is composed of files of a plurality of physical blocks. The application discloses a partition self-increment, which mainly refers to a range partition of a time type, wherein the range partition mainly comprises three parts, namely a name (name), a start value (startValue) and an end value (endValue).

For example: partition name p1, start value 2024-01-01, end value 2024-02-01.

Datanode: i.e. the data nodes of the distributed database; each data node independently manages one data slice. In a distributed database system, there are typically multiple data nodes. The aggregate set of data managed by these data nodes constitutes the data set managed by the database system.

Coordinator: i.e. the coordinator node of the distributed database. The node is used as an access layer of SQL and responds to various requests sent by the client; and storing metadata information of the table, and sending SQL sent by the user to the corresponding DN node.

GTM: a global transaction manager (Global Transaction Manager) assigns global transaction numbers to transactions of the distributed database and provides a snapshot of global consistency.

DDL: a data definition language (Data definition language).

CN: coordinator abbreviations.

DN: datanode abbreviations.

OID: an object identifier (object identifier) for uniquely marking a database object, the OID must be different for different data objects (tables, indexes etc) of the same type. There is no necessarily link where the OIDs of the various CNs and DN nodes are independently assigned under the classical PGXC distributed architecture.

GOID: the global object identifier (global object identifier) proposed by the present application is globally unique and incremental compared to the traditional OID, CN and DN sharing GOID. GOID is to enable global consistent storage of certain metadata information under distribution.

System table: the data stores metadata information of the user table, such as metadata information of a column and metadata information of a partition.

PGXC: a distributed database which is open-sourced and is realized based on Postgres open-sourced relational database, PGXC mainly comprises CN, DN, GTM roles.

Next, the technical scheme provided in the embodiment of the present application is described.

If the centralized implementation is directly applied, the following problems exist:

1. Metadata of a table is stored in both coordination nodes and data nodes of the distributed database, the data nodes automatically increase partition, the partition information of CN and DN may be inconsistent, and if the situation occurs, an execution plan generated on CN may be incorrect, so that a problem occurs in SQL execution results.

2. The data in the distributed database is divided into different data nodes according to a certain strategy, the data quantity of the different data nodes is different, if DN is automatically partitioned according to the inserted data, the partitions of each data node are different, how to record the different partitions on each node, if the partition information CN of each node is independently recorded, the metadata on CN is expanded along with the quantity of DN, thus leading to the performance reduction of CN.

In view of the above problems, in the distributed database, the industry mainly adopts a timing task to realize automatic partition. A conventional distributed database (illustrated by doris) typically deploys a plurality of data nodes (datanode) and a plurality of coordinator nodes (coordinator). The coordination node serves as an access layer of the client to provide service to the outside. After the client is connected with the coordination node to create the partition table, the coordination node automatically creates a timing task running in a certain time period for the table, and the timing task checks the insertion condition of data in the partition of the table to judge whether a new partition needs to be created when each time of execution, and the specific flow is as follows:

1. Acquiring partition (partition) information of the partition from the coordination node;

2. acquiring the largest partition Pn of the table;

3. obtaining a partition Pm into which data are inserted currently from the distribution condition of the data node lookup table in the current partition;

4. If n-m < k, then a new partition is automatically created (k is the threshold for creating a partition).

The method for automatically creating the new partition based on the timing task mainly has the following problems:

1. The timing tasks are required to be introduced outside the database engine to finish the creation of the partition, so that the complexity of the system and the cost of operation and maintenance are increased;

2. in the case where the inserted partition key is not in order, normal insertion of data cannot be guaranteed, for example, if a particularly large value of the partition key is inserted, insertion must fail.

In this regard, the present application proposes GOID concept based on the traditional PGXC distributed database through the distributed transaction creation table, the GTM allocates global GOID as global transaction management and stores partition information, and the adoption of GOID +gtm storage partition information realizes the storage of the partition information of the partition table in global consistency under the distributed architecture, so that the automatic creation of the partition can be realized only through the SQL engine itself without depending on external timing tasks.

The embodiment of the application provides a method for creating a partition table based on a distributed database, wherein the distributed database comprises a coordination node CN, a plurality of data nodes DN and a global transaction manager GTM;

Fig. 1 is a flowchart of a method for creating a partition table based on a distributed database according to an embodiment of the present application, where, as shown in fig. 1, the method for creating a partition table includes the following steps:

In step S103, the SQL statement for creating the partition table sent by the CN to each DN is generated by performing equivalent conversion on the SQL statement sent by the CN to the client, and is used for creating the partition table for each DN. In addition, after each DN establishes the partition table, an execution result may be returned to the CN, and after the CN receives the execution result of each DN, a subsequent partition table establishment operation may be performed.

The CN node in the traditional distributed database creates a partition table, broadcasts the partition table to other CNs and DNs, other nodes can generate OIDs for the partition table independently, the OIDs of the nodes are different, when insert is executed, the DN inserts data, the partition is found to be absent, and the local partition is different from the CN partition due to the creation of the partition; the partition storage and modification of the application are uniformly executed by the GTM, and if the GTM is modified, the CN can also obtain the partition information corresponding to GOID because the CN, DN and GTM all use the same GOID, and the global consistency of the partition information can be ensured. Alternatively, the partition is specifically directed to a range partition of the time type.

According to the embodiment of the application, the initial partition is set while the partition table is created, so that the success rate and the efficiency of subsequent data insertion are improved. In addition, the embodiment of the application strips the partition information of the partition table from the metadata on the CN and the DN and stores the partition information into the GTM, the GTM is used as a global transaction manager to ensure the consistency of the partition, and the CN and the DN only store GOID of the partition and the cache of the partition, so that the global uniqueness of the partition information can be ensured, and a foundation is laid for ensuring the global consistency of the partition information for the subsequent newly added partition.

The method provided by the embodiment of the application provides a GOID concept of global object identification on the basis that a distributed database creates a table through a distributed transaction, a GTM is used as a global transaction manager to allocate global GOID for the partition of a partition table, the GTM is used for storing and managing partition information, the way of combining GOID with GTM to store partition information is adopted to realize the storage of the global consistency of the partition information of the partition table under a distributed architecture, the accuracy of SQL execution and the high performance of CN are ensured, and simultaneously, the foundation is laid for the global consistency of the partition information for the subsequent newly added partition.

Based on the above embodiment, in step S101, after receiving the SQL statement sent by the client to establish the partition table, the CN first requests the global transaction number GXID from the GTM, and then performs subsequent operations after receiving the global transaction number returned by the GTM, so as to globally identify and track the transaction.

Based on any one of the above embodiments, the CN specifically stores GOID locally by storing a correspondence between GOID and OID in a system table; the OID is an identifier of the partition table which is independently allocated and used by the CN, so that the CN obtains column information of the corresponding partition table based on the OID.

Note that, the manner in which DN pair GOID is stored locally is similar to CN, and is also the correspondence between stored GOID and OID generated by DN alone.

The partition table information includes column information of the partition table and partition information. Because the DN is not modified once the columns of the partition table are created, only the CN is modified, and the CN broadcasts the modification to the DN, the inconsistent problem does not occur; the partition DN may be modified, but the CN does not perceive that there is an inconsistency, so that the identifier GOID of the partition table shared by the CN, DN and GTM is required to ensure global consistency of partition information.

Based on any one of the above embodiments, step S104 further includes locally caching partition information corresponding to GOID by each DN; step S105 further includes the CN locally caching the partition information corresponding to GOID.

It should be noted that, considering that the performance of the CN or DN will be slow if the CN or DN checks the partition information with the GTM, in the embodiment of the present application, the CN and DN may locally cache the partition information in addition to the GOID of the partition table.

Based on any of the foregoing embodiments, the GTM specifically stores GOID the partition information of the partition table and the corresponding partition table by using a key value storage (K, V structure), where GOID is a key K, and the partition information of the partition table is a value V corresponding to the key.

Based on any of the above embodiments, in order to improve reliability and stability of the distributed database, the distributed database in the embodiment of the present application specifically adopts PGXC (Postgresql-XC) distributed database.

Based on any one of the above embodiments, the embodiment of the present application provides a partition automatic expansion method based on a distributed database, and fig. 2 is a flow chart of the partition automatic expansion method provided by the embodiment of the present application, as shown in fig. 2, where the distributed database includes a coordination node CN, a plurality of data nodes DN, and a global transaction manager GTM;

The partition automatic expansion method comprises the following steps:

Step S202, analyzing the received second SQL statement by the DN corresponding to the data, searching the corresponding partition according to the value of the data, if not, acquiring the GOID of the partition table corresponding to the data and the partition information corresponding to GOID, determining the information of the newly added partition according to the partition information and the data, and packaging GOID and the information of the newly added partition into a message to be sent to the GTM; the partition table is built based on the method for creating the partition table based on the distributed database according to any one of the embodiments;

It will be appreciated that the CN may determine which DN to insert data into according to the distribution key, and then determine which partition to insert data into according to the partition key by the DN. The first SQL statement received by the CN is sent by the client, the CN needs to perform equivalent conversion on the SQL to obtain a second SQL statement executed by the DN, and the second SQL statement is sent to the DN. The first and second SQL statements may be executed by SQL engines on the CN and DN, respectively. The process of updating GOID the partition information corresponding to the DN corresponding to the data achieves the automatic expansion of the partition corresponding to GOID.

It should be noted that, by adopting a mode of automatically creating a new partition by a timing task, no matter how much data is inserted at present, the partition can be created at intervals, so more partitions than actual ones can be additionally created.

In addition, by adopting a mode of automatically creating a new partition by a timing task, under the condition that the inserted partition key is not in order, normal insertion of data cannot be ensured, for example, if a value with a particularly large inserted partition key is inserted, the insertion must fail, but in the embodiment of the application, the automatic creation of the partition is independent of the sequence of the partition key for inserting the data, for example, only the partition 1-3 wants to insert the data of the partition 10, and the newly added partition 4-10 is created.

According to the method provided by the embodiment of the application, when the DN node executes an INSERT statement, if the distribution is found to be absent, the partition needing to be added is calculated, GOID of the partition and partition information needing to be added are sent to the GTM, the GTM modifies the partition information according to GOID and broadcasts the partition information to other CNs and DNs to update local caches, so that the global consistency of the partition is further ensured; in addition, the mechanism for automatically creating the partition based on GOID is completely embedded into the distributed execution framework, the correctness of the mechanism is ensured by distributed transactions, and the partition is expanded to be successful if the data insertion is successful; scheduling is carried out without depending on timing tasks of a third party, and the complexity of the system and the complexity of operation and maintenance are reduced; the automatic creation of the partition is independent of the sequence of partition keys for inserting data, so that the success of data insertion is ensured.

Based on any of the above embodiments, the partition information includes a partition self-increasing interval, and names, start values, and end values of the respective partitions;

It will be appreciated that the number of newly added partitions is one or more, and specifically, calculation is required according to partition information and the inserted data. Considering the situation that a particularly large value of a partition key is inserted, the embodiment of the application can continue to calculate the information of the next newly added partition to ensure the success of inserting data if the newly added partition is calculated to still not cover the value of the data to be inserted.

Wherein the names of the various partitions may be sequentially incremented, e.g., p_sys1, p_sys2 ….

Based on any of the above embodiments, in step S201, after receiving the first SQL statement for inserting data sent by the client, the CN may first request the global transaction number GXID from the GTM, and then perform subsequent operations after receiving the global transaction number returned by the GTM, so as to globally identify and track the transaction.

Based on any of the above embodiments, the flow of the present application is divided into two phases, phase one: the partition information of the partition table under PGXC distributed architecture is stored in a globally consistent manner; stage two: the SQL engine automatically implements the incrementing of the part when the user performs the insert operation.

The phase flow is as follows, wherein 3-7 is proposed and implemented for the present application, and other flows are consistent with the flow of executing the creation table under the classical PGXC distributed architecture. In the process, the application provides GOID a concept of global object identification, GOID is allocated to the partition of the partition table, the GOID is obtained from GTM application, CN and DN no longer store partition information of the partition table, the partition information is only stored on GTM, CN and DN only store GOID of the partition and cache of the partition information corresponding to GOID.

1. The client sends SQL sentences for establishing the partition table to the CN node;

The CN node requests the global transaction number from the GTM;

the CN node sends the partition information to GTM request allocation GOID;

The GTM is divided into an overall unique and incremental integer GOID for the partition, GOID is associated with the partition, stored in a memory and persisted to a local disk, and GOID and corresponding partition information are returned to the CN node;

After receiving GOID of the partition returned by the GTM, the CN node sends the SQL which is subjected to equivalent conversion and establishes the partition table and GOID of the partition to each DN node;

6. each DN node establishes a partition table according to SQL and GOID, and returns an execution result to the CN;

7. The CN node establishes a partition table while storing GOID into the local metadata.

The second stage is as follows, wherein 4), 5) and 6) are proposed and realized for the present application, and other processes are consistent with the process of executing INSERT statement under classical PGXC architecture. In the process, when the DN node executes an INSERT statement, if the distribution is found to be absent, the partition needing to be added is calculated, GOID and partition information of the partition are sent to the GTM, and the GTM modifies the partition information according to GOID and broadcasts the partition information to other CNs and DNs to update local caches.

1) The client sends an SQL statement of insert to the CN node;

2) The CN node analyzes SQL and sends the SQL to the corresponding DN node;

3) The DN node SQL engine executes SQL sentences;

4) The DN node finds the corresponding partition according to the value of the partition column of the inserted data, if the partition does not exist, the global GOID of the partition is obtained from the metadata of the partition table, startValue, endValue and part name of the next partition are calculated according to the interval value of the partition table (the partition names automatically created for the system are generally sequentially increased by p_sys1 and p_sys2 …), and if the value of the inserted data cannot be still included in the newly added partition, startValue, endValue and part name of the next partition are continuously calculated; packaging GOID, startValue, endValue and the part name into a message and sending the message to a GTM;

5) The GTM modifies partition information in the local metadata cache and persists the partition information to a disk according to GOID, broadcasts GOID and corresponding partition information to other CN and DN nodes, and updates the cache of the partition information corresponding to GOID;

6) The DN node inserts data into the new partition;

7) The CN node returns an INSERT execution success message to the client.

Based on any of the above embodiments, fig. 3 is a flowchart of a distributed database implementation provided in the prior art for automatically creating partitions, and as shown in fig. 3, partition information and data distribution of a timed task automatic lookup table are used to determine whether to create a new partition.

Fig. 4 is a flowchart of a client connection CN sending SQL for creating a partition table, where, as shown in fig. 4, a storage space of a GTM opens up a K and V structure for storing partition information, where K is GOID of partition and V is information of partition. The Partition of one table corresponds to a unique GOID, and once the Partition table is successfully created, the Partition of the Partition table allocated by the GTM cannot be changed GOID, so that the Partition global consistent storage is realized, and the corresponding relationship between GOID and Partition of two Partition tables is shown in table 1.

TABLE 1 correspondence between GOID of Partition tables and Partition

Fig. 5 is an exemplary diagram of a correspondence between GOID of a system table on a CN and partition information on a GTM, where, as shown in fig. 5, each cylindrical border in the CN represents a row of record in the system table of the CN, and each cylindrical border in the GTM represents a part of record in the GTM. From the system table record of CN, it can be seen that there are three tables whose OID is classified as 1,2,3. The table GOID =0 of oid=1 indicates that the table has no partition information and is a non-partition table; the table GOID =4 of oid=2 indicates that the table is a partition table, and partition information thereof corresponds to a partition record of GOID =4 on the GTM; the table GOID =5 of oid=3 indicates that the table is a partition table, and partition information thereof corresponds to a partition record of GOID =4 on the GTM.

Fig. 6 is a flowchart of implementing automatic partition creation of a distributed database according to an embodiment of the present application, as shown in fig. 6, after partition information is stored in a global consistency manner, if a DN finds that a partition does not exist only needs to calculate new partition information and send the new partition information to a GTM along with GOID of the partition, the GTM updates a local cache and persists the partition information to a local disk, and broadcasts the partition information to other nodes to update partition information corresponding to GOID in the partition cache.

Based on any of the above embodiments, the following is a specific implementation case, which is divided into two cases, one is an implementation step when a partition table is established, and the other is an implementation procedure that the client connection CN executes an INSERT statement to trigger automatic partition creation.

1. Implementation steps in establishing partition Table

1. The client is connected with the CN node;

2. the client sends an SQL statement for building a partition table to the CN, wherein the SQL statement comprises a partition self-increasing interval;

The CN node sends a message for requesting registration of the part to the GTM;

The GTM takes the local maximum GOID value as GOID of the part, refreshes and persists the memory cache of the part information to a local disk, and returns GOID to the CN; here, the local maximum GOID value is a value added by one on the basis of GOID allocated last time;

Cn sends GOID and equivalent converted table build SQL to each DN node;

6. each DN requests GOID the corresponding partition information to the GTM to locally create a partition table, simultaneously writes GOID into the system table, updates the local GOID and the cache of the partition information, and returns success to the CN;

7. the CN creates a partition table locally and stores GOID into the OID field of the partition system table;

8. The CN returns success to the client.

2. Implementation step of triggering partition automatic creation by client executing SQL statement

1. The client sends INSERT SQL sentences to the CN;

2. The coordination node requests the global transaction number from the GTM and issues the converted SQL statement to the DN;

If the DN finds that the partition where the inserted data is located does not exist when executing SQL, namely, the occasion of creating a new partition is needed, a message is sent to the GTM to request to create the partition, wherein the message contains GOID of the partition and information of the partition;

After receiving the message, the gtm parses the value GOID, retrieves the partition information corresponding to GOID from the local partition cache, if no partition corresponding to GOID is found in the cache, returns GOID an invalid value to the CN, otherwise modifies the local partition, returns to the DN to create a partition successfully, and broadcasts GOID to other CN and DN nodes to notify the nodes to modify the partition corresponding to GOID;

DN inserts data into the partition and returns success to CN;

The cn returns success to the client.

Based on any of the foregoing embodiments, the present application provides a distributed database, and fig. 7 is a schematic diagram of the distributed database provided by the embodiment of the present application, as shown in fig. 7, where the distributed database includes a global transaction manager GTM710, a coordinating node CN720, and a plurality of data nodes DN730, where:

the CN720 is configured to receive an SQL statement for creating a partition table sent by the client, and send partition information of the partition table to the GTM710 based on the SQL statement for creating the partition table;

The GTM710 is configured to allocate a corresponding GOID to the partition table based on the received partition information of the partition table, perform local persistent storage and management on GOID and the partition information of the corresponding partition table, ensure global consistency of the partition information, and return GOID to the CN 720; the GOID is an identifier of the partition table shared by CN720, DN730 and GTM 710;

CN720 is configured to send GOID and the SQL statement for creating the partition table to each DN730 after receiving GOID;

Each DN730 is configured to, after receiving GOID and the SQL statement, request GOID partition information from the GTM710, establish a partition table based on the partition information corresponding to GOID returned by the GTM710 and the SQL statement, and simultaneously store GOID locally;

CN720 is configured to establish a partition table based on the partition information corresponding to GOID after each DN730 establishes the partition table, and simultaneously store GOID locally.

Based on any of the above embodiments, CN720 is further configured to receive a first SQL statement sent by the client and used for inserting data, to convert the first SQL statement into a second SQL statement corresponding to DN730, and to send the second SQL statement to DN730 corresponding to the data;

The DN730 corresponding to the data is further configured to parse the received second SQL statement, search for a corresponding partition according to the value of the data, and if not found, obtain partition information corresponding to GOID and GOID of the partition table corresponding to the data, determine information of a newly added partition according to the partition information and the data, and package GOID and the information of the newly added partition into a message to send to the GTM710;

The GTM710 is further configured to modify partition information corresponding to GOID in the local persistent storage after receiving the message, and send GOID and partition information corresponding to the modified GOID to the CN720 and each DN730, so that the CN720 and each DN730 update the partition information corresponding to GOID;

the DN730 corresponding to the data is further used to insert the data into the newly added partition after updating GOID the partition information corresponding to the data.

It should be understood that the GTM, CN and DN in the above distributed database achieve the principle and technical effects similar to those described in the above method, and will not be repeated here.

Based on the method in the above embodiment, the embodiment of the application provides an electronic device. Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application, as shown in fig. 8, the electronic device may include: processor 810, communication interface (Communications Interface) 820, memory 830, and communication bus 840, wherein processor 810, communication interface 820, memory 830 accomplish communication with each other through communication bus 840. The processor 810 may invoke logic instructions in the memory 830 to perform the methods of the embodiments described above.

Further, the logic instructions in the memory 830 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application.

Based on the method in the above embodiment, the embodiment of the present application provides a computer-readable storage medium storing a computer program, which when executed on a processor, causes the processor to perform the method in the above embodiment.

Based on the method in the above embodiments, an embodiment of the present application provides a computer program product, which when run on a processor causes the processor to perform the method in the above embodiments.

It is to be appreciated that the processor in embodiments of the present application may be a central processing unit (central processing unit, CPU), other general purpose processor, digital signal processor (DIGITAL SIGNAL processor, DSP), application Specific Integrated Circuit (ASIC), field programmable gate array (field programmable GATE ARRAY, FPGA) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. The general purpose processor may be a microprocessor, but in the alternative, it may be any conventional processor.

The steps of the method in the embodiment of the present application may be implemented by hardware, or may be implemented by executing software instructions by a processor. The software instructions may be comprised of corresponding software modules that may be stored in random access memory (random access memory, RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (erasable PROM, EPROM), electrically Erasable Programmable ROM (EEPROM), registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Drive (SSD)), etc.

It will be appreciated that the various numerical numbers referred to in the embodiments of the present application are merely for ease of description and are not intended to limit the scope of the embodiments of the present application.

It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the application and is not intended to limit the application, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the application are intended to be included within the scope of the application.

Claims

1. The method for creating the partition table based on the distributed database is characterized in that the distributed database comprises a coordination node CN, a plurality of data nodes DN and a global transaction manager GTM;

the partition table creating method comprises the following steps:

2. The method for creating partition table according to claim 1, wherein the CN stores GOID locally by storing the corresponding relationship between GOID and OID in a system table; the OID is an identifier of the partition table which is independently allocated and used by the CN, so that the CN obtains column information of the corresponding partition table based on the OID.

3. The method for creating a partition table according to claim 1, wherein the GTM specifically stores GOID partition information of the partition table corresponding to the GTM by using a key value storage manner, wherein GOID is a key, and the partition information of the partition table is a value corresponding to the key.

4. The method of claim 1, wherein step S104 further includes locally caching the partition information corresponding to GOID by each DN; step S105 further includes the CN locally caching the partition information corresponding to GOID.

5. The method for creating partition table according to claim 1, wherein the distributed database is a PGXC distributed database.

6. The partition automatic expansion method based on the distributed database is characterized in that the distributed database comprises a coordination node CN, a plurality of data nodes DN and a global transaction manager GTM;

The partition automatic expansion method comprises the following steps:

Step S202, analyzing the received second SQL statement by the DN corresponding to the data, searching the corresponding partition according to the value of the data, if not, acquiring the GOID of the partition table corresponding to the data and the partition information corresponding to GOID, determining the information of the newly added partition according to the partition information and the data, and packaging GOID and the information of the newly added partition into a message to be sent to the GTM; the partition table is established based on the method for creating the partition table based on the distributed database according to any one of claims 1 to 5;

7. The partition automatic extension method according to claim 6, wherein the partition information includes a partition self-increasing interval, and names, start values, and end values of the respective partitions;

8. A distributed database comprising a coordinator node CN, a plurality of data nodes DN and a global transaction manager GTM, wherein:

9. The distributed database of claim 8, wherein the CN is further configured to receive a first SQL statement sent by the client for inserting data, to convert the first SQL statement into a second SQL statement corresponding to the DN, and to send the second SQL statement to the DN corresponding to the data;

10. An electronic device, comprising:

at least one memory for storing a computer program;

at least one processor for executing the program stored in the memory, the processor being configured to execute the method of creating a partition table according to any one of claims 1 to 5 or the method of automatically expanding partitions according to any one of claims 6 to 7 when the program stored in the memory is executed.