CN115934311A

CN115934311A - Data redistribution method and electronic equipment

Info

Publication number: CN115934311A
Application number: CN202211158038.2A
Authority: CN
Inventors: 刘影
Original assignee: XFusion Digital Technologies Co Ltd
Current assignee: XFusion Digital Technologies Co Ltd
Priority date: 2022-09-22
Filing date: 2022-09-22
Publication date: 2023-04-07

Abstract

The embodiment of the application discloses a data redistribution method and electronic equipment, wherein the method is applied to a coordination node, the coordination node is used for managing a plurality of data nodes, and the method comprises the following steps: determining that a data table with the corresponding data quantity larger than or equal to a first threshold exists in the data tables to be redistributed in the plurality of data nodes; determining the current disk IO resource utilization rate of the plurality of data nodes; according to the current utilization rate of the IO resources of the disk, redistributing the data of the data table to be redistributed to part or all of the data nodes; the current disk IO resource utilization rate is determined according to the current disk IO resource utilization rate of part or all of the plurality of data nodes, and the to-be-redistributed data table is a data table to be redistributed in the data tables included in the plurality of data nodes. According to the embodiment of the application, the current residual disk IO resources can be effectively utilized, so that the data redistribution efficiency can be improved.

Description

Data redistribution method and electronic equipment

Technical Field

The invention relates to the technical field of computers, in particular to a data redistribution method and electronic equipment.

Background

With the rapid development of networks and communication technologies, various industries are impacted by the unprecedented growth trend of mass data. In order to effectively manage, maintain and process the data, the demand of users on the storage capacity of the database is increasing, and the demand on the performance of the database is also increasing. Among them, the distributed cluster database is more and more widely used in practical applications due to its characteristics of high availability, scalability, and load balancing.

Due to the scalability of the distributed cluster database, the number of nodes in the distributed cluster database may be adjusted according to business needs. For example, as data continues to grow, database clusters may be augmented, i.e., new nodes added to the database cluster, in order to support larger-scale data storage and querying. In addition, in order to fully utilize the newly added nodes and realize load balancing of each node, after the new nodes are added into the database cluster, the data stored in the original nodes can be redistributed to all the nodes after capacity expansion.

Meanwhile, how to design a suitable redistribution strategy, and to fully utilize input/output (IO) resources of the disk, improving data redistribution efficiency is a problem that technical personnel pay attention to.

Disclosure of Invention

The embodiment of the application discloses a data redistribution method and electronic equipment, which are used for improving data redistribution efficiency.

A first aspect discloses a data redistribution method that may be applied to a coordinating node, to a module (e.g., a chip) in a coordinating node, and to a logic module or software that may implement all or part of the functionality of a coordinating node. The coordinating node may be configured to manage a plurality of data nodes, and is described below as applied to the coordinating node. The data redistribution method may include: determining that a data table with the corresponding data quantity larger than or equal to a first threshold exists in the data tables to be redistributed in the plurality of data nodes; determining the current utilization rate of the input/output IO resources of the disks of the data nodes; according to the current disk IO resource utilization rate, redistributing the data of the data table to be redistributed to part or all of the data nodes; the current disk IO resource utilization rate is determined according to the current disk IO resource utilization rate of part or all of the data nodes, and the data table to be redistributed is a data table to be redistributed in the data tables included in the data nodes.

In the embodiment of the application, because the data volumes of the data tables are different, the disk IO resources occupied when the data are redistributed are different. Meanwhile, the data table with the data volume smaller than the first threshold value can be a small table, and the disk IO resources are not basically occupied when the data is redistributed. Therefore, when the data table with the corresponding data volume larger than or equal to the first threshold exists in the data table to be redistributed, the coordination node can redistribute the data of the data table to be redistributed to part or all of the data nodes according to the current utilization rate of the input/output IO resources of the disk. Therefore, the current residual disk IO resources can be effectively utilized, and the data redistribution efficiency can be improved.

As a possible implementation, before the determining that, in the plurality of data nodes, there is a data table in the to-be-redistributed data table whose corresponding data amount is greater than or equal to the first threshold, the method may further include: creating M threads, wherein M is an integer greater than or equal to 1; the redistributing the data of the to-be-redistributed data table to some or all of the plurality of data nodes comprises: and redistributing the data of the data table to be redistributed to part or all of the data nodes by the M threads.

In the embodiment of the application, the coordinating node can perform data redistribution on the data table to be redistributed in parallel through the M threads, so that the data redistribution efficiency can be improved.

As a possible implementation manner, the distributing the data of the to-be-redistributed data table to some or all of the data nodes according to the current disk IO resource utilization rate specifically includes: determining a first disk IO resource utilization rate interval corresponding to a first disk IO resource utilization rate, wherein the first disk IO resource utilization rate is the disk IO resource utilization rate at a first moment; determining a first data volume interval corresponding to the first disk IO resource utilization rate interval according to the corresponding relation between the disk IO resource utilization rate interval and the data volume interval of each data table to be redistributed; selecting at least one data table with the data volume in the first data volume interval from the data tables to be redistributed; and distributing the data in the at least one data table to part or all of the plurality of data nodes.

In the embodiment of the application, the coordination node can determine the current corresponding disk IO resource utilization rate interval according to the disk IO resource utilization rates at different moments, then can determine a data volume interval according to the corresponding relation between the disk IO resource utilization rate interval and the data volume interval, and then can select the data table with the data volume in the data volume interval to perform redistribution processing. Therefore, when the current utilization rate of the disk IO resources is low, the residual disk IO resources can be used for redistributing the large meter, and when the current utilization rate of the disk IO resources is high, the residual disk IO resources can be used for redistributing the small meter, so that the disk IO resources can be fully utilized, and the data redistribution efficiency can be improved.

As a possible implementation manner, the distributing the data of the to-be-redistributed data table to some or all of the data nodes according to the current disk IO resource usage rate includes: and under the condition that the current disk IO resource utilization rate is determined to be less than or equal to a second threshold value, distributing the data of the data table to be redistributed to part or all of the data nodes according to the current disk IO resource utilization rate.

In the embodiment of the application, under the condition that the current utilization rate of the disk IO resources is determined to be less than or equal to the second threshold, the coordination node can determine that a certain amount of disk IO resources remain currently and can support data redistribution, so that the coordination node can select a proper data table to redistribute data according to the current utilization rate of the disk IO resources, and thus, influence on user services can be avoided.

As a possible implementation, the method may further include: and stopping redistribution processing of a first data table under the condition that the current disk IO resource utilization rate is determined to be greater than the second threshold, wherein the first data table is a table with the largest data volume in the data redistribution process.

In the embodiment of the application, when it is determined that the current utilization rate of the disk IO resources is greater than the second threshold, the coordination node may stop redistribution processing of the first data table, and release the disk IO resources, so that normal operation of the user service may be ensured, and slow response of the user service or congestion of the user service may be avoided.

As a possible implementation, the method may further include: under the condition that the data quantity of the data tables to be redistributed is smaller than the first threshold value, selecting a fourth data table from the data tables to be redistributed, wherein the fourth data table is any one of the data tables to be redistributed; and redistributing the data of the fourth data table to part or all of the plurality of data nodes.

In the embodiment of the application, the data table with the data volume smaller than the first threshold can be determined as a small table, and the small table basically does not occupy the disk IO resources when the data is redistributed. Therefore, when the data amount of the data tables to be redistributed is smaller than the first threshold, the coordination node can directly select one data table from the data tables to be redistributed for data redistribution without determining the current disk IO resource or redistributing the data according to the current disk IO resource, so that the processing resource of the coordination node can be saved, and the data redistribution efficiency can be improved.

As a possible implementation, the method may further include: under the condition of service congestion, determining a second data table, wherein the second data table is a data table causing the service congestion; the redistribution process of the second data table is stopped.

In the embodiment of the application, under the condition that the user service is congested, the coordination node can temporarily stop processing the data table causing the service congestion, so that the service congestion can be timely solved, and the influence on the user service can be avoided.

A second aspect discloses a data redistribution method that may be applied to a coordinating node, to a module (e.g., a chip) in a coordinating node, and to a logic module or software that may implement all or part of the functionality of a coordinating node. The coordinating node may be configured to manage a plurality of data nodes, and is described below as applied to the coordinating node. The data redistribution method may include: according to the residual available write rate of the disk, redistributing the data of the data table to be redistributed to part or all of the data nodes; the remaining available write rate of the disk is determined according to a write rate threshold and a write rate corresponding to a data table currently undergoing redistribution, and the data table to be redistributed is a data table to be redistributed in the data tables included in the plurality of data nodes.

In the embodiment of the application, the coordinating node may redistribute the data of the to-be-redistributed data table to some or all of the data nodes according to the remaining available write rate of the current disk. In this way, the remaining available write rate of the current disk can be effectively utilized, and thus the data redistribution efficiency can be improved.

As a possible implementation, the method may further include: creating M threads, wherein M is an integer greater than or equal to 1; the redistributing the data of the data table to be redistributed to part or all of the plurality of data nodes comprises: and redistributing the data of the data table to be redistributed to part or all of the data nodes by the M threads.

As a possible implementation, the redistributing the data of the to-be-redistributed data table to some or all of the plurality of data nodes according to the remaining available write rate of the disk comprises: selecting one or more data tables from the data tables to be redistributed according to the residual available write rate of the disk, wherein the write rate corresponding to the one or more data tables is less than or equal to the residual available write rate; and selecting a data table with the maximum corresponding writing rate from the one or more data tables, and redistributing the data of the data table to part or all of the data nodes.

In the embodiment of the application, the coordinating node may screen out the data tables with the write rate less than or equal to the remaining available write rate of the current disk in the data tables to be redistributed, and then may select the table with the maximum write rate in the screened data tables to perform redistribution processing. In this way, the remaining available write rate of the current disk can be maximally utilized, and the redistribution efficiency can be improved.

As a possible implementation, the redistributing the data of the to-be-redistributed data table to some or all of the plurality of data nodes according to the remaining available write rate of the disk includes: and under the condition that the data table with the corresponding data volume larger than or equal to the first threshold exists in the data table to be redistributed, redistributing the data of the data table to be redistributed to part or all of the data nodes according to the residual available write rate of the disk.

In the embodiment of the application, the corresponding writing rates are different due to different data volumes of the data tables. Meanwhile, the data table having the data amount smaller than the first threshold may be a small table, and the corresponding write rate may be 0. Therefore, when the data table with the corresponding data volume larger than or equal to the first threshold exists in the data table to be redistributed, the coordinating node can redistribute the data of the data table to be redistributed to part or all of the data nodes according to the current disk remaining available write rate. In this way, the remaining available write rate of the current disk can be effectively utilized, and thus the data redistribution efficiency can be improved.

As a possible implementation, the method may further include: under the condition that the data quantity of the data tables to be redistributed is smaller than the first threshold value, selecting a fourth data table from the data tables to be redistributed, wherein the fourth data table is any one of the data tables to be redistributed; the data of the fourth data table is redistributed to some or all of the plurality of data nodes.

In this embodiment, the data table with the data amount smaller than the first threshold may be determined as a small table, and the corresponding write rate may be 0. Therefore, when the data amount of the data tables to be redistributed is determined to be smaller than the first threshold, the coordinating node does not need to determine the remaining available write rate of the current disk or redistribute the data according to the remaining available write rate of the current disk, and can directly select one data table from the data tables to be redistributed for data redistribution, so that the processing resources of the coordinating node can be saved, and the data redistribution efficiency can be improved.

A third aspect discloses an electronic device comprising a processor, a memory, and a communication interface for receiving information from and outputting information to other electronic devices other than the electronic device, the processor invoking a computer program stored in the memory to implement the data redistribution method as provided in the first aspect and any one of the possible implementations of the first aspect above, or the data redistribution method as provided in the second aspect and any one of the possible implementations of the second aspect above.

A fourth aspect discloses a computer readable storage medium having stored thereon a computer program or computer instructions which, when executed, implement the data redistribution method as disclosed in the above aspects.

A fifth aspect discloses a chip comprising a processor for executing a program stored in a memory, which program, when executed, causes the chip to perform the data redistribution method disclosed in the above aspects.

As a possible implementation, the memory is located off-chip.

A sixth aspect discloses a computer program product comprising computer program code which, when executed, causes the data redistribution method disclosed in the above aspects to be performed.

It is to be understood that the electronic device provided by the third aspect, the computer-readable storage medium provided by the fourth aspect, the chip provided by the fifth aspect and the computer program product provided by the sixth aspect are all configured to perform the data redistribution method provided in the first aspect and any one of the possible implementations of the first aspect of the present application, or the data redistribution method provided in any one of the possible implementations of the second aspect and the second aspect of the present application. Therefore, the beneficial effects achieved by the method can refer to the beneficial effects in the corresponding method, and are not described herein again.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic view of a scenario of database cluster expansion disclosed in an embodiment of the present application;

FIG. 2 is a schematic diagram of a system architecture disclosed in an embodiment of the present application;

fig. 3 is a schematic flowchart of a data redistribution method disclosed in an embodiment of the present application;

fig. 4 is a schematic diagram illustrating occupation of IO resources of a disk in a redistribution process according to an embodiment of the present application;

FIG. 5 is a schematic flow chart diagram illustrating another data redistribution method disclosed in embodiments of the present application;

fig. 6 is a schematic flowchart illustrating a process of handling user traffic congestion according to an embodiment of the present application;

fig. 7 is a schematic diagram of a software structure of a coordinating node according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.

Detailed Description

The embodiment of the application discloses a data redistribution method and electronic equipment, which are used for improving data redistribution efficiency. The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

For better understanding of the embodiments of the present application, the related art of the embodiments of the present application will be described below.

With the rapid development of network and communication technologies, various industries are impacted by the unprecedented growth trend of mass data. In order to effectively manage, maintain and process the data, the demand of users on the storage capacity of the database is increasing, and the demand on the performance of the database is also increasing. Traditional centralized databases, which include only a single server node, have been unable to meet the storage and computing needs of users. The distributed cluster database generally includes a plurality of server nodes, the plurality of server nodes form a database cluster together, and the server nodes can cooperate with each other.

In an actual business scenario, due to the scalability of the database cluster, the number of nodes in the database cluster can be adjusted according to business requirements. For example, as data continues to grow, database clusters may be augmented, i.e., new database nodes added to the database cluster, in order to support larger-scale data storage and querying. In addition, in order to fully utilize the newly added nodes and realize load balance of each node, after the new database node is added into the database cluster, the data stored in the original node can be redistributed to all the nodes after capacity expansion.

Referring to fig. 1, fig. 1 is a schematic view of a scenario of database cluster expansion disclosed in an embodiment of the present application. As shown in fig. 1, an original database cluster may include two database nodes (i.e., node 1 and node 2), and as data is increased, the disk occupancy of node 1 and node 2 reaches 80%. At this time, in order not to affect the normal operation of the service, the nodes 3 and 4 may be added to the original database cluster through capacity expansion, and the metadata of the old nodes (i.e., the nodes 1 and 2) may be synchronized to the new nodes (i.e., the nodes 3 and 4). The data originally stored in the two nodes (i.e., node 1 and node 2) may then be redistributed to the four nodes (i.e., nodes 1-4 of the expanded database cluster), so that the disk occupancy of a single node may be 40%, thereby relieving the disk pressure of a single node.

For another example, if the amount of data stored in a single node in the database cluster is not large, and the occupancy rate of the disk is small (e.g., 20%), at this time, in order to save storage resources and improve the utilization rate of the storage resources, capacity reduction may be performed on the database cluster, that is, part of the nodes may be removed from the database cluster. And, before removing part of the nodes from the database cluster, the data stored on the nodes can be redistributed to all the nodes after the reduction, so that the load balance of each node can be realized.

It should be noted that, in general, the data stored in the database cluster is one or more data tables, and each data table may include one or more tuples therein. A tuple is an entry in a database, e.g. a row in a row-oriented database, a column in a column-oriented database, or an object in an object-oriented database, usually a tuple is the smallest unit of data read and write. For example, the distributed clustered database may be a relational database in which data may be stored in rows and columns, a series of rows and columns may be regarded as a two-dimensional table, and a row of data is usually a tuple of the relational database.

In a distributed clustered database, for each data table, it may be evenly distributed across each node in the database cluster. The above-described redistribution of data may therefore be understood as a redistribution of each data table stored in the database cluster. For example, suppose that the original database cluster formed by node 1 and node 2 above stores data table 1, which includes 20 tuples, and tuples 1-10 are stored in node 1 and tuples 11-20 are stored in node 2. After the original database cluster is expanded, the data table 1 can be redistributed to the nodes 1-4 through a temporary table creating strategy. Specifically, a temporary table T1 may be created on the database cluster after capacity expansion, and corresponding partitions exist on nodes 1 to 4 in T1, assuming that node 1 stores partition 1 of T1, node 2 stores partition 2 of T1, node 3 stores partition 3 of T1, and node 4 stores partition 4 of T1. Thereafter, the data of data table 1 may be relocated (i.e., written) to temporary table T1, such as tuples 1-5 being written (i.e., stored) into partition 1 of node 1, tuples 6-10 being written into partition 2 of node 2, tuples 11-15 being written into partition 3 of node 3, and tuples 16-20 being written into partition 4 of node 4. After that, the physical files, metadata information, and re-index of the temporary table T1 and the data table 1 may be exchanged. Through the above operations, a redistribution of the tuples 1-20 of node 1 and node 2 to the nodes 1-4 may be achieved. It should be understood that the partitions 1-4 described above can be understood as 4 temporary tables corresponding to the nodes 1-4, respectively, of T1.

It can be understood that, in the data redistribution stage, the input/output (IO) resources of the single-node disk occupied by the data tables with different sizes in the redistribution process are different, the data table with a larger data volume (hereinafter referred to as a large table) generally occupies more IO resources of the disk during data relocation, and the table with a smaller data volume (hereinafter referred to as a small table) generally occupies less IO resources of the disk during data relocation. Furthermore, the redistribution operation for a large table requires more data to be inserted into the temporary table, which generally takes a long time, which may be tens of minutes, one or two hours, or longer, while the redistribution operation for a small table requires less data to be inserted into the temporary table, which generally takes a short time, which may be tens of seconds or several minutes.

In the process of redistributing data tables in the database cluster, generally, in order to improve the efficiency of redistribution, the redistribution of a plurality of data tables can be executed in parallel by a plurality of threads. The multiple threads generally process the data amount of the data table from small to large, that is, the small table is processed first and then the large table is processed, in this way, the situation that the multiple threads simultaneously execute data redistribution of the multiple small tables in the low-peak period of the service may occur, and the situation that the multiple threads simultaneously execute data redistribution of the multiple large tables in the high-peak period of the service may also occur. In the former case, the disk IO resources required for user services and the disk IO resources required for data redistribution are both very small, and the utilization rate of the disk IO resources is very low. In the latter case, the disk IO resources required for user services and the disk IO resources required for data redistribution are both large, and the utilization rate of the disk IO resources is high, at this time, the disk IO resources may be insufficient, which may affect the user services, resulting in slow response of the user services and even congestion of the user services. Therefore, the redistribution strategy can cause the overall efficiency of data redistribution to be low due to the fact that the IO resources of the disk are not effectively utilized.

In order to solve the above problem, in the embodiment of the present application, the utilization rate of the disk IO resources may be monitored, and a data table with a suitable data size may be selected according to the utilization rate of the disk IO resources to perform redistribution. Specifically, when the usage rate of the disk IO resources is low, the remaining disk IO resources may be used to redistribute the large table, and when the usage rate of the disk IO resources is high, the remaining disk IO resources may be used to redistribute the small table. Therefore, the disk IO resources can be fully utilized, the average utilization rate of the disk IO resources in the data redistribution process can be improved, and the overall efficiency of data redistribution can be improved.

For better understanding of the embodiments of the present application, a description will be given below of a system architecture used in the embodiments of the present application.

Referring to fig. 2, fig. 2 is a schematic diagram of a system architecture according to an embodiment of the present disclosure. As shown in fig. 2, the system architecture can include a database cluster 201, a network 202, a first electronic device 203, and a second electronic device 204. Database cluster 201 may include a plurality of server nodes, among other things. For example, as shown in FIG. 2, node 1 2011, node 2 2012, node 3 2013, \8230;, node N2014, N are integers greater than or equal to 2, and each node may be one or more servers.

The database cluster 201 (i.e., node 1-node N), the first electronic device 203, and the second electronic device 204 may be connected to a network 202, respectively, and communicate with each other via the network 202. The N nodes in the database cluster 201 may also communicate with each other via a network.

It should be understood that the nodes in the database cluster 201 may include two classes, namely a Coordinator Node (CN) and a Data Node (DN). The data node is mainly used for storing data, executing data operation instructions (such as data query, data modification and the like) issued by the coordination node, and returning an execution result to the coordination node. For example, for a table in the distributed cluster database, when the table is stored, the tuples in the table may be subjected to hash calculation and then evenly distributed to each data node, so as to ensure that the number of the tuples stored in each data node is the same as much as possible, thereby achieving load balancing.

The coordination node is mainly used for managing and monitoring each data node, can issue data operation instructions to each data node, can monitor the resource use condition (such as processor resources, disk IO resources and disk residual storage capacity) of each data node, and can maintain the whole database cluster to be in a healthy state. In some embodiments, the coordinating node may act as a communication portal for the entire database cluster, and all operations directed to the database (e.g., addition, deletion, modification, querying, etc. of data) may be performed by the coordinating node. The coordinating node may also be referred to as a managing node or a master node. In this embodiment, the node 1 2011 may serve as a coordinating node, and the other nodes (i.e., the node 2 to the node N) may serve as data nodes.

In this embodiment, the electronic devices, such as the first electronic device 203 and the second electronic device 204, may send data operation instructions, for example, data adding, data deleting, data modifying, data querying, and the like, to the database cluster 201. Accordingly, the database cluster 201 may receive a data operation instruction from the electronic device through a coordinating node (e.g., node 1 2011 shown in fig. 2), and then the coordinating node may send the received instruction to other data nodes synchronously or as needed, and after each data node receives the instruction, the data node may perform corresponding processing and return a processing result to the coordinating node, and then the coordinating node may summarize results returned by the data node and return the results to the electronic device that sent the data operation instruction.

In some embodiments, the first electronic device 203 and the second electronic device 204 may be a mobile phone, a tablet computer, a notebook computer, a smart car, a smart wearable device, and the like, which are not limited herein.

In one possible implementation, the coordinating node may also act as a data node.

In this embodiment of the present application, after the database cluster 201 is expanded, the coordinating node may be responsible for a coordinating management function in the data redistribution process, and may execute the redistribution method provided in this embodiment of the present application, so as to improve the data redistribution efficiency, which may specifically refer to the description in the following method embodiment.

It should be noted that the system architecture shown in fig. 2 is only an exemplary one, and is not limited to this. In other embodiments of the present application, the system architecture shown in fig. 2 may include more or fewer devices than those shown, and is not limited to only including the database cluster 201, the network 202, the first electronic device 203, and the second electronic device 204 shown in the figures.

Based on the above system architecture, please refer to fig. 3, and fig. 3 is a flowchart illustrating a data redistribution method according to an embodiment of the present disclosure. As shown in fig. 3, the data redistribution method may include, but is not limited to, the following steps:

301. the coordinating node creates M threads.

Specifically, after the original database cluster is expanded or contracted, i.e., one or more data nodes are added or subtracted, the coordinating node may begin the data redistribution process. The data redistribution can redistribute one or more original tables to all nodes in the database cluster after capacity expansion or capacity reduction, and load balancing can be achieved. Where the original table is a data table stored in the original database cluster, which may include a plurality of tuples.

In the process of data redistribution, generally, in order to improve the efficiency of redistribution, the coordinating node may create M threads and redistribute M data tables in parallel. The M data tables can be selected from the data tables to be redistributed according to the average disk input and output IO resource utilization rate. In addition, since the average disk IO resource usage rates at different times may be different, the coordinating node may select the M data tables multiple times. For example, at a first time, the current average disk IO resource usage rate is the first disk IO resource usage rate, and the coordinating node may select N data tables based on the first disk IO resource usage rate, so that the final M data tables may be obtained through multiple selections. Specifically, for any one of the M threads, it may perform the operations associated with steps 302-312 below. M is an integer greater than or equal to 1, such as 2, 4, 6, etc., and N is an integer greater than or equal to 1 and less than or equal to M.

302. The coordinating node determines whether the data amount of the remaining tables to be redistributed is smaller than the first threshold, and may perform step 309 if it is determined that the data amount of the remaining tables to be redistributed is smaller than the first threshold, and may perform step 303 if it is determined that the data amount of the remaining tables to be redistributed is not smaller than the first threshold.

Since the sizes of the different original tables stored in the original database cluster may be different, the small tables in the original database cluster basically do not occupy the disk IO resources during redistribution, so if the remaining tables to be redistributed (i.e. the tables to be redistributed) are all small tables, then it is not necessary to select the table with the appropriate data size for processing according to the utilization rate of the disk IO resources, and step 309 may be executed. For the non-small tables in the original database cluster, the non-small tables can occupy the disk IO resources with different sizes according to the difference of the data size during redistribution, and generally, the larger the data size of one data table is, the more the disk IO resources are occupied by the data table. Therefore, if the remaining tables to be redistributed are not all small tables, in order to effectively utilize the remaining disk IO resources, a table with a suitable data size may be selected for processing according to the utilization rate of the disk IO resources, and step 303 may be performed first. It should be understood that the remaining tables to be redistributed are tables that have not been redistributed among all tables to be redistributed stored on the original database cluster.

In some embodiments, the coordinating node may determine the table with the amount of data less than the first threshold as a small table. Thus, the coordinating node may determine whether the data amounts of the remaining tables to be redistributed are all less than the first threshold, and if the data amounts of the remaining tables to be redistributed are all less than the first threshold, it may be determined that the remaining tables to be redistributed are all small tables, and step 309 may be performed. If the data amounts of the remaining tables to be redistributed are not all smaller than the first threshold, it may be determined that the remaining tables to be redistributed include non-small tables (i.e. tables with data amounts larger than or equal to the first threshold), which may also be referred to as large tables, and then step 303 may be continued.

The magnitude of the first threshold may be set according to actual conditions. In a possible implementation manner, the first threshold may be set according to the number of data nodes included in the database cluster after capacity expansion and the disk performance of each node. For example, in some embodiments, if the disk performance (including data throughput, input/output per second (IOPS) and the like) of each node is not very different, in this case, a table with a single-node data volume smaller than X1 may be determined as a small table according to the disk performance, that is, a table with a total data volume smaller than (X1 × Q, where Q is the number of data nodes included in the expanded or reduced database cluster) may be determined as a small table. Wherein, the disk performance is different, the value of X1 can be different, the performance is better, and X1 can be larger.

It is understood that in other embodiments of the present application, the coordinating node may also determine whether a table is a small table according to the number of tuples included in the table. For example, a table that includes a number of tuples less than a certain threshold (e.g., 10000) may be determined to be a small table.

303. The coordination node determines whether the average disk IO resource usage rate is greater than a second threshold, where the average disk IO resource usage rate is greater than the second threshold, step 304 may be executed, and where the average disk IO resource usage rate is less than or equal to the second threshold, step 310 may be executed.

Specifically, after the coordination node determines that the data volumes of the remaining tables to be redistributed are not all smaller than the first threshold, the disk IO resource utilization rate of each data node in the database cluster after capacity expansion or capacity reduction may be obtained, and then the average disk IO resource utilization rate of the data nodes may be obtained by calculation. Then, the coordinating node may determine whether the average utilization rate of the disk IO resources is greater than a second threshold, and if the average utilization rate of the disk IO resources is greater than the second threshold, it may determine that the current remaining disk IO resources are insufficient, so the coordinating node may adjust the current redistribution service, reduce the disk IO resources occupied by the redistribution service, and may perform step 304. If the average utilization rate of the disk IO resources is determined to be less than or equal to the second threshold, the coordinating node may determine that a certain amount of disk IO resources remain currently, and may support redistribution, and therefore, step 310 may be executed.

It is understood that, when data is redistributed, the to-be-redistributed data table is redistributed only to all data nodes in the database cluster after capacity expansion or capacity reduction, for example, for capacity expansion shown in fig. 1 (i.e. two data nodes are expanded to four data nodes), the to-be-redistributed data tables on node 1 and node 2 may be redistributed to nodes 1 to 4, and then all nodes 1 to 4 will write the redistributed data. For another example, for the contraction, such as contraction from four data nodes on the right side of fig. 1 to two data nodes on the left side, the to-be-redistributed data tables on the nodes 1-4 can be redistributed to the nodes 1 and 2, and only the nodes 1 and 2 will write the redistributed data. Therefore, the current disk IO resource utilization rate of each data node in the database cluster after capacity expansion or capacity reduction is only considered when the disk IO resource utilization rate is determined. For a data node which does not need to write redistribution data during redistribution, the current disk IO resource utilization rate of the data node does not need to be considered, for example, when the nodes 1 to 4 are reduced, the current disk IO resource utilization rates of the nodes 3 and 4 do not need to be considered, the average disk IO resource utilization rate can be calculated according to the current disk IO resource utilization rates of the nodes 1 and 2, and then a proper data table to be redistributed is selected for processing according to the average disk IO resource utilization rate.

In addition, in a capacity expansion scenario, since no user service exists on a newly added node in a data redistribution process, all the disk IO resources of the newly added node may be used for data redistribution, in some embodiments, when determining the average disk IO resource usage rate, only the current disk IO resource usage rate of an old node may be considered, for example, when the two nodes (i.e., node 1 and node 2) are expanded to four nodes (i.e., node 1 to node 4), only the current disk IO resource usage rates of node 1 and node 2 (i.e., the old node) may be considered when determining the average disk IO resource usage rate, the average disk IO resource usage rate may be calculated according to the current disk IO resource usage rates of node 1 and node 2, and then a suitable data table to be redistributed may be selected for processing based on the average disk IO resource usage rate.

The size of the second threshold may be set according to actual conditions. For example, if the user traffic is frequent under normal conditions, the second threshold may be set to 95%. For another example, if the user traffic is idle under normal conditions, the second threshold may be set to 99%.

304. The coordinating node determines a first data table.

After the coordination node determines that the average disk IO resource usage rate is greater than the second threshold, to avoid the influence of the data redistribution process on the user service, a table (i.e., a first data table) with the largest data volume in the data redistribution process may be determined first. For example, assuming that there are currently three threads to redistribute data table 1, data table 2 and data table 3, respectively, wherein the data size of data table 1 is larger than that of data table 2, and the data size of data table 2 is larger than that of data table 3, in this case, data table 1 may be determined as the first data table.

It should be noted that, in some embodiments, the coordinating node may also determine the other data table as the first data table, for example, the data table with the second largest data amount may be determined as the first data table, which is not limited herein.

305. The coordinating node determines whether the migration of the baseline data of the first data table is completed, and may perform step 306 if the migration of the baseline data of the first data table is completed, and may perform step 308 if the migration of the baseline data of the first data table is not completed.

It is understood that, when the first data table is redistributed, the data of the first data table may be inserted into the temporary table after the temporary table corresponding to the first data table is established. However, there may be an update operation (e.g., addition, deletion, or modification of data) of the first data table during the insertion of data into the temporary table, and at this time, the update data may be recorded. Then, after the baseline data migration of the first data table is completed, the migration of the update data may be performed, that is, the data is deleted from the temporary table according to the recorded update data, or the data in the temporary table is modified, or the data is newly added to the temporary table.

Normally, the baseline data of the first data table will be much more than the updated data. Therefore, after the coordinating node determines the first data table, it may determine whether the baseline data of the first data table has been migrated (i.e. whether the baseline data of the first data table has been completely inserted into the temporary table), if the baseline data of the first data table has been migrated, step 306 may be performed, and if the baseline data of the first data table has not been migrated, step 308 may be directly performed. It is to be understood that the baseline data of the first data table is data that already existed before the first data table was redistributed.

306. The coordinating node exits the thread processing the first data table without clearing the associated resources.

In the case that the migration of the baseline data of the first data table is completed, when the migration of the updated data is not completed, the coordinating node may quit the thread processing the first data table (i.e. end the thread processing the first data table), and may reserve the related resources such as the temporary table, so that the migration of the updated data of the first data table may be completed quickly later.

307. And the coordination node waits for the breakpoint to continue transmitting after the resource is idle.

Because the coordination node reserves the temporary table corresponding to the first data table in step 306, the coordination node may continue to transmit at a breakpoint after determining that the disk IO resource is idle (e.g., the average disk IO resource is less than or equal to the second threshold), that is, may migrate the updated data of the first data table to the temporary table again, so that migration of all data of the first data table may be implemented.

308. And the coordination node quits the thread for processing the first data table and clears the related resources.

Under the condition that the baseline data of the first data table is not migrated, the coordinating node can directly quit the thread for processing the first data table, stop processing the redistribution of the first data table, clean the temporary table and other related resources, and then redistribute the first data table when the IO resources of the disk are idle. It can be understood that after the coordinating node finishes processing the thread of the first data table, the disk IO resources occupied by data redistribution can be reduced.

For example, please refer to fig. 4, where fig. 4 is a schematic diagram illustrating a disk IO resource occupation during a redistribution process disclosed in an embodiment of the present application. As shown in FIG. 4, the total IO resources of a single data node is 100% of disk. However, at a certain moment in the redistribution process, the user service needs to occupy 50% of the disk IO resources of each data node in the original database cluster, the redistribution service needs to occupy 60% of the disk IO resources of each data node in the original database cluster, the 60% of the disk IO resources include 20% of the disk IO resources that the redistribution first data table needs to occupy, and at this time, the coordination node can monitor that the average disk IO resource usage rate is greater than the second threshold and approaches 100%. In order not to affect the user service, the coordinating node may adjust the redistribution service, and may quit processing the thread of the first data table, accordingly, since the redistribution of the first data table is not required, the disk IO resources occupied by the redistribution service may be reduced to 40%, and the average disk IO resource usage rate may be 90%. Therefore, the normal operation of the user service can be ensured by adjusting the redistribution service.

309. And the coordination node selects one from the remaining tables to be redistributed to perform redistribution processing.

Because the redistribution small tables basically do not occupy the disk IO resources, when the coordination node judges that the remaining tables to be redistributed are small tables, one data table (namely, the fourth data table) can be selected from the remaining tables to be redistributed for redistribution processing. After the data table redistribution is complete, step 311 may be performed.

Specifically, the coordinating node may randomly select one from the remaining tables to be redistributed for redistribution, or may select one from the remaining tables to be redistributed for redistribution according to a preset rule. For example, the table with the largest data amount in the remaining tables to be redistributed may be selected for redistribution processing. For another example, the table with the smallest data amount in the remaining tables to be redistributed may be selected for redistribution processing.

310. And the coordination node selects one from the remaining tables to be redistributed according to the average disk IO resource utilization rate to perform redistribution processing.

When the coordination node judges that the average disk IO resource utilization rate is less than or equal to the second threshold, it may be determined that a certain amount of disk IO resources may be left for data redistribution, and at this time, in order to effectively utilize the remaining disk resources, a data table with a proper data size may be selected from the remaining tables to be redistributed according to the average disk IO resource utilization rate to perform redistribution processing. After the data table redistribution is complete, step 311 may be performed.

In some embodiments, K disk IO resource usage intervals may be divided between 0 and the second threshold, where the K disk IO resource usage intervals may respectively correspond to K data volume intervals, and K is an integer greater than or equal to 2. When the current average disk IO resource utilization rate is in the first disk IO resource utilization rate interval, a table with the data volume in the first data volume interval can be preferentially selected from the remaining tables to be redistributed for redistribution, so that the average disk IO resource utilization rate is close to the second threshold but not larger than the second threshold, and the remaining disk IO resources can be fully utilized. The first disk IO resource usage rate interval may be any one of K disk IO resource usage rate intervals, and the first data volume interval may be a data volume interval corresponding to the K data volume intervals.

For example, assuming that the second threshold is 95% and K is 4, [0, 95% ] can be divided into 4 disk IO resource usage intervals. The interval 1 may be [0, 20% ], the interval 2 may be [20%,45% ], the interval 3 may be [45%,70% ], and the interval 4 may be [70%,95% ]. When the average utilization rate of the disk IO resources is located in the interval 1, the coordination node may determine that the disk IO resources used by the current user service and the redistribution service are few, and may preferentially select a table with a data size greater than or equal to D1 from the remaining tables to be redistributed for redistribution processing. When the average disk IO resource utilization rate is located in the interval 2, the coordination node may determine that the disk IO resources used by the current user service and the redistribution service are less, and may preferentially select a table with a data size greater than or equal to D2 and less than D1 from the remaining tables to be redistributed for redistribution. When the average utilization rate of the disk IO resources is located in the interval 3, the coordination node may determine that the disk IO resources used by the current user service and the redistribution service are more, and may preferentially select a table with a data size greater than or equal to D3 and less than D2 from the remaining tables to be redistributed for redistribution processing. When the average utilization rate of the disk IO resources is located in the interval 4, the coordination node may determine that a large number of disk IO resources are used by the current user service and the redistribution service, and may preferentially select a table with a data volume smaller than D3 from the remaining tables to be redistributed for redistribution processing. Wherein D1> D2> D3.

It can be understood that, when the average disk IO resource usage rate is in the first disk IO resource usage rate interval, if there is no table with the data size in the first data size interval in the remaining tables to be redistributed, the coordinating node may select a table with the data size in the second data size interval from the remaining tables to be redistributed for redistribution. The data volume of the second data volume interval is smaller than that of the first data volume interval. For example, when the average disk IO resource utilization rate is in the interval 2, the coordinating node may preferentially select a table with a data size greater than or equal to D2 and smaller than D1 from the remaining tables to be redistributed for redistribution, but at this time, if there is no table with a data size greater than or equal to D2 and smaller than D1 in the remaining tables to be redistributed, the coordinating node may select a table with a data size smaller than D2 from the remaining tables to be redistributed for redistribution (e.g., select a table with a data size greater than or equal to D3 and smaller than D2 for redistribution).

It should be noted that, the larger the value of K is, the smaller the interval of each interval is, and accordingly, the finer the division of each data volume interval is, so that when the coordinating node selects a suitable table from the remaining tables to be redistributed according to the current average utilization rate of the disk IO resources, the table with the more suitable data volume size can be selected, and thus, the disk IO resources can be effectively utilized, and further, the redistribution efficiency can be improved.

311. The coordinating node determines whether there is a table to be redistributed, where step 302 may be performed if it is determined that there is a table to be redistributed, and where step 312 may be performed if it is determined that there is no table to be redistributed.

After the coordinating node processes the previously selected table, it may determine whether there is a table to be redistributed currently, if there is a table to be redistributed, it may continue to select a table from the remaining tables to be redistributed for redistribution, and may perform step 302 again, and if there is no table to be redistributed, it may perform step 312 again.

312. The coordinating node exits the current thread.

When the coordinating node determines that there is no table to redistribute currently, the coordinating node may end the current thread.

It is to be understood that the operations performed by the coordinating node in steps 303-312 above may be operations performed by a thread in the coordinating node. It can also be understood that, in the redistribution process, only a single thread may be used for redistribution, at this time, under the condition that the disk IO resources occupied by the user service are less, in order to reasonably utilize the disk IO resources to improve the redistribution efficiency, redistribution of the large table may be preferentially performed, and under the condition that the disk IO resources occupied by the user service are more, redistribution of the small table may be preferentially performed in order to avoid an influence on the user service.

It should be noted that, in the above flow, the coordinating node only selects an appropriate table from the remaining tables to be redistributed according to the average utilization rate of the disk IO resources to perform processing. However, in other embodiments of the present application, the coordinating node may select from the remaining tables to be redistributed according to more conditions, for example, according to the average disk IO resource utilization rate and the Central Processing Unit (CPU) utilization rate.

It should be further noted that in the above flow, the coordinating node first creates M threads in step 301. However, in other embodiments of the present application, the coordinating node may also perform step 302 after the redistribution is started, and then create a thread to redistribute the table after the table that needs to be redistributed is selected in step 309 or step 310. Then, the coordinating node may determine whether there is any table to be redistributed, if so, step 302 may be performed again, and then when the coordinating node selects the table to be redistributed in step 309 or step 310, a thread may be created to process the table. In this case, the maximum number of threads M may be predetermined in advance. Or, in another possible implementation manner, a thread pool may be created in advance, and after the coordinating node selects the table to be redistributed each time, an idle thread may be selected from the thread pool to redistribute the table.

In the method flow, the coordination node can perform self-adaptive data redistribution according to the data volume of the remaining tables to be redistributed and the average disk IO resource utilization rate, and can effectively utilize the disk IO resources, so that the data redistribution efficiency is improved. Specifically, when the average disk IO resource utilization rate is less than or equal to the second threshold, each thread of the coordination node may select a most suitable table from the remaining tables to be redistributed for processing according to the current average disk IO resource utilization rate, so that the current remaining disk IO resources may be effectively utilized, and the redistribution efficiency may be improved. In addition, in the above process, when it is monitored that the average utilization rate of the disk IO resources is greater than the second threshold, the redistribution service can be adjusted in time, the disk IO resources are released, normal operation of the user service can be ensured, and slow response of the user service or congestion of the user service is avoided.

Based on the above system architecture, please refer to fig. 5, fig. 5 is a flowchart illustrating another data redistribution method according to an embodiment of the present disclosure. As shown in fig. 5, the data redistribution method may include, but is not limited to, the following steps:

501. the coordinating node creates M threads.

Step 501 is the same as step 301, and reference may be made to the related description in step 301.

502. The coordinating node determines whether the data amounts of the remaining tables to be redistributed are all smaller than the first threshold, and in case of determining that the data amounts of the remaining tables to be redistributed are all smaller than the first threshold, step 505 may be executed, as in step 309 above, and in case of determining that the data amounts of the remaining tables to be redistributed are not all smaller than the first threshold, step 503 may be executed.

Since the original tables stored in the original database cluster may have different sizes, for the small tables in the original database cluster, the write rate at a single node is small during redistribution, the write time is short, and the write rate at a single node during redistribution of the small tables may be defaulted to be 0, so if the remaining tables to be redistributed are small tables, step 505 may not be performed without selecting one from the remaining tables to be redistributed according to the currently remaining available write rate for redistribution processing. For the non-small tables in the original database cluster, the writing rate of the non-small tables in the original database cluster can be different at a single node according to the difference of the data size, and generally, the larger the data size of one data table is, the larger the writing rate of the data table in the single node is. Therefore, if the remaining tables to be redistributed are not all small tables, in order to improve the data redistribution efficiency, one of the remaining tables to be redistributed may be selected for redistribution processing according to the current remaining available write rate, and step 503 may be performed first. It should be understood that the write rate is the rate at which the data table to be redistributed is written to the disks of the individual data nodes.

The specific manner of determining the small table in step 502 is the same as that in step 302, and reference may be made to the related description in step 302.

503. The coordinating node calculates the current remaining available write rate.

After the coordinator node determines that the remaining tables to be redistributed are not all small tables, in order to improve the redistribution efficiency, the coordinator node may calculate the current remaining available write rate, i.e., the remaining write rate available for data redistribution of a single node, or the average of the remaining write rates available for data redistribution of all data nodes in the database cluster after expansion or contraction. The coordination node may subtract the sum of the write rates of the tables in which the other threads are performing data redistribution from the write rate threshold value to obtain the current remaining available write rate. The write rate threshold may be a write rate at which a single data node (i.e., a disk of the single data node) may be used for data redistribution, which may be less than or equal to an upper write rate limit of the single data node (i.e., a maximum write rate that the single node is capable of achieving).

For example, assuming a single node write rate cap of 500 megabytes per second (MB/s), a single node write rate threshold may be 250MB/s. At this time, the coordinating node may create 4 threads (i.e., thread 1, thread 2, thread 3, and thread 4) for data redistribution in parallel, where thread 1, thread 2, and thread 3 are performing data table 1, data table 2, and data table 3 redistribution respectively, the write rate of data table 1 at a single data node is 100MB/s, the write rate of data table 2 at a single data node is 70MB/s, and the write rate of data table 3 at a single data node is 50MB/s. At this time, the coordinating node may subtract the write rates already occupied by data table 1, data table 2 and data table 3 from the write rate threshold respectively, and may obtain the current remaining available write rate of 30 (i.e. 250-100-70-50) MB/s.

It is understood that, in the embodiment of the present application, the coordinating node may calculate or test in advance the write rate of each table to be redistributed in a single thread at a single data node without interference of other user services in a new cluster (i.e., a database cluster after expansion or contraction), and then may save it.

504. And the coordination node selects one from the remaining tables to be redistributed according to the current remaining available write rate to perform redistribution processing.

In order to improve the overall redistribution efficiency, after the coordination node calculates the current remaining available write rate, a most suitable table can be selected from the remaining tables to be redistributed according to the current remaining available write rate to perform redistribution processing. Specifically, the coordinating node may first screen out the tables with the write rate less than or equal to the current remaining available write rate in the remaining tables to be redistributed, and then may select the table with the largest write rate among the screened tables to perform redistribution processing. In this way, the write rate threshold can be effectively utilized, and the redistribution efficiency can be improved.

For example, reference may be made to the above example in step 503, where the sum of the write rates of data Table 1, data Table 2, and data Table 3 is 220MB/s, and the current remaining available write rate is 30MB/s. The remaining data tables to be redistributed include data table 4, data table 5, data table 6, data table 7. The write rate of the data table 4 is 10MB/s, the write rate of the data table 5 is 25MB/s, the write rate of the data table 6 is 15MB/s, and the write rate of the data table 7 is 0MB/s. At this time, the coordinator node may determine that the write rates of the data tables 4-7 are all less than the current remaining available write rate, but the write rate of the data table 5 is greater than the write rates of the data tables 4, 6, and 7, and thus, the thread 4 of the coordinator node may perform redistribution processing on the data table 5, which may maximize the utilization of the current remaining available write rate.

505. And the coordination node selects one from the remaining tables to be redistributed to perform redistribution processing.

Step 505 is the same as step 309, and reference may be made to the related description in step 309, which is not described herein again.

506. The coordinating node determines whether there is a table to be redistributed, and may perform step 502 if it is determined that there is a table to be redistributed, and may perform step 507 if it is determined that there is no table to be redistributed.

The coordinating node may determine whether there are tables to be redistributed, and if there are tables to be redistributed, may continue to select one table from the remaining tables to be redistributed for redistribution, may perform step 502 above again, and if there are no tables to be redistributed, may perform step 507.

507. The coordinating node exits the current thread.

It is understood that in steps 502-507, the operation performed by the coordinating node may be an operation performed by a thread in the coordinating node.

It can be understood that when the coordinating node executes the redistribution method corresponding to fig. 3 or fig. 5, a situation that the user traffic is congested may occur during the data redistribution process, especially during the user traffic peak period. For example, a certain data table may be locked for a short time during redistribution, and some operations of the user on the table (such as an update operation on the data table) may be blocked, which may result in congestion due to abnormal user traffic. Therefore, the coordinating node may monitor the congestion condition of the user traffic, and if it is monitored that the user traffic is congested, may perform the following related steps as shown in fig. 6:

601. and the coordination node diagnoses a second data table causing user service congestion through a kernel instruction.

When the coordination node monitors that the user service is congested, the coordination node can diagnose a second data table causing the user service congestion through a kernel instruction. The second data table may be a table currently undergoing redistribution.

602. The coordinating node determines whether the data amount of the second data table is smaller than a first threshold, and if so, may perform step 606, and if not, may perform step 603.

After the coordinating node determines the second data table, the coordinating node may determine whether the data size of the second data table is smaller than the first threshold, and if the data size of the second data table is smaller than the first threshold, the second data table may be determined to be a small table, the redistribution of the current second data table may be suspended directly, and step 606 may be executed. If the data amount of the second data table is greater than or equal to the first threshold, the coordinating node may determine that the second data table is a non-small table, and step 603 may be performed in order to improve the overall redistribution efficiency.

In some embodiments, the coordinating node may perform step 603 directly without performing step 602.

603. The coordinating node determines whether the migration of the baseline data of the second data table is completed, and in the case of the migration, step 604 may be executed, and in the case of the migration not being completed, step 606 may be executed.

604. The coordinating node exits the thread processing the second data table without clearing the associated resources.

605. And the coordination node waits for the breakpoint to continue transmitting after the resource is idle.

606. And the coordination node quits the thread for processing the second data table and clears the related resources.

Steps 603-606 are similar to steps 305-308 described above, and reference may be made to the associated description of steps 305-308 described above.

It should be noted that, the related information (i.e. the same information or similar information) and the related description in the different embodiments described above may be referred to each other.

It should be understood that the above-mentioned processing flows are illustrated in fig. 3, fig. 5, and fig. 6 by taking the coordination node as an execution subject of the interaction schematic as an example, but the application is not limited to the execution subject of the interaction schematic. For example, the coordinating node in fig. 3, fig. 5 and fig. 6 may also be a chip, a chip system, or a processor supporting the coordinating node to implement the method, and may also be a logic module or software capable of implementing all or part of the functions of the coordinating node.

Based on the above system architecture, please refer to fig. 7, and fig. 7 is a schematic diagram of a software structure of a coordinating node according to an embodiment of the present disclosure. Coordinating node 700 may include, among other things, redistribution tool 701 and resource monitoring platform 702. Communication may be between redistribution tool 701 and resource monitoring platform 702.

Redistribution tool 701 and resource monitoring platform 702 may be two software modules deployed in a coordinating node. Resource monitoring platform 702 may monitor the various data nodes in the database cluster, including monitoring resource usage of the various data nodes. For example, CPU utilization, memory utilization, disk IO resource utilization, disk remaining storage capacity, and the like.

The redistribution tool 701 may be responsible for a coordination management function in a data redistribution process, in this embodiment of the present application, the redistribution tool 701 may obtain a disk IO resource usage rate through the resource monitoring platform 702, and then may select an appropriate table according to the disk IO resource usage rate for processing. In some embodiments, the redistribution tool 701 may perform operations performed by the coordination node in the method embodiments shown in fig. 3, fig. 5, and fig. 6, which may refer to the related description above and are not described in detail herein.

The software configuration shown in fig. 7 is merely an example, and is not limited thereto. In other embodiments of the present application, the software architecture shown in FIG. 7 may include more or fewer software modules than those shown.

Based on the above system architecture, please refer to fig. 8, and fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. Among other things, the electronic device 800 may include: a processor 801, a communication interface 802, and a memory 803. The processor 801, communication interface 802, and memory 803 may be interconnected or interconnected by a bus 804.

Illustratively, the memory 803 is used for storing computer programs and data of the electronic device 800, and the memory 803 may include, but is not limited to, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), and the like. The communication interface 802 is used to enable the electronic device 800 to communicate, such as to receive or transmit data.

Illustratively, the processor 801 may be a CPU, complex programmable logic device, general purpose processor, digital signal processor, application specific integrated circuit, field programmable gate array or other programmable logic device, transistor logic device, hardware component, or any combination thereof. A processor may also be a combination of computing functions, e.g., a combination of one or more microprocessors, a digital signal processor and a microprocessor, or the like.

In one embodiment, the electronic device 800 may be a data node, and may be configured to store data, execute data operation instructions (such as data query, data modification, and the like) issued by the coordinating node, and return an execution result to the coordinating node.

In an embodiment, the electronic device 800 may be a coordinating node, and the processor 801 may be configured to read a program stored in the memory 803, and execute operations performed by the coordinating node in the method embodiments shown in fig. 3, fig. 5, and fig. 6, which may refer to the above description and are not described in detail herein.

It should be noted that the electronic device 800 shown in fig. 8 is only one implementation manner of the embodiment of the present application, and in practical applications, the electronic device 800 may further include more or less components, which is not limited herein.

The embodiment of the application also discloses a computer readable storage medium, wherein instructions are stored on the storage medium, and the instructions execute the method in the embodiment of the method when executed.

The embodiment of the application also discloses a computer program product comprising instructions, and the instructions are executed to execute the method in the embodiment of the method.

It should be apparent that the above-described embodiments are only some of the embodiments of the present application, and not all of the embodiments. Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art can explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application. The terms "first," "second," "third," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between different objects and not necessarily for describing a particular order. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process may comprise a sequence of steps or elements, or may alternatively comprise steps or elements not listed, or may alternatively comprise other steps or elements inherent to such process, method, article, or apparatus. It is to be understood that the equal sign of the above condition judgment may be greater than one end or less than one end, for example, the above condition judgment for a threshold value being greater than, less than or equal to, or the condition judgment for the threshold value being greater than, less than or equal to may be changed, and is not limited herein. Specifically, for the first threshold value, the table smaller than the first threshold value is determined as a small table, and the table greater than or equal to the first threshold value is determined as a non-small table. However, in other embodiments of the present application, tables having a data amount less than or equal to the first threshold may be determined as small tables, and tables greater than the first threshold may be determined as non-small tables.

It is to be understood that only some, but not all, of the material pertinent to the present application is shown in the drawings. It should be understood that some example embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

As used in this specification, the terms "component," "module," "system," "unit," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, or software in execution. For example, a unit may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a distribution between two or more computers. In addition, these units can execute from various computer readable media having various data structures stored thereon. The units may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., from a second unit of data interacting with another unit in a local system, distributed system, and/or across a network.

The above-mentioned embodiments, objects, technical solutions and advantages of the present application are further described in detail, it should be understood that the above-mentioned embodiments are only examples of the present application, and are not intended to limit the scope of the present application, and any modifications, equivalent substitutions, improvements and the like made on the basis of the technical solutions of the present application should be included in the scope of the present application.

Claims

1. A data redistribution method is applied to a coordination node, wherein the coordination node is used for managing a plurality of data nodes, and the method comprises the following steps:

determining that a data table with the corresponding data quantity larger than or equal to a first threshold exists in the data tables to be redistributed in the plurality of data nodes;

determining the current utilization rate of the input/output IO resources of the disks of the data nodes;

according to the current utilization rate of the disk IO resources, redistributing the data of the data table to be redistributed to part or all of the data nodes;

the current disk IO resource utilization rate is determined according to the current disk IO resource utilization rate of part or all of the data nodes in the plurality of data nodes, and the data table to be redistributed is a data table to be redistributed in the data tables included in the plurality of data nodes.

2. The method of claim 1, wherein before the determining that there is a data table in the plurality of data nodes whose corresponding amount of data is greater than or equal to the first threshold among the data tables to be redistributed, the method further comprises:

creating M threads, wherein M is an integer greater than or equal to 1;

the redistributing the data of the to-be-redistributed data table to part or all of the plurality of data nodes comprises:

redistributing the data of the data table to be redistributed to part or all of the data nodes through the M threads.

3. The method according to claim 1 or 2, wherein the distributing data of the to-be-redistributed data table to some or all of the plurality of data nodes according to the current disk IO resource utilization rate specifically comprises:

determining a first disk IO resource utilization rate interval corresponding to a first disk IO resource utilization rate, wherein the first disk IO resource utilization rate is the disk IO resource utilization rate at a first moment;

determining a first data volume interval corresponding to the first disk IO resource utilization rate interval according to the corresponding relation between the disk IO resource utilization rate interval and the data volume interval of each data table to be redistributed;

selecting at least one data table with the data volume in the first data volume interval from the data tables to be redistributed;

and distributing the data in the at least one data table to part or all of the plurality of data nodes.

4. The method according to claim 1 or 2, wherein the distributing the data of the to-be-redistributed data table to some or all of the plurality of data nodes according to the current disk IO resource usage rate comprises:

and under the condition that the current utilization rate of the disk IO resources is smaller than or equal to a second threshold value, distributing the data of the data table to be redistributed to part or all of the data nodes according to the current utilization rate of the disk IO resources.

5. The method of claim 4, further comprising:

and under the condition that the utilization rate of the IO resources of the current disk is determined to be greater than the second threshold value, stopping redistribution processing of a first data table, wherein the first data table is a table with the largest data volume in the data redistribution process.

6. The method according to any one of claims 1-5, further comprising:

under the condition that the data quantity of the data tables to be redistributed is smaller than the first threshold value, selecting a fourth data table from the data tables to be redistributed, wherein the fourth data table is any one of the data tables to be redistributed;

redistributing data of the fourth data table to some or all of the plurality of data nodes.

7. The method of any one of claims 1-6, further comprising:

determining a second data table under the condition of traffic congestion, wherein the second data table is a data table causing the traffic congestion;

stopping the redistribution process of the second data table.

8. A data redistribution method is applied to a coordination node, wherein the coordination node is used for managing a plurality of data nodes, and the method comprises the following steps:

according to the residual available write rate of the magnetic disk, redistributing the data of the data table to be redistributed to part or all of the data nodes;

the remaining available write rate of the disk is determined according to a write rate threshold and a write rate corresponding to a data table currently being redistributed, and the data table to be redistributed is a data table to be redistributed in the data tables included in the plurality of data nodes.

9. The method of claim 8, further comprising:

creating M threads, wherein M is an integer greater than or equal to 1;

and redistributing the data of the data table to be redistributed to part or all of the data nodes by the M threads.

10. The method according to claim 8 or 9, wherein the redistributing the data of the to-be-redistributed data table to some or all of the plurality of data nodes according to the remaining available write rate of the disk comprises:

selecting one or more data tables from the data tables to be redistributed according to the residual available write rate of the disk, wherein the write rate corresponding to the one or more data tables is less than or equal to the residual available write rate;

and selecting a data table with the maximum corresponding writing rate from the one or more data tables, and redistributing the data of the data table to part or all of the data nodes.

11. The method as claimed in claim 8 or 9, wherein the redistributing the data of the to-be-redistributed data table to some or all of the plurality of data nodes according to the remaining available write rate of the disk comprises:

under the condition that the data table with the corresponding data size larger than or equal to the first threshold exists in the data table to be redistributed, the data of the data table to be redistributed are redistributed to part or all of the data nodes according to the remaining available write rate of the disk.

12. The method of claim 11, further comprising:

13. An electronic device, comprising a processor, a memory, and a communication interface for receiving information from and outputting information to other electronic devices other than the electronic device, wherein the processor invokes a computer program stored in the memory to implement the method of any of claims 1-7 or 8-12.