CN113760897A

CN113760897A - Data re-slicing method, device, computer system and computer readable storage medium

Info

Publication number: CN113760897A
Application number: CN202110071969.8A
Authority: CN
Inventors: 张谦
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2021-01-19
Filing date: 2021-01-19
Publication date: 2021-12-07

Abstract

The present disclosure provides a data re-fragmentation method, a data re-fragmentation apparatus, a computer system, a computer-readable storage medium, and a computer program product. The method comprises the following steps: acquiring target fragmentation nodes, wherein the target fragmentation nodes are composed of at least one initial fragmentation node and at least one newly-added fragmentation node, each target fragmentation node comprises at least one database table, each database table of each initial fragmentation node comprises data stored in a partition mode, and the storage content in each database table of each newly-added fragmentation node is empty; generating a temporary table in each target fragmentation node in response to a data re-fragmentation request; sequentially acquiring initial partition data stored by each partition in each initial fragment node; and averagely distributing the initial partition data to a temporary table of each target fragmentation node so as to perform data re-fragmentation on the initial partition data.

Description

Data re-slicing method, device, computer system and computer readable storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data re-fragmentation method, a data re-fragmentation device, a computer system, a computer-readable storage medium, and a computer program product.

Background

With the rapid development of computer technology, mass data accompanying various business scenes is generated. The production of the wafer separator is a storage solution for mass data, and plays an extremely important role. In an actual service scenario, a constantly changing data volume is inevitably required to be faced, and an existing data fragmentation mode can not meet a new data storage requirement any more, so that it is important to design a reasonable data re-fragmentation scheme.

In the process of implementing the concept of the present disclosure, the inventors found that in the related art, at least the following problems exist, and the existing data re-fragmentation scheme is complex in implementation process and cannot effectively solve the problem of uneven data distribution, so that the performance of each fragment cannot be reasonably utilized.

Disclosure of Invention

In view of the above, the present disclosure provides a data re-fragmentation method, a data re-fragmentation apparatus, a computer system, a computer readable storage medium, and a computer program product.

One aspect of the present disclosure provides a data re-fragmentation method, including: acquiring target fragmentation nodes, wherein each target fragmentation node comprises at least one initial fragmentation node and at least one newly-added fragmentation node, each target fragmentation node comprises at least one database table, each database table of each initial fragmentation node comprises data stored in a partition mode, and the stored content in each database table of each newly-added fragmentation node is empty; generating a temporary table in each target fragmentation node in response to a data re-fragmentation request; sequentially acquiring initial partition data stored by each partition in each initial fragment node; and averagely distributing the initial partition data to a temporary table of each target fragmentation node so as to perform data re-fragmentation on the initial partition data.

Another aspect of the present disclosure provides a data re-fragmentation device, including: the system comprises a first acquisition module, a first fragmentation node and a second acquisition module, wherein the target fragmentation node is composed of at least one initial fragmentation node and at least one newly-added fragmentation node, each target fragmentation node comprises at least one database table, each database table of each initial fragmentation node comprises data stored in a partition mode, and the stored content in each database table of each newly-added fragmentation node is empty; the generating module is used for responding to a data re-fragmentation request and generating a temporary table in each target fragmentation node; the second obtaining module is used for sequentially obtaining initial partition data stored by each partition in each initial fragment node; and the distribution module is used for distributing the initial partition data to the temporary table of each target fragmentation node in an average manner so as to perform data re-fragmentation on the initial partition data.

Another aspect of the present disclosure provides a computer system comprising: one or more processors; memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a data re-fragmentation method as described above.

Another aspect of the present disclosure provides a computer-readable storage medium having stored thereon computer-executable instructions, which when executed, are for implementing a data re-fragmentation method as described above.

Another aspect of the present disclosure provides a computer program product comprising computer executable instructions for implementing the data re-fragmentation method as described above when executed.

According to the embodiment of the disclosure, target fragment nodes are obtained, wherein each target fragment node is composed of at least one initial fragment node and at least one new fragment node, each target fragment node comprises at least one database table, each database table of each initial fragment node comprises data stored in a partition mode, and the storage content in each database table of each new fragment node is empty; generating a temporary table in each target fragmentation node in response to a data re-fragmentation request; sequentially acquiring initial partition data stored by each partition in each initial fragment node; the original partition data can be evenly distributed to the new partition nodes by means of establishing the temporary table, the implementation process is simple, the technical problem that the existing data re-partitioning scheme cannot effectively solve the problem of uneven data distribution is at least partially solved, and the technical effect of reasonably utilizing the performance of each partition is achieved.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

fig. 1 schematically illustrates an exemplary system architecture to which a data re-fragmentation method may be applied, according to an embodiment of the present disclosure;

fig. 2 schematically illustrates a flow diagram of a data re-fragmentation method in accordance with an embodiment of the disclosure;

FIG. 3 schematically illustrates an organizational chart of a target sharded node and initial partition data according to an embodiment of the disclosure;

FIG. 4 schematically illustrates a diagram of creating a temporary table according to an embodiment of the disclosure;

FIG. 5 schematically illustrates a diagram of data re-fragmentation by partition according to an embodiment of the disclosure;

FIG. 6 schematically shows a schematic diagram of a data replacement process according to an embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of a data re-fragmentation device, in accordance with an embodiment of the present disclosure; and

FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing the above-described method, according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

The ClickHouse is a new analysis type database and is used in the analysis scene of mass data. Online applications are often deployed in clusters, where a cluster usually includes multiple shards, and each shard ensures data reliability in a multi-copy manner. With the continuous increase of the traffic, the data amount to be processed by the cluster will continuously increase, and with the increasing speed of the data amount, the cluster at this time needs to expand the capacity and increase the number of the fragments to support the processing of more data.

Before capacity expansion, the routing algorithm will ensure that data is written into each fragment substantially evenly. After capacity expansion, newly generated data can be written into all fragments as evenly as possible, but due to the fact that the newly added fragments lack historical data relative to the old fragments, the data volume is far smaller than that of the old fragments, serious data inclination (uneven data volume distribution on each fragment) can be caused, so that the newly expanded fragments cannot exert due calculation performance, and the calculation pressure of the cluster is still concentrated on the old fragments.

In view of the above problems, the existing online solution is to introduce a weight factor in the adjustment routing algorithm. The routing algorithm allocates a larger weight to the newly added fragment, so that the newly added fragment can obtain much data when data is newly written.

The inventor finds that, in the process of implementing the concept of the present disclosure, firstly, although the influence caused by data skew is alleviated in the prior art, the performance of the newly added shard node is still limited, and the calculation pressure is still on the old shard node until the data of the newly added shard node catches up with the old shard node. Secondly, the timing of adjusting the weight of the routing algorithm is not well known, and it is necessary to monitor whether the data volume of each segment is equivalent or not at all times, and then adjust the weight of the mean value. Moreover, the prior art cannot be applied to the scene of the reduction of the service data needing capacity reduction.

Embodiments of the present disclosure provide a data re-fragmentation method, a data re-fragmentation apparatus, a computer system, a computer-readable storage medium, and a computer program product. The method comprises the steps of obtaining target fragmentation nodes, wherein each target fragmentation node is composed of at least one initial fragmentation node and at least one newly added fragmentation node, each target fragmentation node comprises at least one database table, each database table of each initial fragmentation node comprises data stored in a partition mode, and the storage content of each database table of each newly added fragmentation node is empty; generating a temporary table in each target fragmentation node in response to a data re-fragmentation request; sequentially acquiring initial partition data stored by each partition in each initial fragment node; and averagely distributing the initial partition data to a temporary table of each target fragmentation node so as to perform data re-fragmentation on the initial partition data.

Fig. 1 schematically illustrates an exemplary system architecture 100 to which a data re-fragmentation method may be applied, according to an embodiment of the disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, and/or social platform software.

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the data re-slicing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the data re-slicing apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 105. The data re-fragmentation method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Accordingly, the data re-fragmentation device provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Alternatively, the data re-fragmentation method provided by the embodiment of the present disclosure may also be executed by the

terminal device

101, 102, or 103, or may also be executed by another terminal device different from the

terminal device

101, 102, or 103. Accordingly, the data re-slicing apparatus provided by the embodiment of the present disclosure may also be disposed in the

terminal device

101, 102, or 103, or in another terminal device different from the

terminal device

101, 102, or 103.

For example, the target sharded node and the initial partition data may be originally stored in any one of the

terminal devices

101, 102, or 103 (e.g., the terminal device 101, but not limited thereto), or stored on an external storage device and may be imported into the terminal device 101. Then, the terminal device 101 may locally perform the data re-fragmentation method provided by the embodiment of the present disclosure, or send the target fragmentation node and the initial partition data to other terminal devices, servers, or server clusters, and perform the data re-fragmentation method provided by the embodiment of the present disclosure by other terminal devices, servers, or server clusters receiving the target fragmentation node and the initial partition data.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 2 schematically shows a flow chart of a data re-fragmentation method according to an embodiment of the present disclosure.

As shown in fig. 2, the method includes operations S201 to S204.

In operation S201, a target fragment node is obtained, where the target fragment node is composed of at least one initial fragment node and at least one new fragment node, each target fragment node includes at least one database table, each database table of each initial fragment node includes data stored in a partition form, and the storage content in each database table of each new fragment node is empty.

According to an embodiment of the present disclosure, the initial sharded node is, for example, an existing sharded node in a cluster, the newly added sharded node is, for example, a newly added sharded node for implementing capacity expansion of the cluster, and the target sharded node is, for example, all sharded nodes in the cluster after capacity expansion. In order to ensure the uniformity and consistency of data distribution, a database table is arranged in the newly added fragment node, and the name, structure and number of the database table are consistent with those of the database table in the initial fragment node.

It should be noted that, the numbers of the initial fragment nodes and the newly added fragment nodes are determined by actual service scenarios or data volumes, and are not limited herein.

In operation S202, a temporary table is generated in each target sharding node in response to a data re-sharding request.

According to the embodiment of the present disclosure, after the newly added segment node is set in the cluster, the data in the initial segment node may be evenly distributed to the target segment node according to the data re-segment request sent by the user, and in order to implement the process of the average distribution, a temporary table needs to be established in each segment node (i.e., the target segment node) in the current cluster first to receive the evenly distributed data.

In operation S203, initial partition data stored by each partition in each initial sharded node is sequentially acquired.

In operation S204, the initial partition data is evenly distributed to the temporary table of each target fragmentation node to perform data re-fragmentation on the initial partition data.

According to the embodiment of the present disclosure, for example, the allocation operation of operation S204 may be performed on each piece of initial partition data acquired in sequence, so as to respond to the data re-fragmentation request, and implement data re-fragmentation on each piece of initial partition data.

According to the resharding (re-evenly distributing old data to designated shards) design method based on the partition data, the data re-fragmentation is carried out in a temporary table establishing mode, so that the data re-fragmentation process is clear and controllable, further adjustment is not needed, and uniform distribution of the re-fragmentable data can be achieved without monitoring the data volume of each fragment all the time. The problem of data inclination after cluster expansion is effectively solved, the value of the newly added fragments is quickly reflected, the performance of each fragment is reasonably utilized, and the overall data processing capacity of the cluster is further improved. Meanwhile, the method can also reduce the pressure of the old cluster, and further reduce the possibility of failure.

The method shown in fig. 2 is further described with reference to fig. 3-6 in conjunction with specific embodiments.

Fig. 3 schematically illustrates an organizational chart of a target sharded node and initial partition data according to an embodiment of the present disclosure.

According to the embodiment of the present disclosure, as shown in fig. 3, the initial shard node includes, for example, two shards 0 and a shard1, a newly added shard node for capacity expansion is, for example, a shard2, and the obtained target shard nodes are, for example, a shard0, a shard1, and a shard2, where data exists in the shards 0 and the shard1, and no data exists in the shard 2. Each of Shard0, Shard1 and Shard2 has at least one database table named Shard _ table, wherein each of the database tables of Shard0 and Shard1 includes initial Partition data organized in partitions (partitions), the database table of Shard2 stores empty data, and the name and structure of the database table of Shard2 are the same as those of the database tables of Shard0 and Shard1, and each of the Shard _ table database tables may have multiple partitions.

According to an embodiment of the present disclosure, the operation S202 may include, for example: determining a target table structure of a database table; and generating a temporary table according to the target table structure.

FIG. 4 schematically shows a schematic diagram of creating a temporary table according to an embodiment of the disclosure.

According to the embodiment shown in fig. 4, in the process of creating the temporary table, for example, a temporary table may be newly created on each Shard (including Shard0, Shard1 and Shard2), and the structure of the temporary table is set to be the same as that of the Shard _ table database table to be processed. Meanwhile, to avoid table name collision and establish a corresponding relationship between the original database table and the temporary table, in actual operation, for example, a fixed prefix + original table name may be used, or a fixed prefix + original table name + UUID may be used to generate the table name of the temporary table, as shown in fig. 4, the table name (tmp _ table) of the temporary table is determined, for example, in a manner of using the fixed prefix (tmp) + the original table name (table).

According to an embodiment of the present disclosure, the operation S203 may include, for example: determining a target partition corresponding to initial partition data which does not need to be subjected to data re-fragmentation operation in each initial fragmentation node according to a preset rule; and sequentially acquiring initial partition data corresponding to each partition except the target partition in each initial fragment node.

It should be noted that, in most scenarios, the Partition in the database table is divided according to a time sequence. Newly written data tends to be written into the newest partition, while operations for historical partitions (i.e., partitions that write data earlier) are mostly read operations, with few writes. Therefore, the sequentially acquiring operation may be, for example, sequentially acquiring and processing initial partition data in the data partition corresponding to the historical partition to the latest partition in each initial shard node in a time sequence. Meanwhile, when the data of the history partition is processed, the online writing operation is not affected basically (only in the case of writing operation on the history partition, as described above, the newly written data is often written into the latest partition). Therefore, the partition data corresponding to the latest partition where the data write operation may exist may be determined to be, for example, the initial partition data (i.e., the target partition) without performing the data re-fragmentation operation.

According to the above embodiments of the present disclosure, the influence on online writing may be increased when processing the above latest partition. Therefore, the data re-fragmentation method further provides a partition blacklisting function, and the target partitions can be added into a blacklist, so that when the data re-fragmentation method is implemented on a cluster, resharding operation can not be performed on the target partitions.

It should be noted that, in the case that it can be determined that there may be a history partition in which a data write operation exists, the corresponding history partition may also be added to the partition blacklist. Meanwhile, under the condition that the relevant partitions in the partition blacklist are determined not to have data writing operation any more, the partitions are moved out of the partition blacklist, and at the moment, data re-fragmentation can be performed on the partition data corresponding to the partitions.

Through the embodiment of the disclosure, because the partition blacklist capable of excluding the designated partitions (such as the target partition or the specific historical partition) is set, the resharding processing can be not performed on the partition data of the designated partitions, and the resharding operation can be performed on other partitions in the cluster after the designated partitions are excluded, so that the influence on the online service is reduced to the greatest extent under the condition that the problem of data inclination existing in the traditional data re-fragmentation process is guaranteed to be solved. Meanwhile, since the partition data of the latest partition is often in a small proportion compared with all the data which can be re-fragmented, the above-mentioned way of setting the partition blacklist has little influence on the data skew.

According to an embodiment of the present disclosure, the operation S204 may include, for example: averagely dividing the initial partition data into a plurality of sub-partition data according to the number of the target partition nodes; and distributing the plurality of sub-partition data to the temporary table of each target fragmentation node respectively.

Fig. 5 schematically illustrates a data re-fragmentation by partition according to an embodiment of the present disclosure.

According to the embodiment shown in fig. 5, the above-mentioned average allocation of the initial Partition data into the temporary table may be represented by, for example, performing average allocation on each Partition (for example, Partition0 Partition in fig. 5) in the original database table Shard _ table, and then sequentially copying the allocation results into the temporary table. To achieve this even distribution, it is necessary to first determine the number of target shard nodes, for example, the number of target shard nodes in fig. 5 is 3, so that Partition0 partitions in shard0 and shard1 are each divided into three parts (i.e., divided into 3 sub-Partition data) and written into temporary tables of shard0, shard1, and shard2, respectively.

Through the embodiment of the disclosure, the re-even distribution of the re-shareable data can be realized, the value of newly added shards can be quickly reflected, and the performance of each shard is reasonably utilized.

According to the embodiment of the disclosure, as shown in fig. 5, in the case of data re-fragmentation facing data re-fragmentation of mass data of multiple clusters and multiple partitions, by executing the method shown in operation S204 on each Partition in the database table of each fragmentation node in each cluster, it is possible to re-evenly distribute data of all Partition 0-Partition N partitions in an original cluster (a cluster not subjected to data re-fragmentation operation) into tmp _ tables including all fragmentation nodes, thereby implementing data re-fragmentation for re-fragmentable data in the original cluster.

According to an embodiment of the present disclosure, to implement data re-fragmentation of massive data in multiple clusters and multiple partitions, that is, for example, in a case that the initial fragmentation node at least includes a first initial fragmentation node and a second initial fragmentation node, and each initial fragmentation node at least includes a first partition and a second partition, the data re-fragmentation method may further include: locking a first partition of a first initial fragmentation node and a first partition of a second initial fragmentation node; the method comprises the steps that first initial partition data stored in a first partition of a first initial fragmentation node are evenly distributed to a temporary table of each target fragmentation node, so that first sub-partition data are generated in each temporary table; the second initial partition data stored in the first partition of the second initial fragmentation node are evenly distributed to a temporary table of each target fragmentation node, so that second sub-partition data are generated in each temporary table; taking the first sub-partition data and the second sub-partition data as target data stored by the first partition in each temporary table; carrying out consistency check on the total data of the first initial partition data and the second initial partition data and the target data; under the condition of consistency check, replacing a first partition in a database table of each target fragmented node with a first partition of a temporary table corresponding to the database table; unlocking a first partition of the first initial fragmentation node and a first partition of the second initial fragmentation node; and executing the same operation as the first partition in the first initial fragmenting node and the second initial fragmenting node aiming at the second partition in the first initial fragmenting node and the second initial fragmenting node.

According to the embodiment shown in fig. 5, the above-mentioned locking and unlocking operation may be represented by locking the partition currently being processed and unlocking the partition after the processing of the partition currently being processed is completed, for example. The partitions currently being processed as in fig. 5 may include, for example, Partition0 in the Shard _ table database table of Shard0 and Partition0 in the Shard _ table database table of Shard1, so that in the embodiment of fig. 5, for example, the Partition0 in Shard0 and Shard1 is first locked at the same time to prohibit data updating (i.e., write prohibited as described above), and after reallocation and replacement of data in Partition0 in Shard0 and Shard1 is completed, the Partition0 in Shard0 and Shard1 is unlocked, and then the next Partition is ready to be accessed.

It should be noted that the above-mentioned manner of locking and unlocking the initial partition data is not limited to that shown in fig. 5, and in the actual implementation process, for example, the locking, copying (i.e., allocating) and unlocking of the first partition in the Shard _ table database table of the Shard0 may be performed after the locking, copying (i.e., allocating) and unlocking of the first partition in the Shard _ table database table of the Shard1 are completed, and this manner may be performed sequentially for other partitions in the Shard0 and the Shard 1.

According to the embodiment shown in fig. 5, the data consistency check is performed, for example, after the Partition data is completely written (i.e., allocated to the temporary table), and the check manner may be implemented by comparing data amounts of data corresponding to the original table (Shard _ table) and the temporary table (tmp _ table) (e.g., Partition0 Partition data in the original table and Partition0 Partition data in the temporary table). And can be replaced in case of consistency check, and for example, the corresponding data re-fragmentation program can be interrupted or re-executed in case of inconsistency check.

Fig. 6 schematically shows a schematic diagram of a data replacement process according to an embodiment of the present disclosure.

According to the embodiment shown in fig. 6, the above replacement procedure may be implemented, for example, by a Partition Replace method, which may be represented, for example, by replacing the Partition0 of the original table Shard _ table with the Partition0 of the temporary table. And after the replacement is completed, the lock of the partition is released, and the next partition operation is carried out.

Through the above embodiments of the present disclosure, when part 0 is operated, the partition is locked in a distributed manner, and other operators are prohibited from writing the partition data to avoid final inconsistency of the data, but other operators can operate the data of other partitions at will, and the influence on the online service can be reduced to the greatest extent under the condition of ensuring that the problem of data skew is solved.

According to an embodiment of the present disclosure, the operation of replacing the partition may be directly implemented by replacing the disk data, and the process of replacing the disk data may be represented as: acquiring a first disk path corresponding to a first partition in a database table; acquiring a second disk path corresponding to the first partition of the temporary table corresponding to the database table; and replacing the first disk path with the second disk path.

According to the embodiment shown in fig. 6, the first disk path may be represented by, for example, Shard0\ Shard _ table \ Partition0 or Shard1\ Shard _ table \ Partition0, and the second disk path may be represented by, for example, Shard0\ tmp _ table \ Partition0, Shard1\ tmp _ table \ Partition0 or Shard2\ tmp _ table \ Partition 0.

Through the above embodiments of the present disclosure, the Replace method can be used to Replace the original data, and the checksum data replacement does not take too long time for the lock, so that the implementation efficiency of the data re-fragmentation process can be greatly accelerated.

According to the above embodiment of the present disclosure, after Partition operations (including locking, reallocation, verification, replacement, and unlocking) on Partition0 are completed, Partition0, which originally has three-part data, of the Shard table on each Partition is replaced with Partition0, which is two-part data of the temporary table tmp _ table. By repeatedly executing the above operations on the remaining partitions and the remaining tables, the process of data resharding in the whole cluster can be completed.

According to the embodiment of the disclosure, the method is also applicable to a reduced-capacity scenario, in the reduced-capacity scenario, the initial fragment nodes may include at least two, and the target fragment node is formed by a part of the initial fragment nodes. For example, the number of old fragments (i.e., initial fragment nodes) is three, that is, the number of the shards 0, the shard1 and the shard2, and the shrunk shards 0 and the shard1, in this scenario, the temporary table may be created in the shards 0 and the shard1, and the shrinking operation of the cluster may be implemented by adjusting the data re-fragmentation method according to the corresponding relationship in the shrinking scenario.

Through the embodiment of the disclosure, the data re-fragmentation method can be simultaneously suitable for a capacity reduction scene, the problem that the existing data re-fragmentation method cannot reduce the capacity is effectively solved, and a large amount of calculation and hardware resources can be saved.

Through the embodiment of the disclosure, the designated fragment for evenly writing the data in each partition of each table on the old fragment in sequence is adopted, and the data re-fragmentation method of the original data partition is replaced, so that the method can be simultaneously suitable for the scenes of capacity expansion and capacity reduction.

Fig. 7 schematically illustrates a block diagram of a data re-fragmentation device according to an embodiment of the present disclosure.

As shown in fig. 7, the data re-slicing apparatus 700 includes a first obtaining module 710, a generating module 720, a second obtaining module 730, and a first allocating module 740.

The first obtaining module 710 is configured to obtain a target fragment node, where the target fragment node is composed of at least one initial fragment node and at least one new fragment node, each target fragment node includes at least one database table, each database table of each initial fragment node includes data stored in a partition form, and storage content in each database table of each new fragment node is empty.

A generating module 720, configured to generate a temporary table in each target sharding node in response to the data re-sharding request.

The second obtaining module 730 is configured to sequentially obtain initial partition data stored in each partition in each initial sharded node.

The first allocating module 740 is configured to evenly allocate the initial partition data to the temporary table of each target fragmentation node, so as to perform data re-fragmentation on the initial partition data.

According to the embodiment of the disclosure, the target fragment node is obtained, wherein the target fragment node is composed of at least one initial fragment node and at least one new fragment node, each target fragment node comprises at least one database table, each database table of each initial fragment node comprises data stored in a partition mode, and the storage content in each database table of each new fragment node is empty; generating a temporary table in each target fragmentation node in response to a data re-fragmentation request; sequentially acquiring initial partition data stored by each partition in each initial fragment node; the original partition data can be evenly distributed to the new partition nodes by means of establishing the temporary table, the implementation process is simple, the technical problem that the existing data re-partitioning scheme cannot effectively solve the problem of uneven data distribution is at least partially solved, and the technical effect of reasonably utilizing the performance of each partition is achieved.

According to an embodiment of the present disclosure, the generation module includes a first determination unit and a generation unit.

The first determining unit is used for determining a target table structure of the database table.

And the generating unit is used for generating the temporary table according to the target table structure.

According to an embodiment of the present disclosure, the second obtaining module includes a second determining unit and a first obtaining unit.

And the second determining unit is used for determining a target partition corresponding to the initial partition data which does not need to be subjected to the data re-fragmentation operation in each initial fragmentation node according to a preset rule.

And the first acquisition unit is used for sequentially acquiring initial partition data corresponding to each partition except the target partition in each initial fragment node.

According to an embodiment of the present disclosure, the allocation module includes a dividing unit and an allocating unit.

And the dividing unit is used for averagely dividing the initial partition data into a plurality of sub-partition data according to the number of the target partition nodes.

And the distribution unit is used for distributing the data of the plurality of sub-partitions to the temporary table of each target partition node respectively.

According to an embodiment of the present disclosure, the initial sharded nodes at least include a first initial sharded node and a second initial sharded node, each initial sharded node at least includes a first partition and a second partition, and the data re-sharding apparatus further includes a locking module, a second allocating module, a third allocating module, a defining module, a checking module, a replacing module, an unlocking module, and an executing module.

And the locking module is used for locking the first partition of the first initial fragmentation node and the first partition of the second initial fragmentation node.

And the second distribution module is used for averagely distributing the first initial partition data stored in the first partition of the first initial fragmentation node to the temporary tables of each target fragmentation node so as to generate first sub-partition data in each temporary table.

And the third distribution module is used for averagely distributing the second initial partition data stored in the first partition of the second initial fragmentation node to the temporary tables of each target fragmentation node so as to generate second sub-partition data in each temporary table.

And the defining module is used for taking the first sub-partition data and the second sub-partition data as target data stored by the first partition in each temporary table.

And the checking module is used for carrying out consistency checking on the total data of the first initial partition data and the second initial partition data and the target data.

And the replacing module is used for replacing the first partition in the database table with the first partition of the temporary table corresponding to the database table for each target fragment node under the condition of consistency check.

And the unlocking module is used for unlocking the first partition of the first initial fragmentation node and the first partition of the second initial fragmentation node.

And the execution module is used for executing the same operation as the first partition in the first initial fragmenting node and the second initial fragmenting node aiming at the second partition in the first initial fragmenting node and the second initial fragmenting node.

According to an embodiment of the present disclosure, the replacement module includes a second obtaining unit, a third obtaining unit, and a replacement unit.

And the second acquisition unit is used for acquiring a first disk path corresponding to the first partition in the database table.

And the third acquisition unit is used for acquiring a second disk path corresponding to the first partition of the temporary table corresponding to the database table.

A replacement unit, configured to replace the first disk path with the second disk path.

Any of the modules, units, or at least part of the functionality of any of them according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules and units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, units according to the embodiments of the present disclosure may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by any other reasonable means of hardware or firmware by integrating or packaging the circuits, or in any one of three implementations of software, hardware and firmware, or in any suitable combination of any of them. Alternatively, one or more of the modules, units according to embodiments of the present disclosure may be implemented at least partly as computer program modules, which, when executed, may perform the respective functions.

For example, any plurality of the first obtaining module 710, the generating module 720, the second obtaining module 730, and the first allocating module 740 may be combined and implemented in one module/unit, or any one of the modules/units may be split into a plurality of modules/units. Alternatively, at least part of the functionality of one or more of these modules/units may be combined with at least part of the functionality of other modules/units and implemented in one module/unit. According to an embodiment of the present disclosure, at least one of the first obtaining module 710, the generating module 720, the second obtaining module 730, and the first allocating module 740 may be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the first obtaining module 710, the generating module 720, the second obtaining module 730 and the first allocating module 740 may be at least partly implemented as a computer program module, which when executed may perform a corresponding function.

It should be noted that, the data re-fragmentation device portion in the embodiment of the present disclosure corresponds to the data re-fragmentation method portion in the embodiment of the present disclosure, and the description of the data re-fragmentation device portion specifically refers to the data re-fragmentation method portion, which is not described herein again.

FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing the above-described method, according to an embodiment of the present disclosure. The computer system illustrated in FIG. 8 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.

As shown in fig. 8, a computer system 800 according to an embodiment of the present disclosure includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the present disclosure.

In the RAM 803, various programs and data necessary for the operation of the system 800 are stored. The processor 801, the ROM802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM802 and/or RAM 803. Note that the programs may also be stored in one or more memories other than the ROM802 and RAM 803. The processor 801 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

System 800 may also include an input/output (I/O) interface 805, also connected to bus 804, according to an embodiment of the disclosure. The system 800 may also include one or more of the following components connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM802 and/or RAM 803 described above and/or one or more memories other than the ROM802 and RAM 803.

Embodiments of the present disclosure also include a computer program product comprising a computer program comprising program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, for causing the electronic device to implement the data re-fragmentation method provided by the embodiments of the present disclosure.

The computer program, when executed by the processor 801, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via communication section 809, and/or installed from removable media 811. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of data re-fragmentation comprising:

acquiring target fragmentation nodes, wherein each target fragmentation node comprises at least one initial fragmentation node and at least one newly-added fragmentation node, each target fragmentation node comprises at least one database table, each database table of each initial fragmentation node comprises data stored in a partition mode, and the stored content in each database table of each newly-added fragmentation node is empty;

generating a temporary table in each target fragmentation node in response to a data re-fragmentation request;

sequentially acquiring initial partition data stored by each partition in each initial fragment node; and

and averagely distributing the initial partition data to a temporary table of each target fragmentation node so as to perform data re-fragmentation on the initial partition data.

2. The method of claim 1, wherein generating a temporary table in each of the target sharding nodes in response to a data re-sharding request comprises:

determining a target table structure of the database table; and

and generating the temporary table according to the target table structure.

3. The method of claim 1, wherein sequentially obtaining initial partition data stored by each partition in each of the initial sharded nodes comprises:

determining a target partition corresponding to initial partition data which does not need to be subjected to data re-fragmentation operation in each initial fragmentation node according to a preset rule; and

and sequentially acquiring initial partition data corresponding to each partition except the target partition in each initial fragment node.

4. The method of claim 1, wherein evenly allocating the initial partition data to the temporary table of each of the target sharded nodes comprises:

averagely dividing the initial partition data into a plurality of sub-partition data according to the number of the target fragment nodes; and

and distributing the sub-partition data to a temporary table of each target fragmentation node.

5. The method of claim 1, wherein the initial sharded nodes comprise at least a first initial sharded node and a second initial sharded node, each of the initial sharded nodes comprising at least a first partition and a second partition, the method further comprising:

locking a first partition of the first initial fragmentation node and a first partition of the second initial fragmentation node;

distributing first initial partition data stored in a first partition of the first initial fragmentation node to a temporary table of each target fragmentation node averagely to generate first sub-partition data in each temporary table;

distributing second initial partition data stored in a first partition of the second initial fragmentation node to a temporary table of each target fragmentation node averagely to generate second sub-partition data in each temporary table;

taking the first sub-partition data and the second sub-partition data as target data stored by the first partition in each temporary table;

performing consistency check on the total data of the first initial partition data and the second initial partition data and the target data;

under the condition of consistency check, replacing a first partition in a database table of each target fragmentation node with a first partition of a temporary table corresponding to the database table;

unlocking a first partition of the first initial fragmentation node and a first partition of the second initial fragmentation node; and

and executing the same operation as the first partition in the first initial fragmenting node and the second initial fragmenting node aiming at the second partition in the first initial fragmenting node and the second initial fragmenting node.

6. The method of claim 5, wherein for each of the target sharded nodes, replacing a first partition in a database table therein with a first partition of a temporary table corresponding to the database table comprises:

acquiring a first disk path corresponding to a first partition in the database table;

acquiring a second disk path corresponding to the first partition of the temporary table corresponding to the database table; and

replacing the first disk path with the second disk path.

7. A data re-fragmentation device, comprising:

the system comprises a first acquisition module, a first fragmentation node and a second acquisition module, wherein the target fragmentation node is composed of at least one initial fragmentation node and at least one newly-added fragmentation node, each target fragmentation node comprises at least one database table, each database table of each initial fragmentation node comprises data stored in a partition mode, and the stored content in each database table of each newly-added fragmentation node is empty;

the generating module is used for responding to a data re-fragmentation request and generating a temporary table in each target fragmentation node;

the second obtaining module is used for sequentially obtaining initial partition data stored by each partition in each initial fragment node; and

and the distribution module is used for distributing the initial partition data to the temporary table of each target fragmentation node in an average manner so as to perform data re-fragmentation on the initial partition data.

8. A computer system, comprising:

one or more processors;

a memory for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-6.

9. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 6.

10. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 6 when executed.