CN115118604B

CN115118604B - Dynamic capacity expansion data migration method, device, system and medium

Info

Publication number: CN115118604B
Application number: CN202210766267.6A
Authority: CN
Inventors: 刘明
Original assignee: Hangzhou Yuxin Digital Technology Co ltd
Current assignee: Hangzhou Yuxin Digital Technology Co ltd
Priority date: 2022-07-01
Filing date: 2022-07-01
Publication date: 2023-04-11
Anticipated expiration: 2042-07-01
Also published as: CN115118604A

Abstract

The embodiment of the invention provides a data migration method, equipment, a system and a medium for dynamic capacity expansion, wherein the method comprises the following steps: determining a routing mapping table of the fragments to be migrated, wherein the routing mapping table comprises one or more fragments to be migrated; sequentially selecting each fragment to be migrated as a current migration fragment according to a fragment to be migrated route mapping table, and circularly executing blocking operation, migration operation and unlocking operation until no fragment to be migrated exists in the fragment to be migrated route mapping table; the blocking operation comprises blocking the execution of the transaction according to the current migration fragment; the migration operation comprises migrating the current migration fragment after blocking the execution of the transaction; and the unlocking operation comprises unlocking to execute the transaction after the data migration of the current migration fragment is completed. The method can realize the application of the service system and the dynamic expansion and contraction capacity processing of the data nodes by adopting a fragment routing mapping table and a mode of blocking the transaction at a plurality of layers, and ensures that no influence is caused on the service transaction of an IT system client.

Description

Dynamic capacity expansion data migration method, device, system and medium

Technical Field

The present invention relates to the field of internet related technologies, and in particular, to a method, device, system, and medium for data migration of dynamic scaling.

Background

With the continuous and rapid expansion of the internet service range, the service access volume, the transaction volume and the data volume of the distributed system deployed by an enterprise fluctuate with various factors, including the steep increase of the data volume and the slow data volume with the service change. Therefore, dynamic capacity expansion or capacity reduction needs to be performed on the distributed system, and meanwhile, dynamic migration needs to be performed on stored data.

In order to make the IT system match the hardware resource investment with the change of active service data volume at any time, the IT system must have the capability of dynamic lateral expansion or contraction, and the increase and decrease of host nodes are ensured to be completed on the premise of not stopping service and not influencing normal service. Current data slicing schemes in the IT industry rely entirely on distributed databases (e.g., tiDB, golden db), and related middleware (Shardingsphere, myCat). In the application process, the data node expansion and reduction processing is complex, the expansion and reduction process is uncontrollable for applying opaque services, and particularly, the aspects of data relating range, processing progress, exception processing and the like can bring great burden to operation and maintenance.

Disclosure of Invention

In view of the foregoing problems, an object of the embodiments of the present invention is to provide a method, an apparatus, a system, and a medium for dynamically scaling data, so as to implement the goal of performing horizontal scaling on corresponding data nodes and data nodes in combination with actual service requirements.

To achieve the above object, in a first aspect, an embodiment of the present invention provides a data migration method for dynamic scaling, including:

determining a routing mapping table of fragments to be migrated, wherein the routing mapping table of the fragments to be migrated comprises one or more fragments to be migrated;

sequentially selecting each fragment to be migrated as a current migration fragment according to the fragment to be migrated route mapping table, and circularly executing blocking operation, migration operation and unlocking operation until no fragment to be migrated exists in the fragment to be migrated route mapping table;

wherein the lockout operation comprises: blocking execution of the transaction according to the current migration fragment; the migration operation comprises: after blocking the execution of the transaction, migrating the current migration fragment; the unlocking operation includes: and after the data migration of the current migration fragment is finished, unlocking to execute the transaction.

Further, determining a to-be-migrated fragmentation routing mapping table includes:

determining a current fragment routing mapping table according to the current fragment container of the current active partition and fragments contained in each current fragment container;

determining a target fragment route mapping table according to the current fragment route mapping table and a target fragment container of a planned active partition;

and determining a to-be-migrated fragment routing mapping table according to the current fragment routing mapping table and the target fragment routing mapping table.

Further, determining a target fragment route mapping table according to the current fragment route mapping table and the target fragment container specifically includes:

determining the average value of the number of fragments contained in the current fragment container according to the current fragment routing mapping table;

sequencing the current fragment containers in sequence according to the number of fragments contained in each fragment container;

and according to the expansion and reduction tasks and the average value of the number of the fragments, sequentially determining the migration direction of the fragments in the current fragment container corresponding to the target fragment container according to the sequence of the current fragment container, and determining a target fragment routing mapping table.

Further, blocking execution of the transaction according to the current migration fragment comprises:

according to the current migration fragment, executing a first-stage transaction blocking, wherein the first-stage transaction blocking comprises the following steps: blocking an external transaction entrance related to the current migration fragment, and delaying for N seconds to wait for the transaction processing in execution to be finished;

after the first-stage transaction lockout is successfully performed, performing a second-stage transaction lockout, the second-stage transaction lockout including: respectively blocking the program interface and the data processing related to the current migration fragment in the database.

Further, the external transaction entrance comprises an internal gateway and an external gateway;

blocking the external transaction entrance related to the current migration fragment, comprising:

and blocking all newly initiated transaction requests related to fragment data modification for the internal gateway and the external gateway related to the current migration fragment.

Further, the external transaction entrance also comprises an external message queue and a batch processing task;

blocking the external transaction entrance related to the current migration fragment, further comprising:

for an external message queue, blocking transaction at a message production end through the fragment route mapping table to be migrated;

for the batch processing task currently being executed, the blocking processing is carried out in the batch processing method section, and the new batch processing task is blocked.

Further, after blocking the execution transaction, migrating the current migration segment specifically includes:

according to the database table replication process, executing database-to-library data replication on the fragment data of the current migration fragment to complete the replication of the fragment data from the current fragment container to the target fragment container,

if the copying is successful, clearing the fragment data of the current migration fragment in the current fragment container;

if the copy fails, the current operation is terminated, and the data migrated to the new container is automatically cleared.

Further, after the data migration of the current migration fragment is completed, unlocking to execute the transaction, including:

after the data migration of the current migration fragment is completed, executing first-stage transaction unlocking, wherein the first-stage transaction unlocking comprises the following steps: firstly, unblocking a database, and deleting all data records corresponding to the current migration fragment; then, the blocking of the program interface is released, the current migration fragment is removed from the to-be-migrated fragment routing mapping table, and the control of the transaction corresponding to the current fragment factor of the current migration fragment is opened;

after the first-stage transaction unlocking is successfully executed, second-stage transaction unlocking is executed, wherein the second-stage transaction unlocking comprises the following steps: unblocking the external transaction entry and updating the migration processing state.

Further, before each execution of sequentially selecting each fragment to be migrated as the current migration fragment, the method further includes:

judging whether the current execution state meets the migration condition;

if the migration conditions are met, sequentially selecting the fragments to be migrated as the current migration fragments;

if the migration condition is not met, automatically pausing the migration processing, suspending the current migration task after the execution state is saved, and returning to execute the step of judging whether the current execution state meets the migration condition or not after the next migration period.

Further, determining whether the current execution state satisfies the migration condition includes:

judging whether the residual migration time window on the current day is sufficient;

and/or judging whether the current migration process is abnormal or not.

In a second aspect, an embodiment of the present invention provides a data migration system for dynamic scaling, including:

the fragment routing determining module is used for determining a fragment routing mapping table to be migrated, wherein the fragment routing mapping table to be migrated comprises one or more fragments to be migrated; and the number of the first and second groups,

the fragment migration processing module is used for sequentially selecting each fragment to be migrated as the current migration fragment according to the fragment to be migrated route mapping table; sequentially controlling the locking submodule, the migration submodule and the unlocking submodule to be circularly executed until the to-be-migrated fragments to be migrated do not exist in the to-be-migrated fragment routing mapping table;

the blocking submodule is used for blocking the execution of the transaction according to the current migration fragment;

the migration submodule is used for migrating the current migration fragment after blocking the execution of the transaction;

and the unlocking submodule is used for unlocking and executing the transaction after the data migration of the current migration fragment is completed.

Further, the fragment routing determining module is specifically configured to:

determining a current fragment routing mapping table according to a current fragment container of a current active partition and fragments contained in each current fragment container;

and determining a fragment route mapping table to be migrated according to the current fragment route mapping table and the target fragment route mapping table.

Further, the lockout submodule is specifically configured to:

executing a first-stage transaction blocking according to the current migration fragment, wherein the first-stage transaction blocking comprises the following steps: blocking an external transaction entrance related to the current migration fragment, and delaying for N seconds to wait for the transaction processing in execution to be finished;

after the first-stage lockout is successfully performed, performing a second-stage lockout, the second-stage lockout including: and respectively blocking the program interface and the data processing related to the current migration fragment in the database.

In a third aspect, an embodiment of the present invention provides a computer device, including:

one or more processors;

storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the dynamically scaled data migration method as described above.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements any one of the above-mentioned data migration methods for dynamic scaling.

The data migration method, the device, the system and the computer readable storage medium for dynamic expansion and contraction capacity provided by the embodiment of the invention have the following beneficial effects:

in order to ensure that the daytime transaction is not influenced to the maximum extent by data migration operation during capacity expansion and reduction, the method adopts a mode of migrating only one piece of fragment data at a time and blocking the service corresponding to the fragment, avoids stopping all services in the process of capacity expansion and reduction blocking, and effectively reduces the transaction requests influenced by the capacity expansion and reduction process.

In order to ensure that the new transaction can be normally blocked when the fragmented data is migrated, a two-stage blocking mode is adopted, wherein the complete transaction blocking in the first stage is directly processed at each gateway layer, and the transaction blocking in the second stage is processed at two layers of an API and a database. Through blocking transactions at multiple levels, correctness and integrity of the data being migrated are ensured as much as possible. The transaction blocking is performed as far as possible at the front of the transaction chain, so that the complexity of various abnormal scenes in the blocking process is reduced, and the influence on the customer experience is also reduced.

By the dynamic capacity expansion and contraction data migration method, the device, the system and the computer readable storage medium, application of a service system and dynamic capacity expansion and contraction processing of data nodes can be realized, the characteristics of data range control, capacity expansion and contraction progress control, interruptible rollback and the like are provided, no influence on service transactions of IT system clients is guaranteed, and a controllable target is achieved for system operation and maintenance personnel.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic control block diagram of a data migration method of dynamic scaling according to an embodiment of the present invention;

FIG. 2 is a schematic flow diagram of a method for dynamically scaling a volume of data migration in accordance with an embodiment of the present invention;

FIG. 3 is a schematic flow chart of a preferred mode of operation corresponding to step S1 in FIG. 2;

FIG. 4 is a schematic flow chart of a preferred mode of operation corresponding to step S12 in FIG. 3;

FIG. 5 is a schematic view of slice migration of a data migration method for dynamic scaling according to an embodiment of the present invention;

FIG. 6 is a schematic flow chart of a preferred manner of operation corresponding to step S21 in accordance with an embodiment of the present invention;

FIG. 7 is a schematic flow chart diagram of a preferred manner of operation corresponding to step S23 in accordance with an embodiment of the present invention;

FIG. 8 is a schematic diagram of a dynamically scaled data migration system according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of an application scenario between a scalability control center and a peripheral business service and a basic support service according to an embodiment of the present invention;

FIG. 10 is a schematic block diagram of a computer device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

For better understanding of the present invention, a data migration method, an apparatus, a system and a computer-readable storage medium for dynamic scaling provided by the embodiments of the present invention are described in detail below with reference to fig. 1 to 10. It should be noted that these examples are not intended to limit the scope of the present disclosure.

Fig. 1 is a schematic control block diagram of a data migration method of dynamic scaling according to an embodiment of the present invention, fig. 2 is a schematic flow chart of the data migration method of dynamic scaling according to an embodiment of the present invention, and as shown in fig. 1 and fig. 2, the data migration method of dynamic scaling according to an embodiment of the present invention may be applied to a distributed system, and specifically, the data migration method of dynamic scaling may be configured in a scaling control center. As shown in fig. 2, the method may specifically include:

step S1: and determining a routing mapping table of the fragments to be migrated, wherein the routing mapping table comprises one or more fragments to be migrated.

Specifically, when the expansion and contraction capacity is started, the number of the current active partitions is reset through the configuration center of the expansion and contraction capacity control center, the expansion and contraction capacity switch is turned on, and the expansion and contraction capacity task of the partitioned container is activated.

And determining a to-be-migrated fragmentation routing mapping table to be migrated according to the number of the current fragmentation containers obtained from the configuration center and the number of the target fragmentation containers set by the expansion and contraction capacity of the time. The method comprises the steps that a fragment to be migrated is obtained according to a fragment routing mapping table, and the fragment to be migrated is transferred from an existing fragment container to another fragment container. And generating a migration task detail list of the expansion and contraction capacity of the current time on the basis, wherein the task for generating the migration task detail list can be started manually, and the subsequent steps carry out migration processing on the data one by one according to the tasks listed in the migration task detail list.

Specifically, as shown in fig. 1, after the capacity expansion and reduction task of the fragment container is activated, the partition activation and the update configuration are determined, and one fragment to be migrated is selected from the determined route mapping table of the fragment to be migrated as the current migration fragment. Before fragment data migration is carried out, blocking transaction for gateways related to current migration fragments is carried out, blocking fragment transaction for services related to current migration fragments is carried out, and migration is carried out on current migration fragments after blocking is confirmed to be completed. After the migration of the fragment data is completed, firstly, the fragment transaction is unlocked for each service related to the current migration fragment, and then the transaction is unlocked for the gateway related to the current migration fragment. After unlocking is completed, judging whether the fragment to be migrated still exists in the routing mapping table of the fragment to be migrated, namely whether the fragment which is not migrated still exists, if so, continuously judging whether the current execution state meets the migration condition, if so, continuously reselecting the fragment to be migrated as a new current migration fragment, and executing a fragment migration task; if not, automatically pausing migration processing, suspending the current task after the execution state is saved, waiting for the next migration period, resuming to determine the current migration fragment, and continuing to execute the fragment migration task. And if the to-be-migrated fragment does not exist in the to-be-migrated fragment routing mapping table, finishing the expansion and reduction of the capacity.

Optionally, in an embodiment, as shown in fig. 3, step S1 may specifically include:

s11: and determining a current fragment routing mapping table according to the current fragment container of the current active partition and the fragments contained in each current fragment container.

TABLE 1

Specifically, the current fragment routing mapping table is determined according to the number of current fragment containers obtained from the configuration center and the fragments contained in each fragment container, as shown in table 1, the current fragment routing mapping table is a partial fragment routing mapping table, and table 1 schematically shows four different situations corresponding to different numbers of fragment containers when the number of fragment containers is 4. The fragment routing mapping table comprises a mapping relation between a slot ID and a fragment container ID. Each data fragment corresponds to a slot ID, the total number of slot IDs is set by the system, and the number of slot IDs is generally set to be far greater than the number of data fragments in order to satisfy that all data fragments can successfully configure slot IDs.

S12: and determining a target fragment route mapping table according to the current fragment route mapping table and the target fragment container of the planned active partition.

Optionally, in an embodiment of the present invention, as shown in fig. 4, step S12 may specifically include:

s121: and determining the average value of the number of the fragments contained in the current fragment container according to the current fragment routing mapping table. Specifically, the average value of the number of the fragments contained in the multiple fragment containers of the current active partition is calculated through the current fragment routing mapping table. For example, the current active partition includes n sharded containers, and the number of shards included in each sharded container is M1, M2.

；

Wherein M is an integer.

S122: sequencing the current fragment containers in sequence according to the number of fragments contained in each fragment container;

s123: and determining the migration direction of the fragments in the current fragment container corresponding to the target fragment container according to the expansion and contraction task and the average value of the number of the fragments and the sequence of the current fragment container, and determining a target fragment routing mapping table.

Specifically, for the capacity expansion task: (1) And sequencing the current fragment container in sequence according to the fragment number contained in each fragment container from high to low, and selecting the fragment containers according to the sequence of the fragment number from high to low. (2) Preferably selecting the fragment containers which are larger than the average value of the fragments and have more remainders, and circularly selecting the fragments from the selected fragment containers as the fragments to be migrated according to the sequence, wherein one or more fragments can be selected from one selected fragment container as the fragments to be migrated each time, and the fragments to be migrated are circularly selected from the plurality of fragment containers according to the sequence. (3) And sequentially corresponding the selected fragments to be migrated to a target fragment container in the plan active partition. And correspondingly moving and mapping the fragments from the fragment container with a large number of fragments to the newly added fragment container or the fragment container with a small number of fragments, so that the number of the fragments mapped to the target fragment container is uniformly distributed as much as possible. (4) And generating a corresponding target fragment routing mapping table according to the corresponding mobile mapping relation between the fragments to be migrated and the target fragment container until the number of the fragments in the new fragment container in the target fragment routing mapping table reaches an average value or no fragment can be migrated.

Aiming at the capacity reduction task: (1) And sequencing the current fragment containers in sequence from low to high according to the number of fragments contained in each fragment container, and selecting the fragment containers according to the sequence from low to high of the number of fragments. (2) And preferentially selecting the fragment containers smaller than the average value of the fragments, sequentially selecting the fragments to be migrated from the selected fragment containers in a circulating manner according to the sequence, wherein one or more fragments can be selected from one selected fragment container each time to be migrated as the fragments to be migrated, and sequentially and circularly selecting the fragments to be migrated from the fragment containers according to the arrangement sequence. (3) And sequentially corresponding the selected fragments to be migrated to a target fragment container in the plan active partition. And correspondingly moving and mapping the fragments from the fragment container with less fragments to the fragment container with less fragments than the average value of the fragments, so that the number of the fragments mapped to the target fragment container is uniformly distributed as much as possible. (4) And generating a corresponding target fragment routing mapping table according to the corresponding movement mapping relation between the fragments to be migrated and the target fragment container until the number of the fragments in the fragment container in the target fragment routing mapping table reaches an average value or no fragment can be migrated.

Specifically, fig. 5 is a fragment migration diagram of the data migration method for dynamic expansion and contraction capacity according to the embodiment of the present invention, and as shown in fig. 5, in the iterative computation process of the capacity expansion task, capacity expansion may be performed on the fragment container a to the fragment container F in step 6 by using the data fragments D1 to D8 included in one fragment container a in step 1, and 8 data fragments are uniformly distributed in the plurality of expanded fragment containers.

Firstly, from step 1 to step 2, the 8 data fragments in the fragment container a are respectively selected from fragments D5, D6, D7, D8, and migrated to the fragment container B, wherein the current migration of the data fragment is an iterative computation process, and is not the actual migration of the physical data.

In step 3, the slice D4 in the slice container a and the slice D8 in the slice container B are respectively selected and transferred to the slice container C. From step 3 to step 4, the number of fragments in the fragment containers a to C is sorted from high to low, and then the fragments in the fragment containers with a larger number are sequentially selected for migration.

In step 4, the fragment D3 in the fragment container a and the fragment D7 in the fragment container B are respectively selected as migration fragments, the migration fragments are migrated into a new fragment container D to complete new capacity expansion, and so on, and the process is repeated until step 6, and the fragment containers with the number of fragments more than the average value are preferentially selected to migrate into the new fragment container or the fragment containers with the number of fragments less, and the data fragments in each expanded fragment container are uniformly distributed.

In the capacity reduction task, the data fragments D1 to D8 included in one slice container a in step 1 can be reduced from the slice container a to the slice container F in step 6.

Firstly, sorting the number of the fragments in the fragment containers A to F from low to high from step 6 to step 5, and then sequentially selecting the fragments in the fragment containers with a small number for migration. The slice D6 in the slice container F may be migrated to the slice container a, slice container B or slice container E with a smaller number of slices.

Then, as shown in fig. 5, in step 5, the slice D6 is migrated into the slice container B.

Then, from step 5 to step 4, selecting the fragment container a with a smaller number of fragments or the fragments in the fragment container E for migration, as shown in fig. 5, in step 4, migrating the fragment D2 in the fragment container E into the fragment container a, further completing the capacity reduction task, and so on, and sequentially circulating until step 1, preferably selecting the fragment container with the number of fragments less than the average value to migrate into other fragment containers with a smaller number of fragments, and uniformly distributing the data fragments in each of the reduced fragment containers.

In fig. 5, a ShardKey (ShardKey) is also called a sharding factor, which refers to a service field for sharding data and its value (e.g., a client identification number or a channel number, etc.) for calculating to which service slot and which sharding container the current service data should belong.

Firstly, through the above-mentioned expansion and contraction capacity calculation, the mapping relation between the fragment to be moved and the fragment container, that is, the mapping relation between the fragment slot ID and the fragment container ID is determined, a fragment routing mapping table is obtained, and then the actual physical data migration is performed through the fragment routing mapping table.

In the process of selecting the fragments to be migrated each time, the fragments to be moved are selected from the plurality of fragment containers as much as possible, so that the data volume to be migrated in each expansion and reduction container is minimum in each fragment container. By the method for determining the target fragment routing mapping table, fragments can be uniformly distributed in a target fragment container, so that the required computing power among the servers is kept substantially consistent, hardware resource idling is effectively avoided, and automatic allocation of hardware resources by an automatic cloud platform is facilitated. The relation between the fragments under any number of fragment containers and the fragment container is stored through the obtained target fragment routing mapping table, so that the current M fragment containers can be directly obtained when the capacity is expanded and reduced, and how to migrate the fragments to the positions under the N fragment containers. Because the relation between the data fragments and the fragment container is stored by adopting the fragment routing mapping table, all the fragments to be migrated do not need to be migrated at one time in the migration process, and the migrated fragments and the fragments which are not executed are distinguished by labeling the routing mapping table of the fragments to be migrated so as to realize partial migration of the data and achieve the aim of online capacity expansion.

S13: and determining a to-be-migrated fragment routing mapping table according to the current fragment routing mapping table and the target fragment routing mapping table.

Specifically, the difference between the two fragment route mapping tables is determined as the to-be-migrated fragment route mapping table by comparing the difference between the current fragment route mapping table and the target fragment route mapping table. The fragment to be migrated routing mapping table also includes a mapping relationship between the data fragment to be migrated and the fragment container. And finally, migrating the actual physical data fragment according to the determined routing mapping table of the fragment to be migrated.

Step S2: and sequentially selecting each fragment to be migrated as the current migration fragment according to the route mapping table of the fragment to be migrated, and circularly executing the following locking operation, migration operation and unlocking operation until the fragment to be migrated does not exist in the route mapping table of the fragment to be migrated. Wherein the lockout operation comprises: blocking the execution of the transaction according to the current migration fragment; the migration operation comprises: migrating the current migration fragment after blocking the execution transaction; the unlocking operation comprises the following steps: and after the data migration of the current migration fragment is finished, unlocking the execution transaction.

Specifically, the currently determined routing mapping table of the migration-waiting fragment includes one or more migration-waiting fragments, and one or more migration-waiting fragments are selected as the current migration fragment each time according to the sequence of the routing mapping table of the migration-waiting fragment. And circularly executing the following locking operation, migration operation and unlocking operation until the migration-waiting fragment to be migrated does not exist in the migration-waiting fragment routing mapping table.

In some embodiments, step S2 specifically includes the following steps S21, S22, and S23:

step S21: and executing a blocking operation, wherein the blocking operation comprises blocking execution of the transaction according to the current migration fragment.

Optionally, the blocking of the execution of the transaction may specifically include first-stage transaction blocking and second-stage transaction blocking; in an embodiment of the present invention, as shown in fig. 6, the step S21 may specifically include the following steps S211 and S212:

s211: according to the current migration fragment, executing a first-stage transaction blocking, wherein the first-stage transaction blocking comprises the following steps: and blocking the external transaction entrance related to the current migration fragment, and delaying for N seconds to wait for the transaction processing in execution to be finished.

The first stage transaction block belongs to system entry block, once the block succeeds, any new transaction request which may modify the fragment data of the current migration fragment is not generated, and the aim is to make the on-the-way transaction in each service be executed before the data migration. And, let the control center of the scale and shrink capacity have enough time to notify all services of the migration message, in the first stage transaction lockout if the present system does not meet the lockout condition, for example, there are other slices in the course of migrating at present, or the time window of migrating has already reached, or have happened unusually in the course of carrying out, unlock automatically.

The external transaction entries may include an internal gateway and an external gateway, and blocking the external transaction entries related to the current migration fragment may include: and blocking all newly initiated transaction requests related to fragment data modification for the internal gateway and the external gateway related to the current migration fragment.

Specifically, when the activity parameters are changed, the scale-and-scale control center sends blocking instructions to each transaction gateway service, so that all transactions related to fragment data processing of the current migration fragment are completely closed, especially the update transactions matched according to the URL, and new transaction requests are avoided. And delaying for N seconds to wait for the transaction being executed to be processed, and then starting the subsequent processing steps. The gateway is an entry point of the system, and is not a specific gateway such as Zuul or BSP, but any gateway needs to support integration with the configuration center, and specifically, apollo may be selected as the configuration center inside the scalability control center.

In order to reduce the influence on daily transaction during capacity expansion as much as possible, all newly initiated transaction requests related to fragment data modification of the current migration fragment are blocked on the gateway, and all services are not blocked. In order to effectively distinguish transaction requests, a specific transaction blocking filter, such as ServletFilter, is used on the gateway, which blocks and blocks transaction requests by means of URL configuration items, such as URL wildcard definitions and manifest definitions. For transaction requests which are not within the interception range, normal execution is guaranteed to be unaffected.

The external transaction entry may further include an external message queue and a batch processing task, and block the external transaction entry related to the current migration fragment, and may further include:

for an external message queue directly serving as an external system inlet, blocking transaction at a message production end through a fragment to be migrated routing mapping table.

Specifically, for a message queue directly serving as an entry of an external system, for example, a third-party file or data is directly read and then enters the queue, wherein the internal message queue between internal services of the system is not included, and the queue in which messages are processed by a gateway and then are listed is not included, and a message producer blocks transactions through a fragment route of message enqueuing processing, so that no erroneous message delivery processing is generated during switching. And for the message sending method section at the message queue production end, blocking the message sending method section, wherein the message sending method section identifies which are the entry queues of the external system in the message sending queue through specific annotations, and blocks the identified entry queues. When receiving that the expansion and contraction capacity starting switch sent by the expansion and contraction capacity control center is changed to be ON, directly rejecting a new message to continue the enqueue processing by throwing an exception. And for the consumption end of the message, the message is not processed when the expansion and contraction switch is turned ON, the service is processed continuously, and the message is processed when the transaction blocking of the second stage is carried out subsequently. For the message queue between the internal services of the system, the blocking is not carried out in the first stage so as to ensure that the message queue is operated completely.

For the batch processing task currently executing, the blocking processing is carried out in the batch processing method section, and the new batch processing task is blocked.

Specifically, when the expansion and contraction capacity switch is changed to be ON, a new batch processing task cannot be started, and meanwhile, if the currently executed batch processing task is not completed, a successful transaction blocking identifier is not returned to wait for processing of subsequent personnel. The blocking process is carried out in the batch processing method section of the batch processing task, wherein the batch processing method section identifies which entries are entries of the external system in the batch processing task through specific annotations, and blocks the identified entries, and the current batch processing task can be blocked without invasion by adopting the batch processing method section.

When the expansion and contraction switch is changed to ON, all external transaction inlets of each service in the system are blocked, including blocking of an internal gateway (UI page), an external gateway, an external message queue and a batch processing task. But the transaction between the internal service and the service of the system is not blocked, including the real-time transaction request and the message queue between the internal services, so as to avoid influencing the normal execution and ending of the transaction in transit.

In the process, system management personnel start processing ON an interface of a scaling capacity control center, when each transaction entrance service receives that a scaling capacity starting switch is changed into ON, execution blocking is automatically triggered, and after the transaction entrance service successfully executes the blocking, the blocked state is fed back to the scaling capacity control center, so that the scaling capacity control center can accurately judge whether the blocking is successful, and subsequent steps are triggered ON the condition.

For all the transaction forms, when all the gateways can not be blocked in the first-stage transaction within a specified time, the expansion and shrinkage capacity control center judges that the expansion and shrinkage capacity fails, automatically releases the blockage of all the gateways again, and records corresponding operation logs. Meanwhile, manual intervention processing is needed, the state of the starting expansion and contraction volume switch is changed into OFF, the failure reason of transaction blocking is confirmed and eliminated, and then the processing is restarted.

S212: after the first-stage lockout is successfully performed, performing a second-stage lockout, the second-stage lockout including: and respectively blocking the program interface and the data processing related to the current migration fragment in the database.

Specifically, after the transaction of the first-stage transaction blocking is successful, the scale and shrinkage control center notifies each service to enter the second-stage transaction blocking, and completely shields the processing related to the fragment data of the current migration fragment from two aspects of A Program Interface (API) and a Database (DB). Transaction processing on the fragment number is directly terminated at the API layer, a database script is called at the DB layer, and a trigger for locking the DB-level write operation is generated on a table to be migrated, so that data cannot be changed during migration.

The second stage transaction block belongs to deep block of system internal services, and various types of calls are also carried out between micro service applications, and a transaction entrance does not necessarily contain a fragmentation factor, so that the blocking of the calls between the internal services cannot be guaranteed only by blocking the transaction entrance, wherein the second stage transaction block can be divided into API block and DB block, and the data in the migration processing process can not be changed in the subsequent transaction process after the two blocks, so that the consistency of the data before and after migration is guaranteed. The fragmentation factor refers to a service field for fragmenting data and a value thereof (e.g., a client number, a channel number, etc.), which is used for calculating which fragment, which service slot, and which fragmentation container the current service data should belong to.

The API blocking of the current migration fragment is the first level of barrier of the second-stage transaction blocking, so as to ensure that no matter where the service corresponding to the fragment migration data is called, the service cannot be triggered to be executed. The API blocking is controlled when all transactions need to obtain fragment routes, namely when the second-stage transaction blocking is started, the second-stage transaction blocking identification of each service is directly opened through the first stage, the blocking mechanism takes effect, the scaling control center judges whether the current transaction is the current migration fragment, and if the blocking mechanism takes effect and the current transaction is judged to be the current migration fragment, the current transaction processing is terminated by directly throwing exception. The blocking method may be implemented directly in a fragmentation tool, for example, in a routeBiz method in ShardTools, all clients of the fragmentation container service related to the current migration fragment are forcibly limited by the blocking method, the current processing is forcibly blocked by directly using an exception, and a user corresponding to the fragmentation container service needs to process according to the downstream service exception. In the whole fragment expansion and reduction stage, two fragment routing mapping tables are reserved in the fragment tool, one is the original fragment routing mapping table before expansion and reduction, the other is the temporary fragment routing mapping table which is currently migrated, and the temporary fragment routing mapping table is released only after all the fragments to be migrated are migrated. In order to ensure that the running service in the process of expanding the capacity is not required to be restarted, all API (Restful interface) instances are dynamically generated in advance (but the corresponding service is not in running) during starting, and the corresponding service can be used after being started in the system for capacity expansion.

The partition DB block of the current migration partition is the second pass of the second-stage transaction block, and is used to ensure that the corresponding records in all data tables involved in the migration process of the current migration partition are not modified in any case. Through DB blocking, transactions irrelevant to the current migration fragment or modification of data of the current migration fragment by a fragmentation tool are effectively avoided, and the irrelevant fragmentation tool does not use projects of ShardTools, for example. The DB blocking is to directly realize the blocking of the table operation of the current migration fragment through a database trigger.

When the second stage transaction blocking is started, the expansion and contraction capacity control center is responsible for generating 3 scripts for blocking each table related to fragments of each corresponding library, wherein the scripts comprise an event adding function, an event modifying function and an event deleting function, whether the record related to the current DB operation is the current migration fragment or not is judged according to the existing service main table, such as a data borrowing table, in the scripts, according to the fragment number, and if the record is the current migration fragment, a specific custom database exception is thrown out from the trigger, so that an application program is forced to terminate processing. And if the current migration fragment finishes the migration, unlocking the fragment data of the current migration fragment only by deleting the corresponding trigger script on the table.

In addition, for the internal message queue, at the message producer, the API blocking mechanism of the current migration fragment can prevent a new message from entering the queue, on the other hand, at the message consumer, when the second-stage transaction blocking has been started, for the message which is not processed in the queue, if the fragment factor contained in the message is found to belong to the current migration fragment, including the message which is not processed yet after the first-stage transaction blocking waiting, the message is put into an abnormal queue for temporary non-processing, or the message is considered to be put back into the original queue for manual compensation processing after the migration is finished.

And after the second-order blocking of all related services related to the current migration fragment is successful, feeding back a successful state or a message to the scaling control center to trigger the subsequent processing step of scaling. And when the state is not updated to the success state for a period of time, manual processing is carried out.

And (3) degrading the externally provided service properly before executing the scaling content, and blocking the transactions of a small part of clients to ensure the performance of each piece of fragmented data in migration and the correctness of the migration. Because a plurality of services are operated in the system at the same time, a two-stage transaction blocking mode is selected to improve the reliability of transaction blocking as much as possible, namely, no transaction is ensured to modify the fragment data of the current migration fragment after the transaction blocking.

In order to ensure that new transactions can be blocked, related transactions carried out on the migration fragments are blocked in a two-stage blocking mode, complete transaction blocking in the first stage is directly processed at each gateway layer, and transaction blocking in the second stage is processed at two layers of an API and a database. Through blocking transactions at multiple levels, the correctness and the integrity of the data being migrated are ensured as much as possible. The transaction blocking is executed as far as possible at the front of the transaction chain, so that the complexity of various abnormal scenes in the blocking process is reduced, and the influence on the customer experience is also reduced.

In some embodiments, step S2 specifically includes step S22: and executing a migration operation, wherein the migration operation comprises migrating the selected single current migration fragment after blocking the execution of the transaction.

Optionally, in an embodiment of the present invention, step S22 may specifically include:

performing database-to-database data replication on the fragment data of the current migration fragment according to a database table replication process, completing the replication of the fragment data from the current fragment container to a target fragment container, and if the replication is successful, clearing the fragment data of the current migration fragment in the current fragment container; if the copy fails, the current operation is terminated, and the data migrated to the new container is automatically cleared. If the current operation is terminated and/or the processing action of automatically cleaning the data migrated to the new container also fails, manual processing is carried out.

Specifically, after confirming that the transaction related to the current migration fragment is blocked, the scaling capacity control center starts to execute database-to-database data replication through a database table replication program, wherein the database table replication program may be DataX, and if the replication is successful, the data is cleared and continues to be processed. If the copy fails, the current operation is terminated, manual processing is carried out, the original fragment data is not changed, so that the service is directly recovered, and then the data in the target fragment container needs to be cleaned as required. For the capacity expansion task, a new fragment container is added and new server resources are correspondingly added, and for the capacity reduction task, the target fragment container is a part of the existing current fragment container.

The data corresponding to the current migration fragment realizes off-line migration based on a database table copy program, and only one fragment is migrated at each time, so that the number of affected users and the affected time can be controlled. The migration is to migrate all table data related to the fragment number of the current migration fragment to a new library for the fragment number already stored in the service main table, for example, the fragment number in the borrow table.

Before the migration starts, a sub-table with only a main table main key (e.g. debit number) and two fields of a fragment number, such as TMP _ SD _ MIG _ xxx, is generated based on a service main table corresponding to the current migration fragment, where xxx may be the fragment number, and the main table service data of the current migration fragment is imported into the sub-table. When the fragments are migrated, the full data of the service main table does not need to be associated and inquired, and only the data of the current migration fragments need to be associated and inquired by the sub-table. The method for establishing the sub-table can improve the processing performance by inquiring the sub-table on one hand, and can also avoid the influence of the migration processing on normal business transaction on the other hand.

In order to ensure that the daytime transaction is not influenced to the maximum extent by data migration operation during capacity expansion and contraction, only one piece of fragment data is migrated once, and the service corresponding to the fragment is blocked, so that all services are prevented from being stopped in the process of capacity expansion and contraction blocking, and the transaction requests influenced by the process of capacity expansion and contraction are effectively reduced.

In some embodiments, step S2 specifically includes step S23: performing an unlocking operation, the unlocking operation comprising: and after the data migration of the current migration fragment is finished, unlocking to execute the transaction.

Optionally, the unlocking execution transaction includes first-stage transaction unlocking and second-stage transaction unlocking; in an embodiment of the present invention, as shown in fig. 7, step S23 may specifically include the following steps:

s231: after the data migration of the current migration fragment is completed, executing first-stage transaction unlocking, wherein the first-stage transaction unlocking comprises the following steps: firstly, unblocking a database, and deleting all data records corresponding to the current migration fragment; then, the blocking of the program interface is released, the current migration fragment is removed from the routing mapping table of the fragment to be migrated, and the control of the transaction corresponding to the current fragment factor of the current migration fragment is opened;

s232: after the first-stage transaction unlocking is successfully executed, second-stage transaction unlocking is executed, wherein the second-stage transaction unlocking comprises the following steps: unblocking the external transaction entry and updating the migration processing state.

Specifically, after the migration of the current migration fragment is successful, the blocking on the DB level is released, and all data records corresponding to the current fragment are deleted. Then, the lock on the API layer is released, the current migration fragment is removed from the routing mapping table of the fragment to be migrated, and the control of the transaction corresponding to the current fragment factor is completely opened. Finally, the blocking of the transaction entries such as the gateway is released, and the migration processing state is updated.

Optionally, according to the data migration method for dynamic scaling, before each execution of sequentially selecting each segment to be migrated as the current migration segment, the method further includes:

judging whether the current execution state meets the migration condition;

Specifically, when the scalable capacity control center determines that there is a migration-ready segment that has not completed migration in the migration-ready segment routing mapping table, it continues to determine whether the current execution state meets the migration condition, where the method includes: and judging whether the residual migration time window on the current day is sufficient or not, and/or judging whether the current migration process is abnormal or not. And if the current execution state meets the migration condition, continuing to sequentially select each fragment to be migrated as the current migration fragment, and continuing to sequentially execute the steps S21 to S23 according to the newly determined current migration fragment until the fragment migration task is completed. If the current execution state does not meet the migration condition, automatically pausing the migration processing, suspending the current task after the execution state is saved, and returning to execute the step of judging whether the current execution state meets the migration condition or not after the next migration period.

If the expansion and contraction capacity control center judges that the migration-waiting fragments are not in the migration-waiting fragment routing mapping table, the migration of all the migration-waiting fragments is completed, the task can be considered to be finished, and the dynamic expansion and contraction capacity of the system resources is realized.

In an embodiment of the present invention, when each service process receives a scaling instruction, a successful processing instruction, and a failed processing instruction sent by the scaling control center, the scaling control center still uses, for example, an OpenAPI of a configuration center (Apollo) to feed back a processing state of the service process to the scaling control center, where the processing state includes a state notification and storage function based on the Apollo itself, so that the scaling control center can accurately grasp a situation in real time to determine subsequent step processing.

The method comprises the following steps that a scale and shrink control center collects feedback states of service processes through a change interception interface of a configuration center, wherein the feedback states comprise:

(1) Whether variable capacity basic parameters (a capacity expansion and reduction switch, the number of target partitions and the current migration fragment ID) are received or not;

(2) A1: whether a first stage transaction lockout message is received;

(3) A2: whether the first stage transaction blocking is successful or not;

(4) A3: whether a second stage transaction lockout message is received;

(5) A4: whether API lockout was successful;

(6) A5: whether DB lockout was successful;

(7) A6: whether the migration was successful;

(8) A7: whether the unlocking is successful;

each service feedback information or message is directly recorded in a Namespace (Namespace) in a configuration center (Apollo), for example, and the scale and scale control center monitors the Namespace (Namespace) at regular time to acquire the feedback state of each service, so as to discriminate the overall state of the slice migration process. For example, when the current feedback state is in the A2 phase and the number of activities affecting each service is N, where N = Σ (the number of activities affected by each service in the A1 phase), it can be determined that the first-phase blocking has been performed only if the N A1-phase information is received as the confirmation state, and the A2-phase information is represented as the confirmation state. How many active processes of a service are acquired by the registry (Eureka).

The data migration method of dynamic scaling according to the embodiment of the present invention is described above, and the data migration system and the computer device of dynamic scaling according to the embodiment of the present invention are described below with reference to fig. 8 to 10, respectively.

FIG. 8 is a schematic structural diagram of a data migration system for dynamic scaling according to an embodiment of the present invention. The dynamically scaled data migration system 200 may be applied, for example, in the application scenario shown in fig. 9. As shown in fig. 8, the data migration system 200 for dynamic scaling includes:

a fragment route determining module 210, configured to determine a fragment route mapping table to be migrated, where the fragment route mapping table includes one or more fragments to be migrated; and the number of the first and second groups,

the fragment migration processing module 220 is configured to sequentially select, according to the to-be-migrated fragment routing mapping table, each to-be-migrated fragment as a current migration fragment; sequentially controlling the locking submodule 221, the migration submodule 222 and the unlocking submodule 223 to execute circularly until the to-be-migrated fragment to be migrated does not exist in the to-be-migrated fragment routing mapping table;

the blocking sub-module 221 is used for blocking the execution transaction according to the to-be-migrated fragmentation routing mapping table;

the migration submodule 222 is configured to, after blocking execution of a transaction, migrate a current migration fragment;

and the unlocking submodule 223 is configured to unlock the execution transaction after the data migration of the current migration fragment is completed.

Optionally, in an embodiment of the present invention, the fragment route determining module 210 is specifically configured to:

determining a current fragmentation routing mapping table according to a current fragmentation container of a current active partition and fragments contained in each current fragmentation container;

and determining the to-be-migrated fragment routing mapping table according to the current fragment routing mapping table and the target fragment routing mapping table.

Optionally, in an embodiment of the present invention, the fragment route determining module 210 is further specifically configured to:

Optionally, in an embodiment of the present invention, the lockout sub-module 221 is specifically configured to:

Optionally, the external transaction entry includes an internal gateway and an external gateway;

blocking the external transaction entrance related to the current migration fragment, comprising: and blocking all newly initiated transaction requests related to fragment data modification for the internal gateway and the external gateway related to the current migration fragment.

Optionally, the external transaction entry further includes an external message queue and a batch processing task;

blocking the external transaction entrance related to the current migration fragment, further comprising: for an external message queue, blocking transaction at a message production end through a fragment route; for the batch processing task currently being executed, the blocking processing is carried out in the batch processing method section, and the new batch processing task is blocked.

Optionally, in an embodiment of the present invention, the migration sub-module 222 is specifically configured to:

according to a database table replication process, performing database-to-database data replication on the fragment data of the current migration fragment, and completing the replication of the fragment data from the current fragment container to the target fragment container;

if the copying is successful, the fragment data of the current migration fragment in the current fragment container is cleared;

Optionally, in an embodiment of the present invention, the unlocking sub-module 223 is specifically configured to:

Optionally, in an embodiment of the present invention, before each time the to-be-migrated segment is sequentially selected as the current migration segment, the segment routing determining module 210 is further specifically configured to:

judging whether the current execution state meets the migration condition or not;

if the migration condition is not met, automatically pausing the migration processing, suspending the current task after the execution state is saved, waiting for the next migration period, and returning to execute the step of judging whether the current execution state meets the migration condition or not.

Optionally, the determining whether the current execution state meets the migration condition includes:

and/or judging whether the current migration process is abnormal or not.

In an embodiment of the present invention, the fragment routing determining module 210, the fragment migration processing module 220, and the unlocking sub-module 223 in the data migration system with dynamic scaling may be disposed in a scaling control center, where the scaling control center is an independent service, and is used to perform overall control and scheduling on the scaling process, process state storage, and overall coordination processing of the fragment data migration process among the services. It includes the following functions: managing a routing mapping table of the fragments to be migrated, monitoring a scaling process, exception handling, synchronizing messages, configuring parameters and storing process states.

The logical relationship between the scalability control center and the surrounding business services, as well as the basic support service, is shown in fig. 9. The final deployment of the scale and shrink control center comprises three services:

the shardctrlggr is a core service of the scale and shrink control center, and may be, for example, a fragment route determination module 210 in the data migration system of dynamic scale and shrink;

the ShardCtrLMgr-Web is a UI interface of the expansion and contraction capacity control center;

the shardmirite is a service dedicated to completing the data fragment migration process, and may be, for example, a migration sub-module 222 in a data migration system of dynamic scaling capacity;

and an Agent (Agent) embedded in each service (including each server, gateway and batch task in fig. 9) and responsible for transaction blocking and unlocking, state collection and the like, for example, a blocking submodule 221 and an unlocking submodule 223 in the data migration system with dynamic scaling. In addition, a configuration center (e.g., apollo which may be open source) and a registration center (e.g., eureka which may be open source) may be used in the scaling control center to implement parameter synchronization of the scaling process and service state awareness of each slice container. In fig. 9, two databases DB1 and DB2 are schematically shown to carry fragmented data, and now a new database DB3 is added to perform capacity expansion to share the storage of the existing data.

The whole process control is triggered and monitored by a scaling control center (shardctrlggr), which converts the received scaling command (corresponding to the "operation" in fig. 9) of the system administrator into a specific identification configuration item, such as a scaling switch. And then writing the expansion capacity instruction into a public configuration item group appointed in the configuration center through an OpenAPI of the configuration center, and notifying a message with changed identification to all the fragment container services in the cluster of the fragment container services by using a parameter synchronization mechanism of the configuration center so as to trigger subsequent blocking processing. After the fragment container service is restarted, the processing state of each fragment data can be acquired from the scale and reduction control center to determine how to process the subsequent fragment data.

The information of the configuration center is automatically intercepted by each fragment container service through an embedded Agent program (Agent), and an instruction, a fragment task state and self-scheduling control are read from the configuration center, so that the expansion and contraction capacity control center does not need to know the details of internal transaction blocking of each fragment container service, and the coupling degree with the service is reduced.

When the configuration of the number of the active partitions is changed, the message of parameter change needs to be pushed to each service in the cluster as soon as possible in time so as to keep the consistency of the parameters in all the services as much as possible. Therefore, the data migration system of the dynamic expansion and contraction capacity synchronously pushes related parameter values such as expansion and contraction capacity switches and the number of active partitions to each partitioned container service based on public parameters (Namespace which is mainly used by an expansion and contraction capacity control center and is related to other services) of a configuration center and OpenAPI of the configuration center. However, since it is impossible to switch all service parameters at the same time, the number of current fragment containers and the number of target fragment containers need to be strictly distinguished when the active partition parameters are applied, so as to ensure that all on-the-way transactions are not called to the wrong partition due to parameter change in the switching moment of the parameters.

And after a configuration center in the capacity expansion and reduction control center receives the routing mapping table of the fragment to be migrated determined by the data migration method of the dynamic capacity expansion and reduction, and processes a result message, initializing a capacity expansion and reduction task flow in the data migration method of the dynamic capacity expansion and reduction, and starting subsequent processing logic according to the steps. When the first-stage transaction is blocked to the second-stage transaction is blocked, each service in the cluster is informed to start the blocking processing based on the configuration center pushing mechanism.

When the service receiving the message is restarted or a new service process is added in the processing process, the state of the current task can still be read from the configuration center, and corresponding processing, namely blocking transaction or unlocking transaction, is executed according to the state of the current task. The number of the key parameter active partitions of the partition is not directly placed in the engineering configuration of a developer, but placed in a configuration center, and necessary change authority control is given to avoid unnecessary problems caused by wrong configuration in development or test.

According to the data migration method and system for dynamic capacity expansion, the relation between the data fragments and the fragment container is stored by adopting the fragment routing mapping table, the migration tasks of all fragments to be migrated can be completed in one step in the migration process, and the migrated fragments and the unexecuted fragments can be distinguished by labeling the fragment routing mapping table, so that partial migration of data is realized, and the aim of online capacity expansion is fulfilled.

The fragment migration paths are calculated uniformly for a small number of times, and the fragment routing mapping table is determined according to the calculated fragment migration paths, so that uniform distribution of fragments is ensured, the required calculation force among all servers can be kept generally consistent, hardware resource idling is effectively avoided, and meanwhile, automatic allocation of hardware resources by an automatic cloud platform is facilitated.

In order to ensure that the daytime transaction is not influenced to the maximum extent by data migration operation during capacity expansion and contraction, the method adopts a mode of migrating only one piece of fragment data at a time and blocking the service corresponding to the fragment, avoids stopping all services in the process of capacity expansion and contraction blocking, and effectively reduces the transaction requests influenced by the capacity expansion and contraction process.

In order to ensure that the running service in the process of expanding the capacity is not required to be restarted, all API (Restful interface) instances are dynamically generated in advance (but the corresponding service is not in running) during starting, and the corresponding service can be used after being started in the system for capacity expansion.

By supporting breakpoint relay, the task is suspended and the task is redone from the error step.

The dynamic capacity expansion and reduction data migration method, the system and the computer equipment provided by the embodiment of the invention can realize the application of a service system and the dynamic capacity expansion and reduction processing of data nodes, provide the characteristics of data range control, capacity expansion and reduction progress control, interruptible rollback and the like, and ensure the goals of no influence on the service transaction of IT system clients and controllability of system operation and maintenance personnel.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when being executed by a processor, the computer program realizes the steps of the data migration method of the dynamic scaling capacity.

The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, and software distribution medium, etc. Of course, there are other ways of storing media that can be read, such as quantum memory, graphene memory, and so forth. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

FIG. 10 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device executes a data migration method of dynamic scaling. As shown in fig. 10, the computer device 400 includes an input device 401, an input interface 402, a central processor 403, a memory 404, an output interface 405, and an output device 406. The input interface 402, the central processing unit 403, the memory 404, and the output interface 405 are connected to each other through a bus 410, and the input device 401 and the output device 406 are connected to the bus 410 through the input interface 402 and the output interface 405, respectively, and further connected to other components of the computer device 400.

Specifically, the input device 401 receives input information from the outside and transmits the input information to the central processor 403 through the input interface 402; the central processor 403 processes the input information based on computer-executable instructions stored in the memory 404 to generate output information, stores the output information temporarily or permanently in the memory 404, and then transmits the output information to the output device 406 through the output interface 405; output device 406 outputs the output information external to computer device 400 for use by a user.

In one embodiment, the computer device 400 shown in FIG. 10 may be implemented as a dynamically scaled data migration system that may include: a memory configured to store a program; and a processor configured to execute the program stored in the memory to execute the data migration method of dynamic scaling capacity described in the above embodiments.

According to an embodiment of the invention, the process described above with reference to the flow chart may be implemented as a computer software program. For example, embodiments of the invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network, and/or installed from a removable storage medium.

In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions which, when run on a computer, cause the computer to perform the methods described in the various embodiments above. The procedures or functions according to the embodiments of the invention are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), among others.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and these modifications or substitutions do not depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamically scaling data migration, the method comprising:

sequentially selecting each fragment to be migrated as a current migration fragment according to the route mapping table of the fragments to be migrated, and circularly executing blocking operation, migration operation and unlocking operation until the fragment to be migrated does not exist in the route mapping table of the fragments to be migrated;

wherein the lockout operation comprises: blocking the execution of the transaction according to the current migration fragment; the migration operation comprises: migrating the current migration fragment after blocking the execution transaction; the unlocking operation comprises the following steps: after the data migration of the current migration fragment is completed, unlocking the execution transaction;

the determining the routing mapping table of the fragment to be migrated includes:

determining a target fragment routing mapping table according to the current fragment routing mapping table and a target fragment container of a planned active partition;

determining the fragment route mapping table to be migrated according to the current fragment route mapping table and the target fragment route mapping table;

the determining a target fragment routing mapping table according to the current fragment routing mapping table and the target fragment container of the planned active partition specifically includes:

and determining migration directions of the fragments in the current fragment container corresponding to the target fragment container according to the expanding and contracting task and the average value of the number of the fragments and the sequence of the current fragment container, and determining a target fragment routing mapping table.

2. The dynamically scaled capacity data migration method according to claim 1, wherein said blocking execution of transactions according to said current migration slice comprises:

executing a first-stage transaction lockout according to the current migration fragment, wherein the first-stage transaction lockout comprises: blocking an external transaction entrance related to the current migration fragment, and delaying for N seconds to wait for the transaction processing in execution to be finished;

after the first phase transaction lockout is successfully performed, performing a second phase transaction lockout, the second phase transaction lockout comprising: and respectively blocking program interfaces and data processing related to the current migration fragment in the database.

3. The dynamically scalable data migration method of claim 2, wherein the outbound transaction portal comprises an internal gateway and an external gateway;

blocking the external transaction entrance related to the current migration fragment, including:

4. The dynamically scalable data migration method according to claim 3, wherein the external transaction entry further comprises an external message queue and a batch processing task;

the blocking of the external transaction entrance related to the current migration fragment further comprises:

for the external message queue, blocking transaction at a message production end through the fragment to be migrated routing mapping table;

5. The dynamic scaling-up data migration method according to claim 1, wherein the migrating the current migration fragment after blocking the execution transaction specifically includes:

according to a database table replication process, performing database-to-database data replication on the fragment data of the current migration fragment, and completing the replication of the fragment data from the current fragment container to a target fragment container;

6. The dynamically scalable data migration method according to claim 2, wherein unlocking the execution transaction after completing the data migration of the current migration segment comprises:

after the data migration of the current migration fragment is completed, executing first-stage transaction unlocking, wherein the first-stage transaction unlocking comprises the following steps: firstly, the blockage of the database is removed, and all data records corresponding to the current migration fragment are deleted; then, the blocking of the program interface is released, the current migration fragment is removed from the routing mapping table of the fragment to be migrated, and the control of the transaction corresponding to the current fragment factor of the current migration fragment is opened;

after the first-stage transaction unlocking is successfully executed, second-stage transaction unlocking is executed, wherein the second-stage transaction unlocking comprises the following steps: unblocking the external transaction entry and updating a migration processing state.

7. The dynamic capacity-expanding data migration method according to claim 1, further comprising, before each execution of the sequential selection of the fragments to be migrated as the current migration fragment:

if the migration conditions are met, executing the sequential selection of the fragments to be migrated as the current migration fragments;

8. The dynamic scaling data migration method according to claim 7, wherein the determining whether the current execution state satisfies the migration condition comprises:

and/or judging whether the current migration process is abnormal or not.

9. A dynamically scaled data migration system, comprising:

the device comprises a fragment routing determining module, a fragment routing determining module and a fragment migration determining module, wherein the fragment routing determining module is used for determining a fragment routing mapping table to be migrated, and the fragment routing mapping table to be migrated comprises one or more fragments to be migrated; and the number of the first and second groups,

the fragment migration processing module is used for sequentially selecting each fragment to be migrated as the current migration fragment according to the fragment to be migrated route mapping table; sequentially controlling the locking submodule, the migration submodule and the unlocking submodule to be circularly executed until the to-be-migrated fragment to be migrated does not exist in the to-be-migrated fragment routing mapping table;

the blocking submodule is used for blocking the execution of transaction according to the current migration fragment;

the migration sub-module is used for migrating the current migration fragment after blocking the execution transaction;

the unlocking submodule is used for unlocking the execution transaction after the data migration of the current migration fragment is completed;

the fragment route determining module is specifically configured to:

determining a fragment route mapping table to be migrated according to the current fragment route mapping table and the target fragment route mapping table;

wherein, the determining a target fragment route mapping table according to the current fragment route mapping table and a target fragment container of a planned active partition specifically includes:

10. The system of claim 9, wherein the lockout sub-module is specifically configured to:

after the first-stage lockout is successfully performed, performing a second-stage lockout, the second-stage lockout including: respectively blocking the program interface and the data processing related to the current migration fragment in the database.

11. A computer device, characterized in that it comprises:

one or more processors;

storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the dynamically scaled data migration method of any of claims 1-8.

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the dynamic scaled-up data migration method according to any one of claims 1 to 8.