CN114328458A - Data migration method and device and computer storage medium - Google Patents
Data migration method and device and computer storage medium Download PDFInfo
- Publication number
- CN114328458A CN114328458A CN202111646319.8A CN202111646319A CN114328458A CN 114328458 A CN114328458 A CN 114328458A CN 202111646319 A CN202111646319 A CN 202111646319A CN 114328458 A CN114328458 A CN 114328458A
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- source
- incremental
- batch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure provides a data migration method and device, a computer storage medium, electronic equipment and a computer program product, and belongs to the technical field of computers. The data migration method comprises the following steps: under the condition that the data migration type selected by data migration between a source data end and a target data end is incremental data migration, acquiring a preconfigured incremental migration index table, wherein the incremental migration index table at least comprises a first source table primary key column, and each row in the first source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in a source data table; and performing incremental data migration between the source data end and the target data end according to each primary key value of the source table in the incremental migration index table.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data migration method and apparatus, a computer storage medium, an electronic device, and a computer program product.
Background
With the development of services, the data volume is increasing day by day, the database list table of the source data end has hundreds of GB, even the data of the single table TB level is a common phenomenon, and the performance of the service server is also tested by the huge data volume. Therefore, how to migrate the data of the source data end to the target data end for storage to ensure the data consistency between the source data end and the target data end becomes a technical problem to be solved urgently.
Disclosure of Invention
Embodiments of the present disclosure provide a data migration method and apparatus, a computer storage medium, an electronic device, and a computer program product, so as to overcome or alleviate the above technical problems in the prior art.
The technical scheme adopted by the disclosure is as follows:
a method of data migration, comprising:
under the condition that the data migration type selected by data migration between a source data end and a target data end is incremental data migration, acquiring a preconfigured incremental migration index table, wherein the incremental migration index table at least comprises a first source table primary key column, and each row in the first source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in a source data table;
and performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table.
Optionally, the method further comprises: before incremental data migration is performed between the source data end and the target data end, under the condition that the data migration type selected by data migration performed between the source data end and the target data end is full data migration,
acquiring a pre-configured full migration index table, wherein the full migration index table at least comprises a second source table primary key column, and each row in the second source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in the source data table;
and carrying out full data migration between the source data end and the target data end according to each source table primary key value in the full migration index table.
Optionally, the incremental migration index table and the full migration index table each further include: and the integer self-increment column is used for configuring the transferable data volume of each batch during batch migration.
Optionally, the incremental migration index table further includes: a data operation type, wherein the data operation type comprises at least one of insertion, deletion and update.
Optionally, the performing full data migration between the source data end and the target data end according to each source table primary key value in the full migration index table includes: and performing batch full migration of data between the source data end and the target data end according to each batch of transferable data size configured in the full migration index table and the corresponding primary key value of the source table and according to each batch of transferable data size.
Optionally, the performing data migration between the source data end and the target data end according to each source table primary key value in the full migration index table further includes: in the process of batch full-scale data migration between the source data end and the target data end, monitoring the data which is subjected to full-scale data migration at the source data end to obtain corresponding incremental data;
and performing batch incremental migration of the incremental data between the source data end and the target data end according to each batch of migratable data volume configured in the incremental migration index table and the corresponding source table primary key value and according to the data operation type and each batch of migratable data volume.
Optionally, the monitoring the data migrated in full at the source data end to obtain corresponding incremental data includes: and monitoring the data which is migrated in full at the source data end according to a set trigger so as to obtain corresponding incremental data.
Optionally, the performing batch full migration of data between the source data end and the target data end according to each batch of migratable data size configured in the full migration index table and the corresponding source table primary key value according to each batch of migratable data size includes:
determining the data volume of the data which is successfully migrated in the previous batch to determine the starting point data when the data in the next batch is migrated, and performing batch full-scale migration of the data between the source data end and the target data end according to the data volume which is configured in the full-scale migration index table and the source table primary key value corresponding to the starting point data and the data volume which is configured in the full-scale migration index table.
Optionally, the performing, according to the data operation type and each batch of migratable data volume, batch incremental migration of the incremental data between the source data end and the target data end includes: aiming at the incremental migration data, the data operation type is newly added data which is inserted and updated, and batch data replacement of the incremental data is carried out between the source data end and the target data end through a data replacement instruction and a source table primary key value corresponding to a maximum integer self-increment value corresponding to each newly added data.
Optionally, the performing, according to the data operation type and each batch of migratable data volume, batch incremental migration of the incremental data between the source data end and the target data end includes: and for the newly added data with the data operation type of deletion in the data subjected to incremental migration, performing batch data deletion of the incremental data between the source data end and the target data end according to the transferable data volume of each batch.
A data migration apparatus, comprising:
the data migration method includes the steps that a first processing unit is used for obtaining a preconfigured incremental migration index table under the condition that a data migration type selected by data migration between a source data end and a target data end is incremental data migration, the incremental migration index table at least comprises a first source table primary key column, and each row in the first source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in a source data table;
and the second processing unit is used for performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table.
A computer storage medium having stored thereon a computer-executable program that is executed to implement a method according to any one of the embodiments of the present disclosure.
An electronic device comprising a memory for storing thereon a computer-executable program and a processor for executing the computer-executable program to implement a method according to any of the embodiments of the present disclosure.
A computer program product comprising a computer program which, when executed by a processor, implements a method according to any one of the embodiments of the present disclosure.
In the embodiment of the disclosure, under the condition that the data migration type selected by data migration between a source data end and a target data end is incremental data migration, a preconfigured incremental migration index table is obtained, the incremental migration index table at least comprises a first source table primary key column, and each row in the first source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in a source data table; and performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table, so that the incremental data migration is performed based on the incremental migration index table instead of the source data table, and the table locking phenomenon is avoided, thereby realizing the incremental data migration and ensuring the data consistency between the source data end and the target data end.
Drawings
Fig. 1A schematically illustrates a schematic diagram of an application scenario according to an embodiment of the present disclosure.
Fig. 1B schematically illustrates a schematic diagram of another application scenario of the embodiment of the present disclosure.
Fig. 1C schematically illustrates a schematic diagram of still another application scenario of an embodiment of the present disclosure.
Fig. 2 schematically shows a flow chart of a data migration method according to an embodiment of the present disclosure.
Fig. 3 schematically shows a flowchart of a data migration method according to an embodiment of the present disclosure.
Fig. 4 schematically shows a flowchart of a data migration method according to an embodiment of the present disclosure.
Fig. 5 schematically shows a schematic structural diagram of a data migration apparatus according to an embodiment of the present disclosure.
Fig. 6 schematically shows a schematic view of an electronic device of an embodiment of the disclosure.
Detailed Description
To make the technical problems, technical solutions and advantages to be solved by the present disclosure clearer, the following detailed description is made with reference to the accompanying drawings and specific embodiments.
Fig. 1A schematically illustrates a schematic diagram of an application scenario according to an embodiment of the present disclosure. As shown in fig. 1A, the application scenario includes a source data end and a target data end, in this embodiment, the source data end is specifically a source data table, and the target data end is specifically a target data table. The source data table and the target data table are set based on Mysql data, for example, when data in the source data table is modified and changed, the change is migrated to the target data table, for example, the data in the source data table is saved to the target data table in an off-line or real-time manner.
In this embodiment, data migration is performed between the target data table and the source data table one to one.
Fig. 1B schematically illustrates a schematic diagram of another application scenario of the embodiment of the present disclosure. As shown in fig. 1B, the application scenario includes a source data end and a target data end, in this embodiment, the source data end is specifically a source data table, and the target data end includes a plurality of target data tables, where the plurality of target data tables may be on the same target server only or on different target servers.
Unlike the embodiment of fig. 1A, in this embodiment, data migration is performed between the target data table and the source data table in a one-to-many manner.
Fig. 1C schematically illustrates a schematic diagram of still another application scenario of an embodiment of the present disclosure. As shown in fig. 1C, two servers need to perform data synchronization, one is a source server (which may also be referred to as a source database instance) serving as a source data end, and the other is a target server (which may also be referred to as a target database instance) serving as a target data end. A plurality of source data tables are arranged on the source server, a plurality of target data tables are arranged on the target server, and data migration is carried out between the source data tables and the target data tables so as to migrate the whole data in the source data tables to the corresponding target data tables).
In this embodiment, data migration is performed between the target data table and the source data table many to many.
In addition, similar to fig. 1C, when data migration is performed between the target data table and the source data table many-to-many, in another embodiment, the target data tables may be located in different servers, and the source data table may also be located in different servers, or referred to as distributed migration of data.
The server is, for example, a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, web services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
In one scheme, data migration is directly performed based on a source data table, a table locking phenomenon occurs in the migration process, so that migration cannot be continued, data consistency between a target data table and the source data table cannot be guaranteed, and the use of a service scene is difficult to meet. In addition, because a table locking phenomenon occurs, incremental data migration cannot be achieved, and further data consistency between the target data table and the source data table cannot be guaranteed.
In the following embodiments of the present disclosure, when a data migration type selected by data migration between a source data end and a target data end is incremental data migration, a preconfigured incremental migration index table is obtained, where the incremental migration index table at least includes a first source table primary key column, and each row in the first source table primary key column may be used to record a source table primary key value corresponding to data to be migrated in a source data table; and performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table, so that the occurrence of a table locking phenomenon is avoided based on the incremental migration index table instead of the source data table, thereby realizing the migration of incremental data and ensuring the data consistency between the source data end and the target data end.
The following is an exemplary description of the data migration process in the application scenario, however, it should be noted that the following embodiments are only exemplary and not limiting, and it will be apparent to those skilled in the art that other substitutions or modifications may be made to the following steps without departing from the spirit of the present disclosure.
Fig. 2 schematically shows a flow chart of a data migration method according to an embodiment of the present disclosure. As shown in fig. 2, in the data migration method provided in this embodiment, the full data migration is performed between the source data end and the target data end. Specifically, the data migration method of the present embodiment includes:
s201, under the condition that the data migration type selected by data migration between the source data end and the target data end is incremental data migration, acquiring a preconfigured incremental migration index table.
In this embodiment, the incremental migration index table (also referred to as an incremental temporary table) at least includes a first source table primary key column, each row in the first source table primary key column may be used to record a source table primary key value corresponding to data to be migrated in a source data table, and the source table primary key value is also referred to as an ID of the data to be migrated.
Optionally, the incremental migration index table may further include: the integer self-increment column is used for configuring the transferable data volume of each batch during batch migration, so that the incremental data migration of the batches can be realized during the execution of the step S202, the resource consumption is reduced, the smooth migration of the data is ensured, and the migration process is not sensible to users.
Optionally, the incremental migration index table may further include: a data operation type including at least one of an insert (also known as insert), a delete (also known as delete), and an update (also known as update). The data manipulation type is abbreviated as dm (data manipulation). The new addition of a piece of data in the source data table can be realized through insertion, the old data in the source data table can be deleted through deletion, and the existing piece of data in the source data table can be modified through updating. By including the data operation type in the incremental migration index table, incremental data migration can be performed in a targeted manner according to the data operation type when the subsequent step S202 is executed, so that accuracy and efficiency of data migration are ensured.
Specifically, three fields may be configured for the incremental migration index table, which are respectively: and the incremental migration index table can record a source table main key value corresponding to the data to be migrated in the source data table, configure each batch of migratable data volume and record the data operation type. Illustratively, the value of the data operation type field: is 1, represents insert; is 2, representing udpate; is 3, representing delete.
S202, performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table.
Optionally, in this embodiment, the performing, according to each source table primary key value in the incremental migration index table, incremental data migration between the source data end and the target data end may include: and performing batch incremental migration of the incremental data between the source data end and the target data end according to the data operation type and the transferable data volume of each batch.
As previously described, considering that the increments are likely due to the different data operation types described above, for this reason, in performing step S202, the increment migration is carried out in conjunction with the different data operation types. In addition, in order to reduce the consumption of resources by migration, batch migration is performed during incremental migration.
To this end, optionally, the performing batch incremental migration of the incremental data between the source data end and the target data end according to the data operation type and each batch of migratable data volume includes: aiming at the incremental migration data, the data operation type is newly added data which is inserted and updated, and batch data replacement of the incremental data is carried out between the source data end and the target data end through a data replacement instruction and a source table primary key value corresponding to a maximum integer self-increment value corresponding to each newly added data. Specifically, the format of the data replacement instruction is, for example, replace to.
In this embodiment, batch migration may be performed specifically based on the integer self-increment column in the incremental migration index table. Specifically, as mentioned above, the integer self-increment column is used for configuring each batch of migratable data volume during batch migration, and for this reason, each batch of incremental migration is completed, in the integer self-increment column, a self-increment value corresponding to the newly added data is formed in an incremental manner, for example, if the batch of migratable data volume is 1000 pieces, after each batch of incremental data migration is successfully completed, the self-increment value is increased by 1000 pieces, that is, the completion of the incremental migration of 1000 pieces of newly added data is indicated, and for this reason, in the process of performing the incremental migration, the data operation type is the newly added data that is inserted and updated, and the batch data replacement of the incremental data can be directly realized by the source table primary key value corresponding to the maximum integer self-increment value corresponding to each newly added data. Meanwhile, due to the existence of the maximum integer value, even if failure occurs or the migration process is suspended in the migration process of a batch of data, the data migration can be carried out again based on the maximum integer value as the starting point of the migration without completely carrying out the data migration from the first piece of data to be migrated, so that the efficiency of the data migration is improved.
Optionally, the performing, according to the data operation type and each batch of migratable data volume, batch incremental migration of the incremental data between the source data end and the target data end includes: and for the newly added data with the data operation type of deletion in the data subjected to incremental migration, performing batch data deletion of the incremental data between the source data end and the target data end according to the transferable data volume of each batch. When batch deletion is performed in this step, the amount of batch deletion data may be also implemented with reference to the integer self-increment.
In addition, the processes of batch data deletion and batch data replacement are not limited to a specific time sequence, and may be performed sequentially or in parallel. Further, the above-described processes of batch data deletion and batch data replacement may be assigned to the same task execution or may be assigned to different task executions.
Specifically, when the amount of data to be migrated is large, in order to improve migration efficiency, a plurality of tasks may be created, and the steps in fig. 2 described above may be executed in parallel to implement parallel migration.
Fig. 3 schematically shows a flowchart of a data migration method according to an embodiment of the present disclosure. As shown in fig. 3, in the data migration method provided in this embodiment, before performing incremental data migration between the source data end and the target data end, a processing procedure of performing full data migration on data is performed, specifically, in this embodiment, when a data migration type selected by performing data migration between the source data end and the target data end is full data migration, the data migration method includes:
s301, a preconfigured full migration index table is obtained, the full migration index table at least includes a second source table primary key column, and each row in the second source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in the source data table.
Optionally, the full migration index table (also referred to as a full temporary table) may further include: and the integer self-increment column is used for configuring the transferable data volume of each batch during batch migration. Accordingly, incremental migration of the sub-batches can be realized when step S302 is executed, resource consumption is reduced, and smooth migration of data is ensured, and the migration process is not perceived by the user.
Specifically, two fields may be configured for the incremental migration index table, which are respectively: and the main key field and the integer self-increment field are respectively filled with a main key value and configured with each batch of transferable data volume during batch migration, so that the full-volume migration index table can record a source table main key value corresponding to the data to be migrated in the source data table and configure each batch of transferable data volume.
S302, carrying out full data migration between the source data end and the target data end according to each source table primary key value in the full migration index table.
And S303, monitoring the data which is completely migrated at the source data end to obtain corresponding incremental data.
Optionally, the monitoring the data migrated in full at the source data end to obtain corresponding incremental data includes: and monitoring the data which is migrated in full at the source data end according to a set trigger to obtain corresponding incremental data so as to execute the incremental migration processing shown in the above fig. 2.
The specifically set triggers may include an insertion trigger, an update trigger, and a deletion trigger, so as to monitor the data that has been migrated in full at the source data end, and record the corresponding data operation type and the corresponding primary key value in the data operation type field and the primary key field of the incremental migration index when it is monitored that there is data insertion, data update, and data deletion in the source data table. The trigger may specifically be set on the source data table.
Optionally, when the selected data migration type is full data migration, performing full data migration between the source data end and the target data end according to each source table primary key value in the full migration index table includes: and performing batch full migration of data between the source data end and the target data end according to each batch of transferable data size configured in the full migration index table and the corresponding primary key value of the source table and according to each batch of transferable data size.
Optionally, the performing batch full migration of data between the source data end and the target data end according to each batch of migratable data size configured in the full migration index table and the corresponding source table primary key value according to each batch of migratable data size includes:
determining the data volume of the data which is successfully migrated in the previous batch to determine the starting point data when the data in the next batch is migrated, and performing batch full-scale migration of the data between the source data end and the target data end according to the data volume which is configured in the full-scale migration index table and the primary key value of the source table corresponding to the starting point data and the data volume which is configured in the full-scale migration index table.
Specifically, batch migration is performed based on the full migration index table including an integer self-increment column. Specifically, as described above, the integer self-increment column is used for configuring each batch of migratable data volume during batch migration, for this reason, each time a batch of full-volume migration is completed, in the integer self-increment column, a self-increment value corresponding to migration success data is formed in an incremental manner, for example, if the number of migratable data volumes of each batch is 1000, after each batch of incremental data migration is successfully completed, the self-increment value is increased by 1000, that is, it indicates that the full-volume migration of 1000 pieces of data is completed, and the batch data replacement of the full-volume data can be directly realized with the maximum integer self-increment value. Meanwhile, due to the existence of the maximum integer value, even if failure occurs or the migration process is suspended in the migration process of a batch of data, the data migration can be carried out again based on the maximum integer value as the starting point of the migration without completely carrying out the data migration from the first piece of data to be migrated, so that the efficiency of the data migration is improved.
Fig. 4 schematically shows a flowchart of a data migration method according to an embodiment of the present disclosure. As shown in fig. 4, it includes:
s401, creating and configuring a full migration index table and an increment migration index table.
Step S401 includes: adding two fields in the increment migration index table, wherein the two fields are respectively as follows: a primary key field and an integer self-increment field, wherein three fields are added to the increment migration index table, and the three fields are respectively as follows: a primary key field, a data operation type field and an integer self-increment field.
S402A, acquiring the full data from the source data terminal based on the full migration index table.
S403A, loading the acquired data to be migrated related to the full migration to the target data end.
S404A, updating the maximum auto-increment value in the integer auto-increment field of the full-scale migration index table for the next batch of full-scale migration.
S402B, monitoring and acquiring incremental data in the data which is completely migrated through a set trigger, and acquiring the incremental data from the source data terminal based on an incremental migration index table.
And S403B, migrating the incremental data to the target data terminal.
As described above, the incremental data is migrated to the target data side with reference to the above-described processing of batch data deletion and batch data replacement.
And S404B, updating the maximum auto-increment value in the integer auto-increment field of the increment migration index table for the next batch of increment migration.
After the full and incremental migration are completed, the following clean-up steps are performed.
And S405, deleting the trigger.
S406, deleting the increment migration index table.
And S407, deleting the full migration index table.
In this embodiment, the execution sequence of steps S405 to S407 is not particularly limited.
By executing the cleaning steps, the storage space is saved, the resource consumption is reduced, and the data can be created when the data needs to be migrated again.
Fig. 5 schematically shows a schematic structural diagram of a data migration apparatus according to an embodiment of the present disclosure. As shown in fig. 5, it includes:
a first processing unit 501, configured to obtain a preconfigured incremental migration index table when a data migration type selected by data migration between a source data end and a target data end is incremental data migration, where the incremental migration index table at least includes a first source table primary key column, and each row in the first source table primary key column may be used to record a source table primary key value corresponding to data to be migrated in a source data table;
a second processing unit 502, configured to perform incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table.
Optionally, the first processing unit 501 is further configured to, before performing incremental data migration between the source data end and the target data end, obtain a preconfigured full migration index table when the data migration type selected by performing data migration between the source data end and the target data end is full data migration, where the full migration index table at least includes a second source table primary key column, and each row in the second source table primary key column may be used to record a source table primary key value corresponding to data to be migrated in the source data table.
The second processing unit 502 is further configured to perform full data migration between the source data end and the target data end according to each source table primary key value in the full migration index table.
Optionally, the incremental migration index table and the full migration index table each further include: and the integer self-increment column is used for configuring the transferable data volume of each batch during batch migration.
Optionally, the incremental migration index table further includes: a data operation type, wherein the data operation type comprises at least one of insertion, deletion and update.
Optionally, in a case that the selected data migration type is full data migration, the second processing unit 502 is specifically configured to perform batch full data migration between the source data end and the target data end according to each batch of migratable data volume configured in the full migration index table and a corresponding source table primary key value and according to each batch of migratable data volume.
Optionally, the second processing unit 502 is specifically configured to, in a process of batch full migration of data between the source data end and the target data end, monitor the data that has completed full migration at the source data end to obtain corresponding incremental data; and performing batch incremental migration of the incremental data between the source data end and the target data end according to each batch of migratable data volume configured in the incremental migration index table and the corresponding source table primary key value and according to the data operation type and each batch of migratable data volume.
Optionally, the second processing unit 502 is specifically configured to monitor, according to a set trigger, the data that has been migrated in full at the source data end to obtain corresponding incremental data.
Optionally, the second processing unit 502 is specifically configured to determine a data volume of data that is successfully migrated in a previous batch, to determine starting point data when migrating the next batch of data, perform batch full migration of data between the source data end and the target data end according to each batch of migratable data volume configured in the full migration index table and a corresponding source table primary key value, and according to each batch of migratable data volume.
Optionally, the second processing unit 502 is specifically configured to: aiming at the incremental migration data, the data operation type is newly added data which is inserted and updated, and batch data replacement of the incremental data is carried out between the source data end and the target data end through a data replacement instruction and a source table primary key value corresponding to a maximum integer self-increment value corresponding to each newly added data.
Optionally, the second processing unit 502 is specifically configured to: and for the newly added data with the data operation type of deletion in the data subjected to incremental migration, performing batch data deletion of the incremental data between the source data end and the target data end according to the transferable data volume of each batch.
Fig. 6 schematically shows a schematic view of an electronic device of an embodiment of the disclosure. As shown in fig. 6, the electronic apparatus includes: a memory 601, on which a computer-executable program is stored, and a processor 602 for running the computer-executable program to implement the method of any of the embodiments of the present disclosure.
The disclosed embodiments also provide a computer storage medium having a computer-executable program stored thereon, the computer-executable program being executed to implement the method of any one of the disclosed embodiments.
The disclosed embodiments also provide a computer program product comprising a computer program which, when executed by a processor, implements a method according to any one of the disclosed embodiments.
The above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (14)
1. A method of data migration, comprising:
under the condition that the data migration type selected by data migration between a source data end and a target data end is incremental data migration, acquiring a preconfigured incremental migration index table, wherein the incremental migration index table at least comprises a first source table primary key column, and each row in the first source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in a source data table;
and performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table.
2. The method of claim 1, further comprising: before incremental data migration is carried out between the source data end and the target data end,
under the condition that the data migration type selected by data migration between the source data end and the target data end is full data migration, acquiring a pre-configured full migration index table, wherein the full migration index table at least comprises a second source table primary key column, and each row in the second source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in the source data table;
and carrying out full data migration between the source data end and the target data end according to each source table primary key value in the full migration index table.
3. The method of claim 1 or 2, wherein the incremental migration index table and the full-scale migration index table each further comprise: and the integer self-increment column is used for configuring the transferable data volume of each batch during batch migration.
4. The method of claim 3, wherein the incremental migration index table further comprises: a data operation type, wherein the data operation type comprises at least one of insertion, deletion and update.
5. The method according to claim 3, wherein the performing full volume data migration between the source data side and the target data side according to each source table primary key value in the full volume migration index table comprises:
and performing batch full migration of data between the source data end and the target data end according to each batch of transferable data size configured in the full migration index table and the corresponding primary key value of the source table and according to each batch of transferable data size.
6. The method of claim 5, wherein the performing data migration between the source data side and the target data side according to each source table primary key value in the full migration index table further comprises: in the process of batch full-scale migration of data between the source data end and the target data end,
monitoring the data which is completely migrated at the source data end to obtain corresponding incremental data;
and performing batch incremental migration of the incremental data between the source data end and the target data end according to each batch of migratable data volume configured in the incremental migration index table and the corresponding source table primary key value and according to the data operation type and each batch of migratable data volume.
7. The method according to claim 6, wherein the monitoring the data migrated in full at the source data end to obtain corresponding incremental data comprises:
and monitoring the data which is migrated in full at the source data end according to a set trigger so as to obtain corresponding incremental data.
8. The method according to claim 6, wherein the batch full migration of data between the source data end and the target data end according to each batch of migratable data volume configured in the full migration index table and the corresponding source table primary key value according to each batch of migratable data volume comprises:
determining the data volume of the data which is successfully migrated in the previous batch to determine the starting point data when the data in the next batch is migrated, and performing batch full-scale migration of the data between the source data end and the target data end according to the data volume which is configured in the full-scale migration index table and the source table primary key value corresponding to the starting point data and the data volume which is configured in the full-scale migration index table.
9. The method of claim 6, wherein the performing batch incremental migration of the incremental data between the source data side and the target data side according to the data operation type and the migratable data amount per batch comprises:
aiming at the incremental migration data, the data operation type is newly added data which is inserted and updated, and batch data replacement of the incremental data is carried out between the source data end and the target data end through a data replacement instruction and a source table primary key value corresponding to a maximum integer self-increment value corresponding to each newly added data.
10. The method of claim 6, wherein the performing batch incremental migration of the incremental data between the source data side and the target data side according to the data operation type and the migratable data amount per batch comprises:
and for the newly added data with the data operation type of deletion in the data subjected to incremental migration, performing batch data deletion of the incremental data between the source data end and the target data end according to the transferable data volume of each batch.
11. A data migration apparatus, comprising:
the data migration method includes the steps that a first processing unit is used for obtaining a preconfigured incremental migration index table under the condition that a data migration type selected by data migration between a source data end and a target data end is incremental data migration, the incremental migration index table at least comprises a first source table primary key column, and each row in the first source table primary key column can be used for recording a source table primary key value corresponding to data to be migrated in a source data table;
and the second processing unit is used for performing incremental data migration between the source data end and the target data end according to each source table primary key value in the incremental migration index table.
12. A computer storage medium having stored thereon a computer-executable program that is executed to implement the method of any one of claims 1-10.
13. An electronic device comprising a memory for storing thereon a computer-executable program and a processor for executing the computer-executable program to implement the method of any of claims 1-10.
14. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111646319.8A CN114328458A (en) | 2021-12-29 | 2021-12-29 | Data migration method and device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111646319.8A CN114328458A (en) | 2021-12-29 | 2021-12-29 | Data migration method and device and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328458A true CN114328458A (en) | 2022-04-12 |
Family
ID=81017141
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111646319.8A Pending CN114328458A (en) | 2021-12-29 | 2021-12-29 | Data migration method and device and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328458A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117131023A (en) * | 2023-10-24 | 2023-11-28 | 腾讯科技（深圳）有限公司 | Data table processing method, device, computer equipment and readable storage medium |
-
2021
- 2021-12-29 CN CN202111646319.8A patent/CN114328458A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117131023A (en) * | 2023-10-24 | 2023-11-28 | 腾讯科技（深圳）有限公司 | Data table processing method, device, computer equipment and readable storage medium |
| CN117131023B (en) * | 2023-10-24 | 2024-03-26 | 腾讯科技（深圳）有限公司 | Data table processing method, device, computer equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11379412B2 (en) | Enabling data replication processes between heterogeneous storage systems | |
| US20180067830A1 (en) | Healing cloud services during upgrades | |
| US10445186B1 (en) | Associating a guest application within a virtual machine to create dependencies in backup/restore policy | |
| US11165645B2 (en) | Message queue migration on A/B release environments | |
| CN111274252A (en) | Block chain data chaining method, device, storage medium and server | |
| CN115292026B (en) | Management method, device and equipment of container cluster and computer readable storage medium | |
| CN114064563A (en) | Data migration method and server based on object storage | |
| CN104850416A (en) | Upgrading system, method and device and cloud computing node | |
| US10579283B1 (en) | Elastic virtual backup proxy | |
| CN114328458A (en) | Data migration method and device and computer storage medium | |
| US20210117120A1 (en) | Data movement between heterogeneous storage devices | |
| CN112044061B (en) | Game picture processing method and device, electronic equipment and storage medium | |
| CN112631994A (en) | Data migration method and system | |
| CN117056123A (en) | Data recovery method, device, medium and electronic equipment | |
| CN111488117A (en) | Method, electronic device, and computer-readable medium for managing metadata | |
| CN109814911A (en) | Method, apparatus, computer equipment and storage medium for Manage Scripts program | |
| CN113254271B (en) | Data sequence recovery method, device, equipment and storage medium | |
| CN112685218B (en) | Method, apparatus and computer program product for managing backup systems | |
| CN111538520B (en) | Updating method and device for super-converged cluster, terminal and storage medium | |
| US10972343B2 (en) | System and method for device configuration update | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium | |
| CN108733460B (en) | Method and apparatus for maintaining sessions for network storage devices | |
| CN109542588B (en) | Method and device for managing virtual equipment in cloud environment | |
| CN113296889B (en) | Task management method, device, system, computer equipment and storage medium | |
| CN112100283B (en) | Linux platform based time-sharing multiplexing method for android virtual machine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |