CN114860690A - Data migration method, device, equipment and storage medium - Google Patents

Abstract

The invention relates to the field of data processing, and discloses a data migration method, comprising: obtaining a data set to be migrated and a data migration target table in a data source table; arranging the data set to be migrated to obtain a data set to be migrated; Divide the data set to be migrated to obtain the divided data set to be migrated; sort the multiple data migration tasks according to the preset priority, and migrate the divided data set to be migrated to the data set according to the sorted data migration tasks in the migration target table. The invention also relates to a blockchain technology, and the data set to be migrated can be stored in the blockchain node. The present invention also provides a data migration device, equipment and medium. The present invention can improve data migration efficiency.

Description

Data migration method, apparatus, device and storage medium

technical field

The present invention relates to the field of data processing, and in particular, to a data migration method, apparatus, device and storage medium.

Background technique

Data migration refers to the process of migrating data from a source database to a target database. In the traditional data migration process, data migration is usually realized through data migration tools. Because data migration tools mainly perform simple data synchronization through operation logs, the data integrity during the data migration process cannot be guaranteed, and when data migration tools appear When an exception occurs, the data migration process will be interrupted and needs to be restarted, resulting in low data migration efficiency.

SUMMARY OF THE INVENTION

The present invention provides a data migration method, device, equipment and storage medium, the main purpose of which is to improve data migration efficiency.

To achieve the above purpose, the present invention provides a data migration method, comprising:

Obtain the data set to be migrated and the data migration target table in the data source table;

Perform data arrangement on the data set to be migrated to obtain a data set to be migrated;

performing a segmentation operation on the data set to be migrated, to obtain a segmented data set to be migrated;

Create multiple data migration tasks for dividing the data set to be migrated, sort the multiple data migration tasks according to preset priorities to obtain sorted data migration tasks, and transfer the data migration tasks according to the sorted data migration tasks The data set to be migrated is divided and migrated to the data migration target table.

Optionally, performing data orchestration on the data set to be migrated to obtain the data set to be migrated by orchestration includes:

obtaining the data type of the data set to be migrated;

Identifying the data structure of the data set to be migrated according to the data type;

The data structure is arranged according to a preset arrangement rule to obtain the arranged data set to be migrated.

Optionally, performing a segmentation operation on the orchestration to-be-migrated data set to obtain the segmentation to-be-migrated data set, including:

Performing a sub-database operation on the data source database to obtain a plurality of data source sub-databases;

Perform a table splitting operation in a plurality of the data source sub-databases to obtain a plurality of data source sub-tables;

Divide the data set to be migrated according to the plurality of data source sub-tables to obtain the divided data set to be migrated.

Optionally, the creating multiple data migration tasks for dividing the data set to be migrated includes:

obtaining the segment number and segment address of the segmented data set to be migrated;

Associating the segmented data set to be migrated with the corresponding segmented number and segmented address, respectively, to obtain a plurality of data migration tasks for segmenting the to-be-migrated data set.

Optionally, migrating the segmented data set to be migrated to the data migration target table according to the sorting data migration task includes:

Execute the sorting data migration task, and use a preset plug-in to read the execution parameters of the sorting data migration task;

comparing the execution parameter with a preset load factor;

When the execution parameter does not exceed the load coefficient threshold, use a preset data migration tool to load the segmented data set to be migrated into the data migration target table through a preset migration thread;

When the execution parameter exceeds the load coefficient threshold, multiple migration threads are added, and the data migration tool is used to sequentially load the split data set to be migrated to the data migration target through the multiple migration threads in the table.

Optionally, after migrating the segmented data set to be migrated to the data migration target table according to the sorting data migration task, the method further includes:

Check the consistency of the split data set to be migrated and the full migration data set in the data migration target table according to the preset full volume rule. When the split to be migrated data set is inconsistent with the full migration data set , recording the migration abnormal point of the divided data set to be migrated, and re-executes the data migration of dividing the data set to be migrated according to the migration abnormal point;

Check the consistency between the split data set to be migrated and the incremental migration data set in the data migration target table according to the preset incremental rule, when the split data set to be migrated and the incremental migration data When the sets are inconsistent, record the migration abnormal point of the divided data set to be migrated, and re-execute the data migration of dividing the data set to be migrated according to the migration abnormal point.

Optionally, after obtaining the data set to be migrated and the data migration target table in the data source table, the method further includes:

Perform a deduplication operation on the data set to be migrated, and detect whether there are missing data values in the data set to be migrated after deduplication;

If there is no data missing value, obtain the data set to be migrated after deduplication;

If there is a missing data value, the missing data value is filled to obtain a filled data set to be migrated.

In order to solve the above problems, the present invention also provides a data migration device, the device includes:

The data acquisition module is used to acquire the data set to be migrated and the data migration target table in the data source table;

a data arrangement module, configured to perform data arrangement on the data set to be migrated to obtain a data set to be arranged to be migrated;

a data segmentation module, configured to perform a segmentation operation on the orchestrated data set to be migrated to obtain a segmented data set to be migrated;

A data migration module, configured to create a plurality of data migration tasks for dividing the data set to be migrated, sort the plurality of data migration tasks according to a preset priority, obtain a sorted data migration task, and sort the data migration tasks according to the sorting The data migration task migrates the divided data set to be migrated into the data migration target table.

In order to solve the above problems, the present invention also provides an electronic device, the electronic device includes:

a memory that stores at least one computer program; and

The processor executes the computer program stored in the memory to implement the data migration method described above.

In order to solve the above problems, the present invention also provides a computer-readable storage medium, where at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is executed by a processor in an electronic device to realize the above-mentioned The data migration method described above.

In the embodiment of the present invention, the data set to be migrated is obtained by first performing data arrangement on the data set to be migrated, which can avoid subsequent migration due to inconsistency between the data structure in the data source table and the data structure of the data migration target table. An exception occurs when the data set is migrated; secondly, the data set to be migrated is divided into the data set to be divided, which can avoid the failure of the overall data migration due to abnormal scenarios that occur during the one-time migration of a large amount of data. The result is that when an exception occurs in the data migration process, by identifying and dividing the abnormal data set to be migrated in the data set to be migrated, it is only necessary to perform data migration and repair of the abnormal data set to be migrated again to ensure data integrity during the data migration process. Finally, according to the sorted data migration task, the segmented data set to be migrated is migrated to the data migration target table, which can realize intelligent scheduling of data migration tasks and improve data migration efficiency. Therefore, the data migration method, apparatus, device, and storage medium provided by the embodiments of the present invention can improve data migration efficiency.

Description of drawings

FIG. 1 is a schematic flowchart of a data migration method according to an embodiment of the present invention;

2 is a detailed flowchart of a step in a data migration method provided by an embodiment of the present invention;

3 is a detailed flowchart of a step in a data migration method provided by an embodiment of the present invention;

FIG. 4 is a schematic block diagram of a data migration apparatus according to an embodiment of the present invention;

5 is a schematic diagram of an internal structure of an electronic device implementing a data migration method according to an embodiment of the present invention;

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Embodiments of the present invention provide a data migration method. The execution body of the data migration method includes, but is not limited to, at least one of electronic devices that can be configured to execute the method provided by the embodiments of the present application, such as a server and a terminal. In other words, the data migration method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.

Referring to the schematic flowchart of the data migration method provided by an embodiment of the present invention shown in FIG. 1, in the embodiment of the present invention, the data migration method includes the following steps S1-S4:

S1. Obtain the data set to be migrated and the data migration target table in the data source table.

In the embodiment of the present invention, the data set to be migrated refers to the data that needs to be migrated from the data source table to the data migration target table, for example, the data in the Oracle database is migrated to the Tidb database; the data source table refers to the storage to be The data table of the data set to be migrated; the data migration target table refers to the data set that stores the migrated data set of the data set to be migrated.

In an optional embodiment of the present invention, since there are a lot of useless data in the data set to be migrated, the embodiment of the present invention performs data cleaning on the data to be migrated to obtain a standard data set, and can filter out the data to be migrated Useless data in the data set improves the efficiency of data migration.

In detail, after obtaining the data set to be migrated and the data migration target table in the data source table, the method further includes:

Perform a deduplication operation on the data set to be migrated, and detect whether there are missing data values in the data set to be migrated after deduplication; if there is no missing data value, obtain the data set to be migrated after deduplication; If there is a missing data value, the missing data value is filled to obtain a filled data set to be migrated.

In an embodiment of the present invention, the following method is used to perform a deduplication operation on the data set to be migrated:

Among them, d represents the distance value of any two data in the data set to be migrated, w _1j and w _2j represent any two data in the data set to be migrated, when the distance value is less than the preset distance value, delete any one of the data, If the distance value is not less than the preset distance value, two pieces of data are kept at the same time. Preferably, the preset distance value may be 0.1.

S2. Perform data arrangement on the data set to be migrated to obtain a data set to be migrated.

In the embodiment of the present invention, the main function of the data arrangement is that since the data structure in the data source table is inconsistent with the data structure of the data migration target table, the data arrangement is performed in order to adapt to the data structure of the data migration target table.

In the embodiment of the present invention, by performing data arrangement on the data set to be migrated to obtain the data set to be arranged to be migrated, it is possible to avoid the subsequent data set to be migrated due to the inconsistency between the data structure in the data source table and the data structure of the data migration target table. An exception occurs during migration to improve the efficiency of subsequent data migration.

As an embodiment of the present invention, referring to FIG. 2 , performing data arrangement on the data set to be migrated to obtain the data set to be arranged to be migrated includes the following steps S21-S23:

S21, obtaining the data type of the data set to be migrated;

S22, identifying the data structure of the data set to be migrated according to the data type;

S23. Arrange the data structure according to a preset arrangement rule to obtain the arranged data set to be migrated.

The data type is mainly used to represent the data of the data set information to be migrated; the data structure refers to a collection of data elements with one or more relationships between the data sets to be migrated, for example, in the customer insurance data, the customer Insurance relationship with different types of insurance.

In an embodiment of the present invention, the arrangement rules can be customized according to actual scenarios. For example, in the insurance field, the data set to be migrated in the data source table lists the insurance application status of all customers, and the arrangement rules can be based on customer categories. Arrangement, when it is identified that the customer type is an individual customer, it is arranged to the individual table, and the customer type is recognized to be a group customer, it is arranged to the group table.

S3. Perform a segmentation operation on the orchestrated data set to be migrated to obtain a segmented data set to be migrated.

A segmentation operation is performed on the data set to be migrated, and the segmented data set to be migrated is obtained.

In the embodiment of the present invention, the dividing the data set to be migrated refers to the data after the data set to be migrated is divided into databases and tables.

In the embodiment of the present invention, the data set to be migrated is obtained by dividing the data set to be migrated, which can avoid the consequence of the failure of the overall data migration caused by the abnormal scenario that occurs during the one-time migration of a large amount of data, and is convenient for When an abnormality occurs in the data migration process, by identifying and dividing the abnormal data set to be migrated in the data set to be migrated, it is only necessary to perform data migration and repair of the abnormal data set to be migrated again, and ensure the data integrity during the data migration process.

As an embodiment of the present invention, referring to FIG. 3 , performing a segmentation operation on the data set to be migrated for orchestration to obtain a segmented data set to be migrated includes the following steps S31-S33:

S31, performing a sub-database operation on the data source database to obtain a plurality of data source sub-databases;

S32, performing a table splitting operation in a plurality of the data source sub-databases to obtain a plurality of data source sub-tables;

S33: Divide the data set to be migrated according to the plurality of data source sub-tables to obtain the divided data set to be migrated.

Among them, due to the performance limitations of the data source database, the database capacity of the data source database cannot be too large, and the data source database needs to be sub-database operation to obtain multiple data source sub-databases, for example, if the data source database is A, then multiple The data source sub-database can be A1, A2, A3, A4, etc.; the data source sub-database is divided into tables to prevent the data table from storing more record data and avoid abnormal interruption of data migration. When an exception occurs, it will not affect the migration of other sub-table data. For example, if the number of data source sub-databases is six, and each sub-database stores ten sub-tables, the total number of data source sub-tables is 60. The data set to be migrated is segmented according to the data source segment table, and sixty data sets to be migrated can be segmented.

S4. Create a plurality of data migration tasks for dividing the data set to be migrated, sort the plurality of data migration tasks according to a preset priority to obtain a sorted data migration task, and arrange the data migration tasks according to the sorted data migration task. The divided data set to be migrated is migrated to the data migration target table.

In the embodiment of the present invention, the data migration task refers to the task of dividing the data set to be migrated and migrating to the data migration target table. For example, if the data set to be migrated is divided into the insurance information of all customers of an insurance company, the data migration task is the insurance information of ordinary customers, critical illness insurance, VIP customers, and VIP customers. Data migration tasks.

In the embodiment of the present invention, the preset priority refers to the user-defined priority ordering of multiple data migration tasks according to requirements. For example, the first level may be the auto insurance application information of VIP customers, and the second level may be VIP customers. Customer critical illness insurance insurance information, third-level critical illness insurance insurance information for ordinary customers, and fourth-level auto insurance insurance information for ordinary customers.

In the embodiment of the present invention, by creating a plurality of data migration tasks for dividing the data set to be migrated, and sorting the plurality of data migration tasks according to a preset priority, a sorted data migration task is obtained, and according to the sorting data The migration task migrates the divided data set to be migrated to the data migration target table, which can realize intelligent scheduling of data migration tasks and improve data migration efficiency.

Specifically, the creating multiple data migration tasks for dividing the data set to be migrated includes:

Obtain the segment number and segment address of the segmented data set to be migrated; associate the segmented data set to be migrated with the corresponding segment number and segment address, respectively, to obtain the segment Multiple data migration tasks for the dataset to be migrated.

Further, migrating the segmented data set to be migrated to the data migration target table according to the sorting data migration task includes:

Execute the sorting data migration task, and use a preset plug-in to read the execution parameters of the sorting data migration task; compare the execution parameters with the preset load factor; when the execution parameter does not exceed the load factor When the threshold is set, use a preset data migration tool to load the segmented data set to be migrated into the data migration target table through a preset migration thread; when the execution parameter exceeds the load factor threshold, add a new Multiple migration threads, using the data migration tool to sequentially load the segmented data set to be migrated into the data migration target table through the multiple migration threads.

The execution parameter refers to the resource consumption parameter of the target database where the data migration target table is located during the execution of the sorting data migration task, for example, the operating load of the database such as the number of database connections, tps, qps, CPU, memory, etc. coefficient; the preset load coefficient threshold is 80%.

In an embodiment of the present invention, the sorting data migration task is performed by a migration thread, and the task can be performed by different threads through resource consumption of the target database, so as to realize intelligent scheduling of task execution and improve the migration efficiency of data migration.

Preferably, the data migration tool may be a KETTLE tool.

In another embodiment of the present invention, after migrating the segmented data set to be migrated to the data migration target table according to the sorted data migration task, the method further includes:

Check the consistency of the split data set to be migrated and the full migration data set in the data migration target table according to the preset full volume rule. When the split to be migrated data set is inconsistent with the full migration data set , record the migration abnormal point of the divided data set to be migrated, re-execute the data migration of the divided data set to be migrated according to the migration abnormal point; check the divided data set to be migrated according to the preset increment rule Consistency with the incremental migration data set in the data migration target table, when the split data set to be migrated is inconsistent with the incremental migration data set, record the migration exception of the split data set to be migrated point, re-execute the data migration of dividing the data set to be migrated according to the migration abnormal point.

The main function of the full-scale rule is to identify whether the full-scale migration data set stored in the source data table within a specified period of time is synchronized with the full-scale migration data set migrated to the data target table, and changes in the data source table will be reflected in real time. To the data migration target table, the split migration data set of the final data source table is consistent with the full migration data set; the main function of the incremental rule is to realize the synchronization of data migration without production interruption.

For example, data migration of data source tables before February 2, 2022 is regarded as full data migration, and data migration of data source tables after February 2, 2022 is regarded as incremental data migration.

In the embodiment of the present invention, the data set to be migrated is obtained by first performing data arrangement on the data set to be migrated, which can avoid subsequent migration due to inconsistency between the data structure in the data source table and the data structure of the data migration target table. An exception occurs when the data set is migrated; secondly, the data set to be migrated is divided into the data set to be divided, which can avoid the failure of the overall data migration due to abnormal scenarios that occur during the one-time migration of a large amount of data. The result is that when an exception occurs in the data migration process, by identifying and dividing the abnormal data set to be migrated in the data set to be migrated, it is only necessary to perform data migration and repair of the abnormal data set to be migrated again to ensure data integrity during the data migration process. Finally, according to the sorted data migration task, the segmented data set to be migrated is migrated to the data migration target table, which can realize intelligent scheduling of data migration tasks and improve data migration efficiency. Therefore, the data migration method proposed in the embodiments of the present invention can improve the data migration efficiency.

The data migration apparatus 100 of the present invention can be installed in an electronic device. According to the implemented functions, the data migration device may include a data acquisition module 101, a data arrangement module 102, a data segmentation module 103, and a data migration module 104. The modules in the present invention may also be referred to as units, which refer to a device capable of A series of computer program segments executed by the processor of an electronic device and capable of performing fixed functions, which are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The data acquisition module 101 is configured to acquire the data set to be migrated and the data migration target table in the data source table.

In the embodiment of the present invention, the data set to be migrated refers to the data that needs to be migrated from the data source table to the data migration target table, for example, the data in the Oracle database is migrated to the Tidb database; the data source table refers to the storage to be The data table of the data set to be migrated; the data migration target table refers to the data set that stores the migrated data set of the data set to be migrated.

In an optional embodiment of the present invention, since there are a lot of useless data in the data set to be migrated, the embodiment of the present invention performs data cleaning on the data to be migrated to obtain a standard data set, and can filter out the data to be migrated Useless data in the data set improves the efficiency of data migration.

The data acquisition module 101 can also be used for:

Perform a deduplication operation on the data set to be migrated, and detect whether there are missing data values in the data set to be migrated after deduplication; if there is no missing data value, obtain the data set to be migrated after deduplication; If there is a missing data value, the missing data value is filled to obtain a filled data set to be migrated.

In an embodiment of the present invention, the following method is used to perform a deduplication operation on the data set to be migrated:

Among them, d represents the distance value of any two data in the data set to be migrated, w _1j and w _2j represent any two data in the data set to be migrated, when the distance value is less than the preset distance value, delete any one of the data, If the distance value is not less than the preset distance value, two pieces of data are kept at the same time. Preferably, the preset distance value may be 0.1.

The data arrangement module 102 is configured to perform data arrangement on the to-be-migrated data set to obtain the arranged to-be-migrated data set.

In the embodiment of the present invention, the main function of the data arrangement is that since the data structure in the data source table is inconsistent with the data structure of the data migration target table, the data arrangement is performed in order to adapt to the data structure of the data migration target table.

In the embodiment of the present invention, by performing data arrangement on the data set to be migrated to obtain the data set to be arranged to be migrated, it is possible to avoid the subsequent data set to be migrated due to the inconsistency between the data structure in the data source table and the data structure of the data migration target table. An exception occurs during migration to improve the efficiency of subsequent data migration.

As an embodiment of the present invention, the data orchestration module 102 performs data orchestration on the data set to be migrated by performing the following operations to obtain the data set to be migrated, including:

obtaining the data type of the data set to be migrated;

Identifying the data structure of the data set to be migrated according to the data type;

The data structure is arranged according to a preset arrangement rule to obtain the arranged data set to be migrated.

The data type is mainly used to represent the data of the data set information to be migrated; the data structure refers to a collection of data elements with one or more relationships between the data sets to be migrated, for example, in the customer insurance data, the customer Insurance relationship with different insurance types.

In an embodiment of the present invention, the arrangement rules can be customized according to actual scenarios. For example, in the insurance field, the data set to be migrated in the data source table lists the insurance application status of all customers, and the arrangement rules can be based on customer categories. Arrangement, when it is identified that the customer type is an individual customer, it is arranged to the individual table, and the customer type is recognized to be a group customer, it is arranged to the group table.

The data segmentation module 103 is configured to perform a segmentation operation on the orchestrated data set to be migrated to obtain a segmented data set to be migrated.

A segmentation operation is performed on the data set to be migrated, and the segmented data set to be migrated is obtained.

In the embodiment of the present invention, the dividing the data set to be migrated refers to the data after the data set to be migrated is divided into databases and tables.

In the embodiment of the present invention, the data set to be migrated is obtained by dividing the data set to be migrated, which can avoid the consequence of the failure of the overall data migration caused by the abnormal scenario that occurs during the one-time migration of a large amount of data, and is convenient for When an abnormality occurs in the data migration process, by identifying and dividing the abnormal data set to be migrated in the data set to be migrated, it is only necessary to perform data migration and repair of the abnormal data set to be migrated again, and ensure the data integrity during the data migration process.

As an embodiment of the present invention, the data segmentation module 103 performs the following operations to perform the segmentation operation on the data set to be migrated to obtain the segmented data set to be migrated, including:

Performing a sub-database operation on the data source database to obtain a plurality of data source sub-databases;

Perform a table splitting operation in a plurality of the data source sub-databases to obtain a plurality of data source sub-tables;

Divide the data set to be migrated according to the plurality of data source sub-tables to obtain the divided data set to be migrated.

Among them, due to the performance limitations of the data source database, the database capacity of the data source database cannot be too large, and the data source database needs to be sub-database operation to obtain multiple data source sub-databases, for example, if the data source database is A, then multiple The data source sub-database can be A1, A2, A3, A4, etc.; the data source sub-database is divided into tables to prevent the data table from storing more record data and avoid abnormal interruption of data migration. When an exception occurs, it will not affect the migration of other sub-table data. For example, if the number of data source sub-databases is six, and each sub-database stores ten sub-tables, the total number of data source sub-tables is 60. The data set to be migrated is segmented according to the data source segment table, and sixty data sets to be migrated can be segmented.

The data migration module 104 is configured to create a plurality of data migration tasks for dividing the data set to be migrated, sort the plurality of data migration tasks according to a preset priority, obtain a sorted data migration task, and perform the data migration tasks according to The sorting data migration task migrates the divided data set to be migrated into the data migration target table.

In the embodiment of the present invention, the data migration task refers to the task of dividing the data set to be migrated and migrating to the data migration target table. For example, if the data set to be migrated is divided into the insurance information of all customers of an insurance company, the data migration task is the insurance information of ordinary customers, critical illness insurance, VIP customers, and VIP customers. Data migration tasks.

In the embodiment of the present invention, the preset priority refers to the user-defined priority ordering of multiple data migration tasks according to requirements. For example, the first level may be the auto insurance application information of VIP customers, and the second level may be VIP customers. Customer critical illness insurance insurance information, third-level critical illness insurance insurance information for ordinary customers, and fourth-level auto insurance insurance information for ordinary customers.

In the embodiment of the present invention, by creating a plurality of data migration tasks for dividing the data set to be migrated, and sorting the plurality of data migration tasks according to a preset priority, a sorted data migration task is obtained, and according to the sorting data The migration task migrates the divided data set to be migrated to the data migration target table, which can realize intelligent scheduling of data migration tasks and improve data migration efficiency.

Specifically, the data migration module 104 creates a plurality of data migration tasks for dividing the data set to be migrated by performing the following operations, including:

Obtain the segment number and segment address of the segmented data set to be migrated; associate the segmented data set to be migrated with the corresponding segment number and segment address, respectively, to obtain the segment Multiple data migration tasks for the dataset to be migrated.

Further, migrating the segmented data set to be migrated to the data migration target table according to the sorting data migration task includes:

Execute the sorting data migration task, and use a preset plug-in to read the execution parameters of the sorting data migration task; compare the execution parameters with the preset load factor; when the execution parameter does not exceed the load factor When the threshold is set, use a preset data migration tool to load the segmented data set to be migrated into the data migration target table through a preset migration thread; when the execution parameter exceeds the load factor threshold, add a new Multiple migration threads, using the data migration tool to sequentially load the segmented data set to be migrated into the data migration target table through the multiple migration threads.

The execution parameter refers to the resource consumption parameter of the target database where the data migration target table is located during the execution of the sorting data migration task, for example, the operating load of the database such as the number of database connections, tps, qps, CPU, memory, etc. coefficient; the preset load coefficient threshold is 80%.

In an embodiment of the present invention, the sorting data migration task is performed by a migration thread, and the task can be performed by different threads according to the resource consumption of the target database, so as to realize intelligent scheduling of task execution and improve the migration efficiency of data migration.

Preferably, the data migration tool may be a KETTLE tool.

In another embodiment of the present invention, the data migration module 104 may also be used for:

Check the consistency of the split data set to be migrated and the full migration data set in the data migration target table according to the preset full volume rule. When the split to be migrated data set is inconsistent with the full migration data set , record the migration abnormal point of the divided data set to be migrated, re-execute the data migration of the divided data set to be migrated according to the migration abnormal point; check the divided data set to be migrated according to the preset increment rule Consistency with the incremental migration data set in the data migration target table, when the split data set to be migrated is inconsistent with the incremental migration data set, record the migration exception of the split data set to be migrated point, re-execute the data migration of dividing the data set to be migrated according to the migration abnormal point.

The main function of the full-scale rule is to identify whether the full-scale migration data set stored in the source data table within a specified period of time is synchronized with the full-scale migration data set migrated to the data target table, and changes in the data source table will be reflected in real time. To the data migration target table, the split migration data set of the final data source table is consistent with the full migration data set; the main function of the incremental rule is to realize the synchronization of data migration without production interruption.

For example, data migration of data source tables before February 2, 2022 is regarded as full data migration, and data migration of data source tables after February 2, 2022 is regarded as incremental data migration.

In the embodiment of the present invention, the data set to be migrated is obtained by first performing data arrangement on the data set to be migrated, which can avoid subsequent migration due to inconsistency between the data structure in the data source table and the data structure of the data migration target table. An exception occurs when the data set is migrated; secondly, the data set to be migrated is divided into the data set to be divided, which can avoid the failure of the overall data migration due to abnormal scenarios that occur during the one-time migration of a large amount of data. The result is that when an exception occurs in the data migration process, by identifying and dividing the abnormal data set to be migrated in the data set to be migrated, it is only necessary to perform data migration and repair of the abnormal data set to be migrated again to ensure data integrity during the data migration process. Finally, according to the sorted data migration task, the segmented data set to be migrated is migrated to the data migration target table, which can realize intelligent scheduling of data migration tasks and improve data migration efficiency. Therefore, the data migration apparatus provided by the embodiments of the present invention can improve the data migration efficiency.

As shown in FIG. 5 , it is a schematic structural diagram of an electronic device implementing the data migration method of the present invention.

The electronic device may include a processor 10 , a memory 11 , a communication bus 12 and a communication interface 13 , and may also include a computer program, such as a data migration program, stored in the memory 11 and executable on the processor 10 .

Wherein, the memory 11 includes at least one type of medium, including flash memory, mobile hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), magnetic memory, local magnetic disk, optical disk, and the like. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of the electronic device, such as a pluggable mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) equipped on the electronic device. ) card, flash card (Flash Card) and so on. Further, the memory 11 may also include both an internal storage unit of an electronic device and an external storage device. The memory 11 can not only be used to store application software installed in the electronic device and various types of data, such as the code of a data migration program, etc., but also can be used to temporarily store data that has been output or will be output.

In some embodiments, the processor 10 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more integrated circuits. Central processing unit (Central Processing Unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and by running or executing the programs or modules (such as data) stored in the memory 11. migration programs, etc.), and call data stored in the memory 11 to execute various functions of the electronic device and process data.

The communication bus 12 may be a PerIPheral Component Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The communication bus 12 is configured to implement connection communication between the memory 11 and at least one processor 10 and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

FIG. 5 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 5 does not constitute a limitation on the electronic device, and may include fewer or more components than those shown in the drawings. , or a combination of certain components, or a different arrangement of components.

For example, although not shown, the electronic device may also include a power source (such as a battery) for powering the various components, preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power source can be logically connected through the power management device. Implement functions such as charge management, discharge management, and power management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Optionally, the communication interface 13 may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

Optionally, the communication interface 13 may further include a user interface, and the user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface . Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device and for displaying a visual user interface.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

The data migration program stored in the memory 11 in the electronic device is a combination of multiple computer programs. When running in the processor 10, it can realize:

Obtain the data set to be migrated and the data migration target table in the data source table;

Perform data arrangement on the data set to be migrated to obtain a data set to be migrated;

performing a segmentation operation on the data set to be migrated, to obtain a segmented data set to be migrated;

Create multiple data migration tasks for dividing the data set to be migrated, sort the multiple data migration tasks according to preset priorities to obtain sorted data migration tasks, and transfer the data migration tasks according to the sorted data migration tasks The data set to be migrated is divided and migrated to the data migration target table.

Specifically, for the specific implementation method of the above-mentioned computer program by the processor 10, reference may be made to the description of the relevant steps in the corresponding embodiment of FIG. 1, and details are not described herein.

Further, if the modules/units integrated in the electronic device are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable medium. The computer-readable medium may be non-volatile or volatile. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory) ).

Embodiments of the present invention may further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when executed by a processor of an electronic device, the computer program may implement:

Obtain the data set to be migrated and the data migration target table in the data source table;

Perform data arrangement on the data set to be migrated to obtain a data set to be migrated;

performing a segmentation operation on the data set to be migrated, to obtain a segmented data set to be migrated;

Create multiple data migration tasks for dividing the data set to be migrated, sort the multiple data migration tasks according to preset priorities to obtain sorted data migration tasks, and transfer the data migration tasks according to the sorted data migration tasks The data set to be migrated is divided and migrated to the data migration target table.

Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function, and the like; The data created by the use of the node, etc.

In the several embodiments provided by the present invention, it should be understood that the disclosed medium, apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, but that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the invention.

Therefore, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the invention is to be defined by the appended claims rather than the foregoing description, which are therefore intended to fall within the scope of the claims. All changes within the meaning and range of the equivalents of , are included in the present invention. Any reference signs in the claims shall not be construed as limiting the involved claim.

The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present invention.

Claims (10)

1. a data migration method, is characterized in that, described method comprises: Obtain the data set to be migrated and the data migration target table in the data source table; Perform data arrangement on the data set to be migrated to obtain a data set to be migrated; performing a segmentation operation on the data set to be migrated, to obtain a segmented data set to be migrated; Create multiple data migration tasks for dividing the data set to be migrated, sort the multiple data migration tasks according to preset priorities to obtain sorted data migration tasks, and transfer the data migration tasks according to the sorted data migration tasks The data set to be migrated is divided and migrated to the data migration target table. 2. The data migration method according to claim 1, wherein the performing data arrangement on the to-be-migrated data set to obtain the arrangement of the to-be-migrated data set comprises: obtaining the data type of the data set to be migrated; Identifying the data structure of the data set to be migrated according to the data type; The data structure is arranged according to a preset arrangement rule to obtain the arranged data set to be migrated. 3. The data migration method according to claim 1, wherein the performing a segmentation operation on the orchestrated data set to be migrated to obtain a segmented data set to be migrated, comprising: Performing a sub-database operation on the data source database to obtain a plurality of data source sub-databases; Perform a table splitting operation in a plurality of the data source sub-databases to obtain a plurality of data source sub-tables; Divide the data set to be migrated according to the plurality of data source sub-tables to obtain the divided data set to be migrated. 4. The data migration method according to claim 1, wherein the creating a plurality of data migration tasks for dividing the data set to be migrated comprises: obtaining the segment number and segment address of the segmented data set to be migrated; Associating the segmented data set to be migrated with the corresponding segmented number and segmented address, respectively, to obtain a plurality of data migration tasks for segmenting the to-be-migrated data set. 5 . The data migration method according to claim 1 , wherein the migration of the divided data set to be migrated to the data migration target table according to the sorted data migration task comprises: 5 . Execute the sorting data migration task, and use a preset plug-in to read the execution parameters of the sorting data migration task; comparing the execution parameter with a preset load factor; When the execution parameter does not exceed the load coefficient threshold, use a preset data migration tool to load the segmented data set to be migrated into the data migration target table through a preset migration thread; When the execution parameter exceeds the load coefficient threshold, multiple migration threads are added, and the data migration tool is used to sequentially load the split data set to be migrated to the data migration target through the multiple migration threads in the table. 6 . The data migration method according to claim 1 , wherein after migrating the segmented data set to be migrated into the data migration target table according to the sorted data migration task, the method further include: Check the consistency of the split data set to be migrated and the full migration data set in the data migration target table according to the preset full volume rule. When the split to be migrated data set is inconsistent with the full migration data set , recording the migration abnormal point of the divided data set to be migrated, and re-executes the data migration of dividing the data set to be migrated according to the migration abnormal point; Check the consistency between the split data set to be migrated and the incremental migration data set in the data migration target table according to the preset incremental rule, when the split data set to be migrated and the incremental migration data When the sets are inconsistent, record the migration abnormal point of the divided data set to be migrated, and re-execute the data migration of dividing the data set to be migrated according to the migration abnormal point. 7. The data migration method according to claim 1, wherein after acquiring the data set to be migrated and the data migration target table in the data source table, the method further comprises: Perform a deduplication operation on the data set to be migrated, and detect whether there are missing data values in the data set to be migrated after deduplication; If there is no data missing value, obtain the data set to be migrated after deduplication; If there is a missing data value, the missing data value is filled to obtain a filled data set to be migrated. 8. A data migration device, wherein the device comprises: The data acquisition module is used to acquire the data set to be migrated and the data migration target table in the data source table; a data arrangement module, configured to perform data arrangement on the data set to be migrated to obtain a data set to be arranged to be migrated; a data segmentation module, configured to perform a segmentation operation on the orchestrated data set to be migrated to obtain a segmented data set to be migrated; A data migration module, configured to create a plurality of data migration tasks for dividing the data set to be migrated, sort the plurality of data migration tasks according to a preset priority, obtain a sorted data migration task, and sort the data migration tasks according to the sorting The data migration task migrates the divided data set to be migrated into the data migration target table. 9. An electronic device, characterized in that the electronic device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform any one of claims 1 to 7 The data migration method described in item. 10 . A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the data migration method according to any one of claims 1 to 7 is implemented. 11 .

Images