CN111881110A - Data migration method and device - Google Patents
Data migration method and device Download PDFInfo
- Publication number
- CN111881110A CN111881110A CN202010560502.5A CN202010560502A CN111881110A CN 111881110 A CN111881110 A CN 111881110A CN 202010560502 A CN202010560502 A CN 202010560502A CN 111881110 A CN111881110 A CN 111881110A
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- migration
- tables
- data tables
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013508 migration Methods 0.000 title claims abstract description 103
- 230000005012 migration Effects 0.000 title claims abstract description 103
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000005215 recombination Methods 0.000 claims abstract description 44
- 230000006798 recombination Effects 0.000 claims abstract description 44
- 230000008521 reorganization Effects 0.000 claims description 13
- 238000012163 sequencing technique Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012098 association analyses Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data migration method and a device, wherein the method comprises the following steps of initially grouping a data table according to an external key of the data table to be migrated, and the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes; analyzing the incidence relation of the data tables in the first set and the third set, recombining the first set and the third set to obtain a recombination result with a plurality of new sets; and carrying out data migration according to the recombination result and the second set. The embodiment of the invention can rapidly migrate according to the incidence relation of the data table.
Description
Technical Field
The present invention relates to the field of database migration technologies, and in particular, to a data migration method and apparatus.
Background
In some business systems, the system may undergo major technological upgrades for better system service and lower maintenance costs, which may involve reconstruction and migration of underlying data. For example, migration from a system a to a system B may involve changes to databases, database tables, database fields, field types, and the like. In order to ensure that the migrated data is available in the service flow of the new system and does not affect the existing service, a whole set of verification scheme for data migration correctness is required. Therefore, a complete set of data migration verification scheme needs to be designed to ensure the consistency, accuracy and availability of data.
The existing data migration schemes generally migrate data to be migrated according to a storage location or a data type of the data, for example, data in the same folder is migrated first, and then data in a next folder is migrated, and for example, data of the same suffix (for example, jar) is migrated first, and then data of a new suffix is migrated, but both existing data migration methods have a problem of low migration efficiency.
Disclosure of Invention
Embodiments of the present invention provide a data migration method and apparatus that overcome the above problems or at least partially solve the above problems.
In a first aspect, an embodiment of the present invention provides a data migration method, including:
according to the foreign key of the data table to be migrated, initially grouping the data table, wherein the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes;
analyzing the incidence relation of the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result with a plurality of new sets;
carrying out data migration according to the recombination result and the second set;
wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
Further, the analyzing the association relationship between the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result with a plurality of new sets specifically includes:
creating a new set with the number equal to that of the data tables in the third set, and taking each data table in the third set as a first element corresponding to the new set;
for any data table pair in the first set, searching whether a main table in the data table pair exists in the third set;
if the main table in the data table pair exists in the third set, placing the auxiliary table in the data table pair in a new set where the main table in the data table pair is located, and deleting the data table pair from the first set;
if the main table in the data table pair does not exist in the third set, searching the data table pair of the main table as a slave table in the first set as a first data table pair to be combined, continuously searching whether the main table in the data table pair to be combined exists in the third set, if the main table does not exist, continuously searching the main table in the data table pair to be combined in the first set as a data table pair of a slave table as a next data table pair to be combined until the main table in the searched data table pair exists in the third set, placing the data table pair and all data tables in the data table pair to be combined in a new set where the main table of the last data table pair to be combined is located, and deleting the data table pair and all data tables in the data table pair to be combined from the first set.
Further, the initially grouping the data table according to the foreign key of the data table to be migrated includes:
and determining the number of threads for initial grouping according to the number of the data tables to be migrated, wherein each thread is used for initially grouping the data tables in the self grouping range.
Further, the determining the number of threads for initial grouping according to the number of the data tables to be migrated specifically includes:
when the number of the data tables to be migrated is not less than a preset threshold, the number of the threads is a first number; and when the number of the data tables to be migrated is less than the preset threshold, the number of the threads is a second number.
Further, the performing data migration according to the recombination result and the second set specifically includes:
and sequencing the new set in the recombination result, performing data table migration on the recombination result according to the sequencing, and then migrating the data table in the second set.
Further, the sorting the new set in the reorganization result further includes: sending the ordering of the new set and a migration operation interface for the new set to a front end, the migration operation interface including one or more of a start interface, a pause interface, and a stop interface.
Further, the data migration according to the recombination result and the second set further includes:
acquiring the number of the migrated data tables, and comparing the number of the migrated data tables with the number of the data tables to be migrated which are acquired in advance;
if the number is the same, determining that the migration is successful; and if the number is different, determining that the migration is wrong.
In a second aspect, an embodiment of the present invention provides a data migration apparatus, including;
the initial grouping module is used for initially grouping the data table according to the external key of the data table to be migrated, and the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes;
the recombination module is used for analyzing the incidence relation of the data tables in the first set and the third set, recombining the first set and the third set and obtaining a recombination result with a plurality of new sets;
the migration module is used for carrying out data migration according to the recombination result and the second set;
wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
Further, the restructuring module specifically includes:
a set creating unit, configured to create a new set with the number equal to that of the data tables in the third set, and take each data table in the third set as a first element of the corresponding new set;
a main table lookup unit, configured to, for any data table pair in the first set, lookup whether a main table in the data table pair exists in the third set;
a first arrangement unit, configured to, if a master table in the data table pair exists in the third set, place a slave table in the data table pair in a new set where the master table in the data table pair is located, and delete the data table pair from the first set;
a second homing unit to, if no master table of the pair of data tables exists in the third set, the data table pair of the master table as the slave table is looked up in the first set as the first data table pair to be combined, continuously searching whether a main table in the data table pairs to be combined exists in the third set, if not, continuously searching the main table in the data table pairs to be combined in the first set as a data table pair of a slave table as a next data table pair to be combined until the main table in the searched data table pair exists in the third set, placing the data table pairs and all data tables in the data table pairs to be combined in a new set where the main table of the last data table pair to be combined is located, and deleting the data table pairs and the data tables in all the data table pairs to be combined from the first set.
Further, the initial grouping module is specifically configured to:
determining the number of threads for initial grouping according to the number of the data tables to be migrated, wherein each thread is used for initially grouping the data tables in the self grouping range
Further, the initial grouping module is further configured to: when the number of the data tables to be migrated is not less than a preset threshold, the number of the threads is a first number; and when the number of the data tables to be migrated is less than the preset threshold, the number of the threads is a second number.
Further, the migration module specifically includes:
the sorting unit is used for sorting the new set in the recombination result;
the recombination migration unit is used for performing data table migration on the recombination result according to the sequence;
and the independent migration unit is used for migrating the data tables in the second set.
Further, the migration module further includes:
and the interface providing unit is used for sending the sequencing of the new set and a migration operation interface aiming at the new set to a front end, wherein the migration operation interface comprises one or more of a starting interface, a pausing interface and a stopping interface.
Further, the data migration apparatus further includes:
the inspection module is used for acquiring the number of the migrated data tables and comparing the number of the migrated data tables with the number of the data tables to be migrated which are acquired in advance; if the number is the same, determining that the migration is successful; and if the number is different, determining that the migration is wrong.
In a third aspect, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method provided in the first aspect when executing the program.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the method as provided in the first aspect.
The data migration method and device provided by the embodiment of the invention distribute the data table to one of three sets: and then, carrying out association analysis by utilizing the first set with the main key association and the third set with only the top level node to determine the data table associated with each top level node, thereby realizing the reorganization of the data tables, and finally carrying out data table migration by utilizing the reorganization result and the third set, so that the migration can be carried out quickly according to the association relationship of the data tables.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flow chart illustrating a data migration method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present invention;
fig. 3 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to overcome the above problems in the prior art, the inventive concept of the embodiment of the present invention is: because the data table can also be at most used as a slave table in one data table pair when being used as a master table, but not used as slave tables in a plurality of data table pairs, and meanwhile, if the data table is used as a master table and is not used as a slave table in other data table pairs, the data table is called a top level node, so that the data table is distributed to one of three sets: and then, carrying out association analysis by utilizing the first set with the main key association and the third set with only the top level node to determine the data table associated with each top level node, thereby realizing the reorganization of the data tables, and finally carrying out data table migration by utilizing the reorganization result and the third set, so that the migration can be carried out quickly according to the association relationship of the data tables.
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present invention, as shown in fig. 1, including:
s101, according to the external key of the data table to be migrated, performing initial grouping on the data table, wherein the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting data tables having no foreign key and not being top level nodes, and a third set for counting data tables having no foreign key and being top level nodes.
If a common key is a primary key in one relationship, then this common key is referred to as a foreign key of another relationship. It can be seen that the foreign key represents a relational association between two relationships. A table having a foreign key in another relationship as a primary key is referred to as a primary table, and a table having such a foreign key is referred to as a secondary table of the primary table. The foreign key is also called a foreign keyword.
The data table to be migrated is judged to have the foreign key and the main table in the foreign key relation, the data table is preliminarily divided into 3 sets, namely a first set for counting data table pairs with foreign key association, a second set for counting data tables without foreign keys and not top nodes and a third set for counting data tables without foreign keys and not top nodes; the first set stores identification pairs formed by unique identifications of a master table and a slave table which are associated with foreign keys, the second set stores unique identifications of data tables which do not have foreign keys and are not top level nodes, and the third set stores unique identifications of data tables which do not have foreign keys and are top level nodes.
For example, there are 7 data tables, where a, B, C, D, E, F, and G are unique identifiers of the 7 data tables, respectively, a primary table of B in the 7 data tables is a, C has no foreign key, a primary table of E is D, a primary table of F is E, and F has no foreign key, then { a, B }, { D, E }, { E, F } is recorded in the first set, { C, F } is recorded in the second set, and { a, D } is recorded in the third set.
S102, analyzing the incidence relation of the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result with a plurality of new sets.
It should be understood that the statistics in the first set are only discrete data table pairs, and the statistics in the third set are discrete data tables, so that there is a need to further analyze the association relationship between the data tables in the first set and the third set, and recombine the first set and the third set to obtain a recombined result with a plurality of new sets. Wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
Continuing with the example of 7 data tables above, the new set obtained is { A, B } and { D, E, F }, for the new set { A, B }, since B's primary table is A, there is a foreign key association between A and B, while neither A nor B has a foreign key association with any of { D, E, F }. Similarly, for the new set { D, E, F }, the main table of E is D, the main table of F is E, D, E, F all can find the data table with the foreign key association from the set, but there is no data table with the foreign key association in the other new set { A, B }.
S103, data migration is carried out according to the recombination result and the second set.
Considering that each new set in the reorganization result has at least two data tables, the embodiment of the present invention numbers the reorganization result first, performs data migration according to the numbers, and then performs data migration on the data tables in the second set.
On the basis of the foregoing embodiments, as an optional embodiment, the analyzing an association relationship between data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result having a plurality of new sets specifically includes:
s201, creating a new set with the number equal to that of the data tables in the third set, and taking each data table in the third set as a first element of the corresponding new set.
It should be understood that, since the data tables in the third set do not directly have a foreign key association, the data tables can be considered to be discrete, and therefore, a corresponding new set is set for each data table and is used as the first element of the new set. Taking the above 7 data tables as an example, since there are two data tables of a and D in the third set, two new sets are constructed, namely { a } and { D }, respectively.
S202, for any data table pair in the first set, searching whether a main table in the data table pair exists in the third set.
S203, if the main table in the data table pair exists in the third set, placing the auxiliary table in the data table pair in the new set where the main table in the data table pair is located, and deleting the data table pair from the first set.
Taking the above 7 data tables as an example, three data table pairs of { a, B }, { D, E }, and { E, F } exist in the first set, and for the data table pair { a, B }, the master table is data table a, and whether data table a exists is looked up in the third set. Since there is data table A in the third set, the table will be placed in the new set of Table A, resulting in { A, B }, and the data table pair deleted from the first set.
S204, if the third set does not have the main table in the data table pairs, searching the data table pair of the main table as a slave table in the first set as a first data table pair to be combined, continuously searching whether the main table in the data table pair to be combined exists in the third set, if the main table does not exist, continuously searching the main table in the data table pair to be combined in the first set as a data table pair of the slave table as a next data table pair to be combined until the main table in the searched data table pair exists in the third set, placing the data table pairs and all data tables in the data table pairs to be combined in a new set where the main table of the last data table pair to be combined is located, and deleting the data table pairs and all data tables in the data table pairs to be combined from the first set.
For the data table pair { E, F }, the master table is E, but the data table E does not count in the third set, whether a data table pair with the slave table as E exists in the first set is searched, the slave table of the data table pair { D, E } is found to be E, and the data table D also counts in the third set, so that the data tables E and F in the data table pair { E, F } and the data table pair { D, E } are both placed in the new set { D }, a new { D, E, F } is obtained, and then the { D, E } and the { E, F } in the first set are deleted.
On the basis of the foregoing embodiments, as an optional embodiment, the initially grouping the data table according to the foreign key of the data table to be migrated includes:
and determining the number of threads for initial grouping according to the number of the data tables to be migrated, wherein each thread is used for initially grouping the data tables in the self grouping range.
For example, when there are 1000 data tables to be migrated, 10 threads are started, each thread for initially grouping 100 unrepeated data tables.
On the basis of the foregoing embodiments, as an optional embodiment, the determining the number of threads performing initial grouping according to the number of data tables to be migrated specifically includes:
when the number of the data tables to be migrated is not less than a preset threshold, the number of the threads is a first number; and when the number of the data tables to be migrated is less than the preset threshold, the number of the threads is a second number.
On the basis of the foregoing embodiments, as an optional embodiment, the performing data migration according to the reorganization result and the second set specifically includes:
and sequencing the new set in the recombination result, performing data table migration on the recombination result according to the sequencing, and then migrating the data table in the second set.
Taking the above 7 data tables as an example, taking { a, B } as a first new set, taking { D, E, F } as a second new set, after the data tables in the first new set are migrated, migrating the data tables in the second and new sets, then leaving the data tables { C, G } in the second set, migrating C and C in sequence, and finally completing the migration of all the data tables.
On the basis of the foregoing embodiments, as an optional embodiment, the sorting the new set in the recomposed result further includes: sending the ordering of the new set and a migration operation interface for the new set to a front end, the migration operation interface including one or more of a start interface, a pause interface, and a stop interface.
According to the embodiment of the invention, the sequence of the new set and the migration operation interface aiming at the new set are sent to the front end, so that all the data tables to be migrated are visually displayed by the front end, and the front end can freely start, pause and stop the migration of the new set according to the migration operation interface.
On the basis of the foregoing embodiments, as an optional embodiment, the performing data migration according to the reorganization result and the second set further includes:
acquiring the number of the migrated data tables, and comparing the number of the migrated data tables with the number of the data tables to be migrated which are acquired in advance;
if the number is the same, determining that the migration is successful; and if the number is different, determining that the migration is wrong.
Fig. 2 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present invention, and as shown in fig. 2, the data migration apparatus includes: an initial grouping module 201, a reorganizing module 202, and a migration module 203, wherein:
an initial grouping module 201, configured to perform initial grouping on a data table to be migrated according to a foreign key of the data table, where an initial grouping result includes: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes;
a regrouping module 202, configured to analyze an association relationship between data tables in the first set and the third set, regroup the first set and the third set, and obtain a regrouping result with a plurality of new sets;
a migration module 203, configured to perform data migration according to the recombination result and the second set;
wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
The data migration apparatus provided in the embodiment of the present invention specifically executes the flows of the data migration method embodiments, and please refer to the contents of the data migration method embodiments in detail, which is not described herein again. The data migration device provided by the embodiment of the invention distributes the data table to one of three sets: and then, carrying out association analysis by utilizing the first set with the main key association and the third set with only the top level node to determine the data table associated with each top level node, thereby realizing the reorganization of the data tables, and finally carrying out data table migration by utilizing the reorganization result and the third set, so that the migration can be carried out quickly according to the association relationship of the data tables.
On the basis of the foregoing embodiments, as an optional embodiment, the restructuring module specifically includes:
a set creating unit, configured to create a new set with the number equal to that of the data tables in the third set, and take each data table in the third set as a first element of the corresponding new set;
a main table lookup unit, configured to, for any data table pair in the first set, lookup whether a main table in the data table pair exists in the third set;
a first arrangement unit, configured to, if a master table in the data table pair exists in the third set, place a slave table in the data table pair in a new set where the master table in the data table pair is located, and delete the data table pair from the first set;
a second homing unit to, if no master table of the pair of data tables exists in the third set, the data table pair of the master table as the slave table is looked up in the first set as the first data table pair to be combined, continuously searching whether a main table in the data table pairs to be combined exists in the third set, if not, continuously searching the main table in the data table pairs to be combined in the first set as a data table pair of a slave table as a next data table pair to be combined until the main table in the searched data table pair exists in the third set, placing the data table pairs and all data tables in the data table pairs to be combined in a new set where the main table of the last data table pair to be combined is located, and deleting the data table pairs and the data tables in all the data table pairs to be combined from the first set.
On the basis of the foregoing embodiments, as an optional embodiment, the initial grouping module is specifically configured to:
determining the number of threads for initial grouping according to the number of the data tables to be migrated, wherein each thread is used for initially grouping the data tables in the self grouping range
On the basis of the foregoing embodiments, as an optional embodiment, the initial grouping module is further configured to: when the number of the data tables to be migrated is not less than a preset threshold, the number of the threads is a first number; and when the number of the data tables to be migrated is less than the preset threshold, the number of the threads is a second number.
On the basis of the foregoing embodiments, as an optional embodiment, the migration module specifically includes:
the sorting unit is used for sorting the new set in the recombination result;
the recombination migration unit is used for performing data table migration on the recombination result according to the sequence;
and the independent migration unit is used for migrating the data tables in the second set.
On the basis of the foregoing embodiments, as an optional embodiment, the migration module further includes:
and the interface providing unit is used for sending the sequencing of the new set and a migration operation interface aiming at the new set to a front end, wherein the migration operation interface comprises one or more of a starting interface, a pausing interface and a stopping interface.
On the basis of the foregoing embodiments, as an optional embodiment, the data migration apparatus further includes:
the inspection module is used for acquiring the number of the migrated data tables and comparing the number of the migrated data tables with the number of the data tables to be migrated which are acquired in advance; if the number is the same, determining that the migration is successful; and if the number is different, determining that the migration is wrong.
Fig. 3 is a schematic entity structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 3, the electronic device may include: a processor (processor)310, a communication Interface (communication Interface)320, a memory (memory)330 and a communication bus 340, wherein the processor 310, the communication Interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may invoke a computer program stored on the memory 330 and executable on the processor 310 to perform the data migration methods provided by the various embodiments described above, including, for example: according to the foreign key of the data table to be migrated, initially grouping the data table, wherein the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes; analyzing the incidence relation of the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result with a plurality of new sets; carrying out data migration according to the recombination result and the second set; wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
In addition, the logic instructions in the memory 330 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or make a contribution to the prior art, or may be implemented in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
An embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the data migration method provided in the foregoing embodiments when executed by a processor, and the method includes: according to the foreign key of the data table to be migrated, initially grouping the data table, wherein the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes; analyzing the incidence relation of the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result with a plurality of new sets; carrying out data migration according to the recombination result and the second set; wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of data migration, comprising:
according to the foreign key of the data table to be migrated, initially grouping the data table, wherein the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes;
analyzing the incidence relation of the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result with a plurality of new sets;
carrying out data migration according to the recombination result and the second set;
wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
2. The data migration method according to claim 1, wherein the analyzing the association relationship between the data tables in the first set and the third set, and recombining the first set and the third set to obtain a recombination result having a plurality of new sets specifically includes:
creating a new set with the number equal to that of the data tables in the third set, and taking each data table in the third set as a first element corresponding to the new set;
for any data table pair in the first set, searching whether a main table in the data table pair exists in the third set;
if the main table in the data table pair exists in the third set, placing the auxiliary table in the data table pair in a new set where the main table in the data table pair is located, and deleting the data table pair from the first set;
if the main table in the data table pair does not exist in the third set, searching the data table pair of the main table as a slave table in the first set as a first data table pair to be combined, continuously searching whether the main table in the data table pair to be combined exists in the third set, if the main table does not exist, continuously searching the main table in the data table pair to be combined in the first set as a data table pair of a slave table as a next data table pair to be combined until the main table in the searched data table pair exists in the third set, placing the data table pair and all data tables in the data table pair to be combined in a new set where the main table of the last data table pair to be combined is located, and deleting the data table pair and all data tables in the data table pair to be combined from the first set.
3. The data migration method according to claim 1, wherein the initially grouping the data tables according to the foreign key of the data table to be migrated comprises:
and determining the number of threads for initial grouping according to the number of the data tables to be migrated, wherein each thread is used for initially grouping the data tables in the self grouping range.
4. The data migration method according to claim 3, wherein the determining the number of threads to perform initial grouping according to the number of the data tables to be migrated specifically includes:
when the number of the data tables to be migrated is not less than a preset threshold value, setting the number of the threads as a first number; and when the number of the data tables to be migrated is less than the preset threshold, the number of the threads is a second number.
5. The data migration method according to claim 1, wherein the data migration is performed according to the reorganization result and the second set, and specifically includes:
and sequencing the new set in the recombination result, performing data table migration on the recombination result according to the sequencing, and then migrating the data table in the second set.
6. The data migration method of claim 5, wherein the sorting the new sets in the recomposed results further comprises: sending the ordering of the new set and a migration operation interface for the new set to a front end, the migration operation interface including one or more of a start interface, a pause interface, and a stop interface.
7. The data migration method according to claim 1, wherein the data migration according to the reorganization result and the second set further comprises:
acquiring the number of the migrated data tables, and comparing the number of the migrated data tables with the number of the data tables to be migrated which are acquired in advance;
if the number is the same, determining that the migration is successful; and if the number is different, determining that the migration is wrong.
8. A data migration apparatus, comprising:
the initial grouping module is used for initially grouping the data table according to the external key of the data table to be migrated, and the initial grouping result comprises the following steps: a first set for counting pairs of data tables having a foreign key association, a second set for counting pairs of data tables having no foreign key and not being top level nodes, and a third set for counting pairs of data tables having no foreign key and being top level nodes;
the recombination module is used for analyzing the incidence relation of the data tables in the first set and the third set, recombining the first set and the third set and obtaining a recombination result with a plurality of new sets;
the migration module is used for carrying out data migration according to the recombination result and the second set;
wherein, in the recombination result, any one data table in each new set satisfies the following conditions: the data table has a foreign key association with the other data table in the set; and the data table does not have a foreign key association with any of the data tables in the other sets.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the data migration method according to any one of claims 1 to 7 are implemented when the program is executed by the processor.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the steps of the data migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010560502.5A CN111881110B (en) | 2020-06-18 | 2020-06-18 | Data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010560502.5A CN111881110B (en) | 2020-06-18 | 2020-06-18 | Data migration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111881110A true CN111881110A (en) | 2020-11-03 |
| CN111881110B CN111881110B (en) | 2024-04-09 |
Family
ID=73156830
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010560502.5A Active CN111881110B (en) | 2020-06-18 | 2020-06-18 | Data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111881110B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463132A (en) * | 2020-11-13 | 2021-03-09 | 四川新网银行股份有限公司 | Database switching tool and switching method |
| CN114595229A (en) * | 2022-03-14 | 2022-06-07 | 上海上讯信息技术股份有限公司 | Method and equipment for extracting table object |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160092596A1 (en) * | 2014-09-26 | 2016-03-31 | Samsung Sds Co., Ltd. | Database migration method and apparatus |
| CN106933906A (en) * | 2015-12-31 | 2017-07-07 | 北京国双科技有限公司 | The querying method and device of data multidimensional degree |
| CN106933823A (en) * | 2015-12-29 | 2017-07-07 | 北京国双科技有限公司 | Method of data synchronization and device |
| CN109241027A (en) * | 2018-07-23 | 2019-01-18 | 平安科技（深圳）有限公司 | Method, apparatus, electronic equipment and the computer readable storage medium of Data Migration |
| CN109697066A (en) * | 2018-12-28 | 2019-04-30 | 第四范式（北京）技术有限公司 | Realize the method and system of tables of data splicing and automatic training machine learning model |
-
2020
- 2020-06-18 CN CN202010560502.5A patent/CN111881110B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160092596A1 (en) * | 2014-09-26 | 2016-03-31 | Samsung Sds Co., Ltd. | Database migration method and apparatus |
| CN106933823A (en) * | 2015-12-29 | 2017-07-07 | 北京国双科技有限公司 | Method of data synchronization and device |
| CN106933906A (en) * | 2015-12-31 | 2017-07-07 | 北京国双科技有限公司 | The querying method and device of data multidimensional degree |
| CN109241027A (en) * | 2018-07-23 | 2019-01-18 | 平安科技（深圳）有限公司 | Method, apparatus, electronic equipment and the computer readable storage medium of Data Migration |
| CN109697066A (en) * | 2018-12-28 | 2019-04-30 | 第四范式（北京）技术有限公司 | Realize the method and system of tables of data splicing and automatic training machine learning model |
Non-Patent Citations (1)
| Title |
|---|
| 吕进来;杨秋琳;: "基于表集合划分算法的数据交换方法研究", 计算机工程与设计, no. 06, pages 2036 - 2040 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463132A (en) * | 2020-11-13 | 2021-03-09 | 四川新网银行股份有限公司 | Database switching tool and switching method |
| CN112463132B (en) * | 2020-11-13 | 2023-06-06 | 四川新网银行股份有限公司 | Database switching tool and switching method |
| CN114595229A (en) * | 2022-03-14 | 2022-06-07 | 上海上讯信息技术股份有限公司 | Method and equipment for extracting table object |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111881110B (en) | 2024-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9442979B2 (en) | Data analysis using multiple systems | |
| CN111125059B (en) | Data migration method and device, storage medium and server | |
| US8271523B2 (en) | Coordination server, data allocating method, and computer program product | |
| CN106611064B (en) | Data processing method and device for distributed relational database | |
| CN111767340B (en) | Data processing method, device, electronic equipment and medium | |
| CN106933836B (en) | Data storage method and system based on sub-tables | |
| CN110502546A (en) | A kind of data processing method and device | |
| US9454561B2 (en) | Method and a consistency checker for finding data inconsistencies in a data repository | |
| CN111881110A (en) | Data migration method and device | |
| CN110046181B (en) | Data routing method and device based on database distributed storage | |
| CN111625561A (en) | Data query method and device | |
| CN108073641B (en) | Method and device for querying data table | |
| CN112486532A (en) | Method and device for managing configuration file, electronic equipment and storage medium | |
| US10977150B2 (en) | Data analysis | |
| CN111737351A (en) | Transaction management method and device for distributed management system | |
| CN109492051B (en) | Pipe network data synchronization method and device | |
| CN116680402A (en) | Device detection method and device based on log file and electronic device | |
| CN111400370A (en) | Data monitoring method and device in data circulation, storage medium and server | |
| CN115840767A (en) | Data query method, device, equipment and storage medium | |
| CN107104829B (en) | Physical equipment matching distribution method and device based on network topology data | |
| CN110851437A (en) | Storage method, device and equipment | |
| CN111782634B (en) | Data distributed storage method, device, electronic equipment and storage medium | |
| CN110866037B (en) | Message filtering method and device | |
| CN113032368A (en) | Data migration method and device, storage medium and platform | |
| CN108572994B (en) | Data migration processing method and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |