CN116450602A - Data migration method and device, electronic equipment and storage medium - Google Patents
Data migration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116450602A CN116450602A CN202310219841.0A CN202310219841A CN116450602A CN 116450602 A CN116450602 A CN 116450602A CN 202310219841 A CN202310219841 A CN 202310219841A CN 116450602 A CN116450602 A CN 116450602A
- Authority
- CN
- China
- Prior art keywords
- row
- data
- target
- migrated
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 124
- 230000005012 migration Effects 0.000 title claims abstract description 124
- 238000000034 method Methods 0.000 title claims abstract description 93
- 230000008569 process Effects 0.000 claims abstract description 33
- 230000015654 memory Effects 0.000 claims description 46
- 238000004590 computer program Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 12
- 238000005516 engineering process Methods 0.000 claims description 10
- 238000012795 verification Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 8
- 238000012217 deletion Methods 0.000 claims 1
- 230000037430 deletion Effects 0.000 claims 1
- 238000013507 mapping Methods 0.000 abstract description 3
- 238000007405 data analysis Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 6
- 238000013524 data verification Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the field of data analysis, in particular to a data migration method, a data migration device, electronic equipment and a storage medium, which are used for improving the efficiency of data migration. Based on the uploaded configuration file, determining a preset data structure of each row in a source database and a preset data structure of each row in a target database; taking any row of data of the source database as a row to be migrated, taking any row of the target database as a target row, and executing for each row to be migrated: matching the row to be migrated with each target row based on the data structure to obtain a target row matched with the row to be migrated; and migrating the data in the row to be migrated to the matched target row. The mapping relation table between the source database and the target database is not required to be constructed in advance, and the data migration scene among various databases can be covered, so that the application is more universal, a large number of reads and writes are not required in the data migration process, and resources consumed in the data migration process are saved.
Description
Technical Field
The present disclosure relates to the field of data analysis technologies, and in particular, to a data migration method, a data migration device, an electronic device, and a storage medium.
Background
Different technology stacks and databases are typically used during data development, often requiring migration of data from one database to another, distributed storage databases are typically used during batch data processing, and relational databases are typically used during online real-time queries. In the data development work, different databases can use different data structures, a new data table cannot be directly created by using the data structure of the source data table, and after a target table is required to be created, the process of data migration is completed; however, this method requires a large number of read/write processes, and consumes a large amount of resources.
Disclosure of Invention
The application aims to provide a data migration method, a data migration device, electronic equipment and a storage medium, which are used for improving data migration efficiency.
In a first aspect, an embodiment of the present application provides a data migration method, where the method includes:
determining a preset data structure of each row in a source database and a preset corresponding data structure of each row in a target database based on a configuration file uploaded by a user; the data structures of each row in the source database are the same, the data structures of each row in the target database are the same, and the rows are rows or columns;
taking any row of data of the source database as a row to be migrated, taking any row of the target database as a target row, and executing the following data migration process for each row to be migrated:
matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated;
and migrating the data in the row to be migrated to the matched target row.
In the application, data migration is performed according to the data structure of each row of data in the source database and the data structure corresponding to each row in the target database, a mapping relation table between the source database and the target database is not required to be constructed in advance, and data migration scenes among various different databases can be basically covered, so that the application has universality, a large number of reading and writing are not required in the data migration process, and resources consumed in the data migration process are saved.
In some possible embodiments, after the migrating the data in the row to be migrated to the matched target row, the method further includes:
determining the memory occupied by the data stored in each target row;
taking the target row with the smallest occupied memory as a final target row;
and deleting the target rows except the final target row from the target database.
In the method, the final target row is determined through the memory occupied by each row of data, so that the storage space is saved.
In some possible embodiments, after the deleting the target rows other than the final target row from the target database, the method further includes:
verifying the data in the final target row with the data in the row to be migrated;
if the verification is consistent, determining that the data migration is successful;
if the data migration is inconsistent, determining that the data migration is failed, performing error reporting processing to delete the data in the final target row, and performing the data migration process again for the row to be migrated.
In the application, the accuracy and usability of data migration are ensured through a checking process.
In some possible embodiments, the verifying the data in the final target bank after the data is migrated with the data in the to-be-migrated bank includes:
comparing the data structure of the final target row with the data structure of the row to be migrated;
and comparing the numerical value corresponding to the data in the final target row with the numerical value sum corresponding to the data in the row to be migrated.
In the application, the accuracy of the data migration process is further ensured by comparing the data structure with the data content.
In some possible embodiments, after determining the data structure corresponding to each row of data in the source database, the method further includes:
determining a time range of data migration based on the configuration file;
and filtering the source database based on the time range to obtain a row to be migrated.
In the application, the method for filtering the source database based on the time range of data migration avoids the waste of resources caused by the migration of the full data.
In some possible embodiments, after determining the data structure corresponding to each row of data in the source database and the data structure of each row of the target database, the method further includes:
and establishing a link between the source database and the target database by adopting a remote dictionary technology, and determining that the network environment where the source database and the target database are positioned is normal.
In the method, the link between the source database and the target database is established by adopting the remote dictionary technology, so that the data migration speed is effectively improved.
In a second aspect, the present application further provides a data migration apparatus, the apparatus including:
the data structure determining module is used for determining a preset data structure of each row in the source database and a preset corresponding data structure of each row in the target database based on the configuration file uploaded by the user; the data structures of each row in the source database are the same, the data structures of each row in the target database are the same, and the rows are rows or columns;
the migration module is configured to take any row of data of the source database as a row to be migrated, take any row of data in the target database as a target row, and execute the following data migration process for each row to be migrated:
matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated;
and migrating the data in the row to be migrated to the matched target row.
In some possible embodiments, after the migration module performs migration of the data in the row to be migrated to the matching target row, the migration module is further configured to:
determining the memory occupied by the data stored in each target row;
taking the target row with the smallest occupied memory as a final target row;
and deleting the target rows except the final target row from the target database.
In some possible embodiments, after the migration module performs deleting the target rows other than the final target row from the target database, the migration module is further configured to:
verifying the data in the final target row with the data in the row to be migrated;
if the verification is consistent, determining that the data migration is successful;
if the data migration is inconsistent, determining that the data migration is failed, performing error reporting processing to delete the data in the final target row, and performing the data migration process again for the row to be migrated.
In some possible embodiments, when the migration module performs verification on the data in the final target bank after data migration and the data in the to-be-migrated bank, the migration module is specifically configured to:
comparing the data structure of the final target row with the data structure of the row to be migrated;
and comparing the numerical value corresponding to the data in the final target row with the numerical value sum corresponding to the data in the row to be migrated.
In some possible embodiments, after the data structure determining module performs determining a data structure corresponding to each row of data in the source database, the data structure determining module is further configured to:
determining a time range of data migration based on the configuration file;
and filtering the source database based on the time range to obtain a row to be migrated.
In some possible embodiments, after the data structure determining module performs determining the data structure corresponding to each row of data in the source database, the data structure determining module is further configured to:
and establishing a link between the source database and the target database by adopting a remote dictionary technology, and determining that the network environment where the source database and the target database are positioned is normal.
In a third aspect, another embodiment of the present application also provides an electronic device, including at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the methods provided by the embodiments of the first aspect of the present application.
In a fourth aspect, another embodiment of the present application further provides a computer readable storage medium, where the computer readable storage medium stores a computer program for causing a computer to perform any one of the methods provided by the embodiments of the first aspect of the present application.
In a fifth aspect, the present application provides a computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the method of any of the first aspects.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, and it is obvious that the drawings that are described below are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is an application scenario schematic diagram of a data migration method provided in an embodiment of the present application;
fig. 2A is an overall flow chart of a data migration method according to an embodiment of the present application;
FIG. 2B is a schematic diagram of a source database and a target database of a data migration method according to an embodiment of the present application;
fig. 2C is a schematic data migration flow chart of a data migration method according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of determining a final target bank according to a data migration method provided in an embodiment of the present application;
fig. 4 is a schematic diagram of a data verification flow of a data migration method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of data verification of a data migration method according to an embodiment of the present disclosure;
fig. 6 is a schematic device diagram of a data migration method according to an embodiment of the present application;
fig. 7 is a schematic diagram of an electronic device according to a data migration method provided in an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure. Embodiments and features of embodiments in this application may be combined with each other arbitrarily without conflict. Also, while a logical order of illustration is depicted in the flowchart, in some cases the steps shown or described may be performed in a different order than presented.
The terms first and second in the description and claims of the present application and in the above-described figures are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the term "include" and any variations thereof is intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus. The term "plurality" in the present application may mean at least two, for example, two, three or more, and embodiments of the present application are not limited.
In the technical scheme, the data are collected, transmitted, used and the like, and all meet the requirements of national related laws and regulations.
The inventors have found that different technology stacks and databases are typically used during data development, often requiring migration of data from one database to another, distributed storage databases are typically used during batch data processing, and relational databases are typically used during online real-time queries. In the data development work, different databases can use different data structures, a new data table cannot be directly created by using the data structure of the source data table, and after a target table is required to be created, the process of data migration is completed; however, this method requires a large number of read/write processes, and consumes a large amount of resources.
In view of the foregoing, the present application proposes a data migration method, apparatus, electronic device, and storage medium for solving the above-mentioned problems. The inventive concepts of the present application can be summarized as follows: determining a preset data structure of each row in a source database and a preset corresponding data structure of each row in a target database based on a configuration file uploaded by a user; the data structures of each row in the source database are the same, and the data structures of each row in the target database are the same and are arranged in rows or columns; taking any row of data of the source database as a row to be migrated, taking any row of the target database as a target row, and executing the following data migration process for each row to be migrated: matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated; and migrating the data in the row to be migrated to the matched target row.
For easy understanding, a data migration method provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings:
fig. 1 is an application scenario diagram of a data migration method in an embodiment of the present application. The drawings include: a server 10, a memory 20, and a terminal device 30; wherein:
the server 10 determines a data structure of each row preset in the source database and a corresponding data structure of each row preset in the target database based on the configuration file uploaded by the user in the terminal device 30; the data structures of each row in the source database are the same, the data structures of each row in the target database are the same, and the rows are rows or columns; taking any row of data of the source database as a row to be migrated, taking any row of the target database as a target row, and executing the following data migration process for each row to be migrated: matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated; and migrating the data in the row to be migrated to the matched target row and storing the data in the memory 20.
In the description herein, only a single server 10, memory 20, terminal device 30 is described in detail, but it should be understood by those skilled in the art that the illustrated server 10, memory 20, terminal device 30 are intended to represent the operation of the server 10, memory 20, terminal device 30 in relation to the technical solutions of the present application. And not implying a limitation on the number, type, location, etc. of servers 10, memories 20, terminal devices 30. It should be noted that the underlying concepts of the example embodiments of the present application are not altered if additional modules are added to or individual modules are removed from the illustrated environment. In addition, although a bi-directional arrow from the memory 20 to the server 10 is shown in fig. 1 for convenience of explanation, it will be understood by those skilled in the art that the above-described data transmission and reception is also required to be implemented through a network.
It should be noted that, the memory in the embodiments of the present application may be, for example, a cache system, or may also be a hard disk storage, a memory storage, or the like. In addition, the data migration method provided by the application is not only suitable for the application scene shown in fig. 1, but also suitable for any device with data migration requirements.
Fig. 2A is an overall flow chart of a data migration method according to an embodiment of the present application, where:
in step 201: determining a preset data structure of each row in a source database and a preset corresponding data structure of each row in a target database based on a configuration file uploaded by a user; the data structures of each row in the source database are the same, and the data structures of each row in the target database are the same and are arranged in rows or columns;
for example: as shown in fig. 2B, the data in the source database 1 are arranged vertically, and the data structures of each column of data are the same, so that each row of data in the source database 1 is each column of data; the data in the source data 2 are arranged horizontally, and the data structures of each row of data are the same, so that each row of data in the source database 2 is each row of data. Similarly, the target database 1 is arranged longitudinally, and each row of data in the target database 1 is each column of data; the target database 2 is arranged horizontally, and each row of data in the target database 2 is each column of data.
In step 202: and taking any row of data of the source database as a row to be migrated, taking any row of the target database as a target row, and executing a data migration process for each row to be migrated.
In this application, the data migration process may be implemented as shown in FIG. 2C, wherein:
in step 211: based on the data structure of the row to be migrated and the data structure of each target row, matching the row to be migrated with each target row to obtain at least one target row matched with the row to be migrated;
in step 212: and migrating the data in the row to be migrated to the matched target row.
In the application, data migration is performed according to the data structure of each row of data in the source database and the data structure corresponding to each row in the target database, a mapping relation table between the source database and the target database is not required to be constructed in advance, and data migration scenes among various different databases can be basically covered, so that the application has universality, a large number of reading and writing are not required in the data migration process, and resources consumed in the data migration process are saved.
In order to further understand a data migration process provided in the embodiments of the present application, the following details of the steps in fig. 2A and fig. 2C are described below:
in some possible embodiments, when matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row, there may be a plurality of target rows each matching the data structure of the row to be migrated, and in this application, in order to save storage space, therefore, after migrating the data in the row to be migrated to the matched target row, the steps shown in fig. 3 may be implemented, where:
in step 301: determining the memory occupied by the data stored in each target row;
in step 302: taking the target row with the smallest occupied memory as a final target row;
in step 303: the target rows other than the final target row are deleted from the target database.
For example: for the row 1 to be migrated, determining that a target row matched with the data structure of the row 1 to be migrated has a target row 1, a target row 2 and a target row 3; and migrating the data in the row 1 to be migrated to the target row 1, the target row 2 and the target row 3, then respectively determining the memories corresponding to the target row 1, the target row 2 and the target row 3, and deleting the target row 1 and the target row 3 from the target database if the target row with the smallest occupied memory is determined to be the target row 2.
In the method, the final target row is determined through the memory occupied by each row of data, so that the storage space is saved.
In some possible embodiments, to reduce errors in the data migration process, therefore, after target rows other than the final target row are deleted from the target database, steps as shown in FIG. 4 may be implemented to verify the migrated data:
in step 401: verifying the data in the final target row with the data in the row to be migrated;
in some possible embodiments, to ensure accuracy of the data verification, the data verification in this application includes, but is not limited to: comparing the data structure of the final target row with the data structure of the row to be migrated; and comparing the value corresponding to the data in the final target row with the sum of the values corresponding to the data in the row to be migrated.
For example: as shown in fig. 5, the source database is a database arranged in a longitudinal direction, and the included row to be migrated is: the target databases are longitudinally arranged databases, and the migrated target databases comprise: a final target column 1 corresponding to the column 1 to be migrated, a final target column 2 corresponding to the column 2 to be migrated, and a final target column 3 corresponding to the column 3 to be migrated; then, for the column 1 to be migrated, the data structure of the column 1 to be migrated is compared with the data structure of the final target column 1, and the sum of the data in the column 1 to be migrated is compared with the sum of the data in the final target column 1. In order to further ensure the accuracy of the data, each data in the column 1 to be migrated can be compared with each data in the final target column 1 respectively, namely: and comparing the first row data in the column 1 to be migrated with the first row data in the final target column 1, comparing the second row data in the column 1 to be migrated with the second row data in the final target column 1, and comparing the third row data in the column 1 to be migrated with the third row data in the final target column 1.
In step 402: if the verification is consistent, determining that the data migration is successful;
in step 403: if the data migration is inconsistent, determining that the data migration is failed, performing error reporting processing to delete the data in the final target row, and performing the data migration process again for the row to be migrated.
For example: the row to be migrated included in the source database is: the target database after migration comprises a column 1 to be migrated, a column 2 to be migrated and a column 3 to be migrated: a final target column 1 corresponding to the column 1 to be migrated, a final target column 2 corresponding to the column 2 to be migrated, and a final target column 3 corresponding to the column 3 to be migrated; the final target column 1, the final target column 2, and the final target column 3 need to be checked, respectively.
In the embodiment of the present application, in order to reduce resources consumed in the data migration process while ensuring the efficiency of data migration, in the present application, after determining the data structure corresponding to each row of data in the source database, implementation may be performed: determining a time range of data migration based on the configuration file; and filtering the source database based on the time range to obtain a row to be migrated. In the application, the method for filtering the source database based on the time range of data migration avoids the waste of resources caused by the migration of the full data.
For example: and (3) screening the data in the database based on the time range set by the user in the configuration file, wherein the time range is 2022.01.02-2022.02.02, and obtaining the data in the time period 2022.01.02-2022.02.02.
In the present application, the execution timing of the above steps is not limited, and may be performed after receiving the configuration file of the user, or may be performed after determining the data structure of each row of data, which is not limited in the present application, and the above execution timing is only one embodiment.
In the embodiment of the present application, in order to ensure the speed of data migration, after determining the source database and the target database based on the configuration file uploaded by the user, it is necessary to establish a link between the source database and the target database by using a remote dictionary technology (dis), and determine that the network environments where the source database and the target database are located are normal.
As shown in fig. 6, based on the same inventive concept, a data migration apparatus 600 is proposed, comprising:
a data structure determining module 6001, configured to determine, based on the configuration file uploaded by the user, a data structure of each row preset in the source database and a data structure corresponding to each row preset in the target database; the data structures of each row in the source database are the same, the data structures of each row in the target database are the same, and the rows are rows or columns;
a migration module 6002, configured to take any row of data of the source database as a row to be migrated, take any row of the target database as a target row, and execute, for each row to be migrated, the following data migration process:
matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated;
and migrating the data in the row to be migrated to the matched target row.
In some possible embodiments, after the migration module 6002 performs migration of the data in the row to be migrated to the matching target row, it is further configured to:
determining the memory occupied by the data stored in each target row;
taking the target row with the smallest occupied memory as a final target row;
and deleting the target rows except the final target row from the target database.
In some possible embodiments, after the migration module 6002 performs deleting the target rows other than the final target row from the target database, it is further configured to:
verifying the data in the final target row with the data in the row to be migrated;
if the verification is consistent, determining that the data migration is successful;
if the data migration is inconsistent, determining that the data migration is failed, performing error reporting processing to delete the data in the final target row, and performing the data migration process again for the row to be migrated.
In some possible embodiments, when the migration module 6002 performs verification of the data in the final target bank after data migration with the data in the bank to be migrated, the method specifically is used to:
comparing the data structure of the final target row with the data structure of the row to be migrated;
and comparing the numerical value corresponding to the data in the final target row with the numerical value sum corresponding to the data in the row to be migrated.
In some possible embodiments, after the data structure determining module 6001 performs determining a data structure corresponding to each row of data in the source database, the data structure determining module is further configured to:
determining a time range of data migration based on the configuration file;
and filtering the source database based on the time range to obtain a row to be migrated.
In some possible embodiments, after the data structure determining module 6001 performs determining the data structure corresponding to each row of data in the source database, and the data structure of each row of the target database, the data structure determining module is further configured to:
and establishing a link between the source database and the target database by adopting a remote dictionary technology, and determining that the network environment where the source database and the target database are positioned is normal.
Having described the data migration method and apparatus of the exemplary embodiments of the present application, next, an electronic device according to another exemplary embodiment of the present application is described.
Those skilled in the art will appreciate that the various aspects of the present application may be implemented as a system, method, or program product. Accordingly, aspects of the present application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
In some possible implementations, an electronic device according to the present application may include at least one processor, and at least one memory. The memory stores therein program code that, when executed by the processor, causes the processor to perform the steps in the data migration method according to various exemplary embodiments of the present application described above in this specification.
An electronic device 130 according to this embodiment of the present application is described below with reference to fig. 7. The electronic device 130 shown in fig. 7 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments herein.
As shown in fig. 7, the electronic device 130 is in the form of a general-purpose electronic device. Components of electronic device 130 may include, but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting the various system components, including the memory 132 and the processor 131.
Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, and a local bus using any of a variety of bus architectures.
Memory 132 may include readable media in the form of volatile memory such as Random Access Memory (RAM) 1321 and/or cache memory 1322, and may further include Read Only Memory (ROM) 1323.
Memory 132 may also include a program/utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with the electronic device 130, and/or any device (e.g., router, modem, etc.) that enables the electronic device 130 to communicate with one or more other electronic devices. Such communication may occur through an input/output (I/O) interface 135. Also, electronic device 130 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 136. As shown, network adapter 136 communicates with other modules for electronic device 130 over bus 133. It should be appreciated that although not shown in fig. 7, other hardware and/or software modules may be used in connection with electronic device 130, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
In some possible embodiments, aspects of a data migration method provided herein may also be implemented in the form of a program product comprising program code for causing a computer device to carry out the steps of a data migration method according to various exemplary embodiments of the present application as described herein above, when the program product is run on the computer device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product for data migration of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include program code and may run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device, partly on the remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic device may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., connected through the internet using an internet service provider).
It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functions of two or more of the elements described above may be embodied in one element in accordance with embodiments of the present application. Conversely, the features and functions of one unit described above may be further divided into a plurality of units to be embodied.
Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.
Claims (15)
1. A method of data migration, the method comprising:
determining a preset data structure of each row in a source database and a preset corresponding data structure of each row in a target database based on a configuration file uploaded by a user; the data structures of each row in the source database are the same, the data structures of each row in the target database are the same, and the rows are rows or columns;
taking any row of data of the source database as a row to be migrated, taking any row of the target database as a target row, and executing the following data migration process for each row to be migrated:
matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated;
and migrating the data in the row to be migrated to the matched target row.
2. The method of claim 1, wherein after the migrating the data in the row to be migrated to the matching target row, the method further comprises:
determining the memory occupied by the data stored in each target row;
taking the target row with the smallest occupied memory as a final target row;
and deleting the target rows except the final target row from the target database.
3. The method of claim 2, wherein after the deleting the target rows other than the final target row from the target database, the method further comprises:
verifying the data in the final target row with the data in the row to be migrated;
if the verification is consistent, determining that the data migration is successful;
if the data migration is inconsistent, determining that the data migration is failed, performing error reporting processing to delete the data in the final target row, and performing the data migration process again for the row to be migrated.
4. A method according to claim 3, wherein said verifying the data in the final destination bank after data migration with the data in the bank to be migrated comprises:
comparing the data structure of the final target row with the data structure of the row to be migrated;
and comparing the numerical value corresponding to the data in the final target row with the numerical value sum corresponding to the data in the row to be migrated.
5. The method of claim 1, wherein after determining the data structure corresponding to each row of data in the source database, the method further comprises:
determining a time range of data migration based on the configuration file;
and filtering the source database based on the time range to obtain a row to be migrated.
6. The method according to any one of claims 1 to 5, wherein after determining the data structure corresponding to each row of data in the source database and the data structure of each row of the target database, the method further comprises:
and establishing a link between the source database and the target database by adopting a remote dictionary technology, and determining that the network environment where the source database and the target database are positioned is normal.
7. A data migration apparatus, the apparatus comprising:
the data structure determining module is used for determining a preset data structure of each row in the source database and a preset corresponding data structure of each row in the target database based on the configuration file uploaded by the user; the data structures of each row in the source database are the same, the data structures of each row in the target database are the same, and the rows are rows or columns;
the migration module is configured to take any row of data of the source database as a row to be migrated, take any row of data in the target database as a target row, and execute the following data migration process for each row to be migrated:
matching the row to be migrated with each target row based on the data structure of the row to be migrated and the data structure of each target row to obtain at least one target row matched with the row to be migrated;
and migrating the data in the row to be migrated to the matched target row.
8. The apparatus of claim 7, wherein the migration module, after performing the migration of the data in the row to be migrated to the matching target row, is further configured to:
determining the memory occupied by the data stored in each target row;
taking the target row with the smallest occupied memory as a final target row;
and deleting the target rows except the final target row from the target database.
9. The apparatus of claim 8, wherein the migration module, after performing the deletion of the target tier other than the final target tier from the target database, is further to:
verifying the data in the final target row with the data in the row to be migrated;
if the verification is consistent, determining that the data migration is successful;
if the data migration is inconsistent, determining that the data migration is failed, performing error reporting processing to delete the data in the final target row, and performing the data migration process again for the row to be migrated.
10. The apparatus of claim 7, wherein the migration module is configured to, when performing the verification of the data in the final target bank after the data is migrated and the data in the bank to be migrated:
comparing the data structure of the final target row with the data structure of the row to be migrated;
and comparing the numerical value corresponding to the data in the final target row with the numerical value sum corresponding to the data in the row to be migrated.
11. The apparatus of claim 7, wherein the data structure determination module, after executing the determination of the data structure corresponding to each row of data in the source database, is further configured to:
determining a time range of data migration based on the configuration file;
and filtering the source database based on the time range to obtain a row to be migrated.
12. The apparatus according to any one of claims 7 to 11, wherein after the data structure determining module performs determining the data structure corresponding to each row of data in the source database, and the data structure of each row of the target database, the data structure determining module is further configured to:
and establishing a link between the source database and the target database by adopting a remote dictionary technology, and determining that the network environment where the source database and the target database are positioned is normal.
13. An electronic device comprising at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to implement the method of any of claims 1-6.
14. A computer storage medium, characterized in that the computer storage medium stores a computer program for enabling a computer to perform the method according to any one of claims 1-6.
15. A computer program product, the computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the method of any of the preceding claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310219841.0A CN116450602A (en) | 2023-03-09 | 2023-03-09 | Data migration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310219841.0A CN116450602A (en) | 2023-03-09 | 2023-03-09 | Data migration method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116450602A true CN116450602A (en) | 2023-07-18 |
Family
ID=87132759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310219841.0A Pending CN116450602A (en) | 2023-03-09 | 2023-03-09 | Data migration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116450602A (en) |
-
2023
- 2023-03-09 CN CN202310219841.0A patent/CN116450602A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109471851B (en) | Data processing method, device, server and storage medium | |
| CN107315814B (en) | Method and system for verifying data consistency after data migration of KDB (KDB) database | |
| EP2763055B1 (en) | A telecommunication method and mobile telecommunication device for providing data to a mobile application | |
| CN109634587B (en) | Method and equipment for generating warehousing script and warehousing data | |
| CN112068852B (en) | Method, system, equipment and medium for installing open-source software based on domestic server | |
| CN110555150B (en) | Data monitoring method, device, equipment and storage medium | |
| CN110825694A (en) | Data processing method, device, equipment and storage medium | |
| CN111124872A (en) | Branch detection method and device based on difference code analysis and storage medium | |
| CN110659210A (en) | Information acquisition method and device, electronic equipment and storage medium | |
| CN113297182A (en) | Data migration method, device, storage medium and program product | |
| CN110688111A (en) | Configuration method, device, server and storage medium of business process | |
| CN114442930A (en) | Method and device for controlling group disk array, electronic equipment and readable storage medium | |
| CN114116682A (en) | Method, device, system, equipment and medium for migrating satellite telemetry data | |
| CN118152401A (en) | Data storage method, apparatus, device, storage medium, and program product | |
| CN111143434A (en) | Intelligent data checking method, device, equipment and storage medium | |
| CN103559204A (en) | Database operation request processing method, unit and system | |
| CN112463574A (en) | Software testing method, device, system, equipment and storage medium | |
| CN116450602A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN109918231B (en) | Data reforming abnormity repairing method, device, equipment and storage medium | |
| CN114090514A (en) | Log retrieval method and device for distributed system | |
| CN112559331A (en) | Test method and device | |
| CN113760696A (en) | Program problem positioning method and device, electronic equipment and storage medium | |
| CN114253642A (en) | System information processing method, device, electronic equipment and medium | |
| CN111522832A (en) | Big data real-time updating method, system, equipment and storage medium | |
| CN112597058B (en) | Webpage testing method, device, equipment and storage medium based on volume creation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |