CN114168545A - Data migration method and device - Google Patents

Data migration method and device Download PDF

Info

Publication number
CN114168545A
CN114168545A CN202111493397.9A CN202111493397A CN114168545A CN 114168545 A CN114168545 A CN 114168545A CN 202111493397 A CN202111493397 A CN 202111493397A CN 114168545 A CN114168545 A CN 114168545A
Authority
CN
China
Prior art keywords
database
migrated
file
imported
files
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111493397.9A
Other languages
Chinese (zh)
Inventor
彭文峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
Original Assignee
China Construction Bank Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp filed Critical China Construction Bank Corp
Priority to CN202111493397.9A priority Critical patent/CN114168545A/en
Publication of CN114168545A publication Critical patent/CN114168545A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems

Abstract

A data migration method and device are applicable to the technical field of data storage. The method comprises the following steps: exporting the file in the database to be migrated, splitting the exported file into at least two first subfiles, transcoding the at least two first subfiles in parallel to obtain at least two second subfiles matched with the file format of the database to be migrated, merging the at least two second subfiles to obtain a file to be imported, and importing the file to be imported into the database to be migrated. By splitting the files and then performing parallel transcoding, even if the data volume of the files in the database to be migrated is very large, the time consumed by file migration can be saved by simultaneously transcoding the split files, and the efficiency of data migration of a large database can be improved.

Description

Data migration method and device
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a data migration method and apparatus.
Background
In recent years, with the increase of traffic, the demand of database resources by large financial institutions is increasing, and old databases are gradually unable to meet the existing large traffic demand. Moreover, with the progress of localization of databases, migration of data in old databases for improving the security of data storage is becoming an urgent task for large financial institutions.
Existing database data migration schemes typically migrate all data in a database at once. However, this solution can only be applied to small databases, and because the amount of data in the small databases is small, one-time migration does not take too long. However, for a large financial institution, the data types in the database are numerous, the data volume is also numerous, and it obviously takes a long time to migrate all the data at one time, which is not beneficial to improving the efficiency of data migration.
Based on this, a data migration method is needed to improve the efficiency of data migration in a database and save the time required for data migration in a large database.
Disclosure of Invention
The application provides a data migration method and device, which are used for improving the efficiency of data migration in a database and saving the time required by data migration in a large database.
In a first aspect, the present application provides a data migration method, including: exporting files to be migrated from the database; splitting the exported file into at least two first subfiles; transcoding the at least two first subfiles in parallel to obtain at least two second subfiles, wherein the at least two second subfiles are matched with the file format of the database to be migrated; combining at least two second subfiles to obtain a file to be imported; and importing the file to be imported into the database to be migrated.
By splitting the files and then performing parallel transcoding, even if the data volume of the files in the database to be migrated is very large, the time consumed by file migration can be saved by simultaneously transcoding the split files, and the efficiency of data migration of a large database can be improved.
One possible implementation manner is that the to-be-migrated database comprises K sub-databases corresponding to K data types, wherein K is an integer greater than or equal to 2; and combining at least two second subfiles to obtain a file to be imported, wherein the file to be imported comprises: respectively combining at least two second sub-files according to the K data types to obtain files to be imported, which correspond to the K data types respectively; importing a file to be imported into a database to be migrated, comprising: and importing the files to be imported corresponding to the K data types into K sub-databases corresponding to the data types respectively.
Through the mode, the to-be-migrated database can be set as a distributed database, the to-be-migrated database can be directly merged to obtain the to-be-imported file corresponding to each sub-database by merging after transcoding according to the type of each sub-database in the distributed database, and further the to-be-imported file can be directly imported into the corresponding sub-database, so that compared with a scheme that a complete to-be-imported file is obtained through merging and then split, the processing flow can be effectively saved, and the data migration efficiency is further improved.
A possible implementation manner, which is to import files to be imported, which correspond to K data types, into K sub databases corresponding to the data types, respectively, includes: configuring an Internet protocol IP address of an import tool to be the same as an IP address of a database to be migrated; distributing an identifier for a file to be imported corresponding to each data type in the K data types, wherein the identifier is uniquely corresponding to a sub-database corresponding to the data type; and the control import tool imports the K files to be imported into the K sub-databases in the database to be migrated according to the identifiers of the K files to be imported.
By the mode, the file to be imported is imported into the database to be migrated by the import tool, so that the import process is simpler and easier to operate. And moreover, the identification is distributed to the file to be imported according to the data type, so that the import tool can be accurately imported into the corresponding sub-database according to the identification, the file with the error can be positioned and imported, and the problem can be solved in time.
A possible implementation manner of importing a file to be imported into a database to be migrated includes: importing the file to be imported into a database to be migrated according to the first script; when an error is introduced, generating a second script according to the part which is not successfully introduced in the file to be introduced; and according to the second script, the part which is not successfully imported is re-imported into the database to be migrated.
By the method, the imported script is replaced and executed when the error is imported, the file to be imported in the full text does not need to be re-imported, and only the part which is not successfully imported needs to be re-imported, so that the efficiency of file import is improved.
In one possible implementation, the method further includes: according to the data migration method, migrating the full amount of files in the database to be migrated to the database to be migrated; after the migration of the full files is completed, migrating a first incremental file generated in a first period from the beginning to the end of the migration of the full files in the database to be migrated to the database to be migrated according to a data migration method; stopping the external service of the database to be migrated after the migration of the first incremental file is completed; according to the data migration method, migrating a second incremental file generated in a second time period from the beginning of migrating the first incremental file to the end of migrating the first incremental file in the database to be migrated to the database to be migrated, wherein the duration of the second time period is less than that of the first time period; and after the migration of the second incremental file is completed, starting the external service to be migrated into the database.
By the method, the full-size files, the first incremental files generated in the process of migrating the full-size files and the second incremental files generated in the process of migrating the first incremental files are migrated to the database to be migrated one by one in three batches, and the service of the old database is finished after the first incremental files are migrated, so that not only can all the files be migrated to a new database without omission be ensured, but also the influence of data migration on database service can be avoided, and the method is favorable for flexibly realizing data migration of a large database.
In a second aspect, the present application provides a data migration apparatus, comprising: the export unit is used for exporting the files in the database to be migrated; the splitting unit is used for splitting the exported file into at least two first subfiles; the transcoding unit is used for transcoding the at least two first subfiles in parallel to obtain at least two second subfiles, and the at least two second subfiles are matched with the file format of the database to be migrated; the merging unit is used for merging at least two second subfiles to obtain a file to be imported; and the import unit is used for importing the file to be imported into the database to be migrated.
One possible implementation manner is that the to-be-migrated database comprises K sub-databases corresponding to K data types, wherein K is an integer greater than or equal to 2; the merging unit is further configured to merge the at least two second subfiles according to the K data types, respectively, to obtain files to be imported, which correspond to the K data types, respectively; the importing unit is further used for importing the files to be imported corresponding to the K data types into the K sub-databases corresponding to the data types respectively.
In one possible implementation manner, the importing unit is further configured to: configuring an Internet protocol IP address of an import tool to be the same as an IP address of a database to be migrated; distributing an identifier for a file to be imported corresponding to each data type in the K data types, wherein the identifier is uniquely corresponding to a sub-database corresponding to the data type; and the control import tool imports the K files to be imported into the K sub-databases in the database to be migrated according to the identifiers of the K files to be imported.
In one possible implementation manner, the importing unit is further configured to: importing the file to be imported into a database to be migrated according to the first script; when an error is introduced, generating a second script according to the part which is not successfully introduced in the file to be introduced; and according to the second script, the part which is not successfully imported is re-imported into the database to be migrated.
In a third aspect, the present application provides a computer-readable storage medium storing a computer program which, when executed, performs the method of any one of the above first aspects.
In a fourth aspect, the present application provides a computing device comprising: a memory for storing program instructions; and the processor is used for calling the program instructions stored in the memory and executing the method in any one of the designs of the first aspect according to the obtained program.
In a fifth aspect, the present application provides a computer program product for implementing the method as designed in any one of the first aspects above when the computer program product is run on a processor.
The advantageous effects of the second aspect to the fifth aspect can be found in any design of the first aspect, and are not described in detail herein.
Drawings
Fig. 1 schematically illustrates a scenario architecture provided by an embodiment of the present application;
fig. 2 is a schematic flowchart illustrating a data migration method provided by an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating an architecture of a database to be migrated according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating a time node of data migration according to an embodiment of the present application;
fig. 5 exemplarily shows a schematic diagram of a data migration apparatus provided by an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 exemplarily shows a scene architecture diagram provided in the embodiment of the present application, as shown in fig. 1, the scene includes a to-be-migrated database, a migration environment, and a to-be-migrated database, the to-be-migrated database is connected to the migration environment, the other end of the migration environment is connected to the to-be-migrated database, and the connection mode may be a wired or wireless mode.
For example, the database to be migrated may be a database of a financial institution, which is generally a foreign database, has high data processing efficiency, can generate a large amount of data per unit time, and can provide a large amount of service in a certain time. For users, the operating system of the database is simple and easy to learn, the efficiency of processing data is high, and errors are few. Therefore, over the years, the database has long been used in large financial institutions for storing data generated by the large financial institutions.
Further exemplary, the to-be-migrated database may be a distributed database. The distributed database is a combination of a database technology and a distributed technology, and particularly refers to a database which is geographically dispersed and is logically combined with database nodes belonging to the same system in a computer system to store data together. The distributed database has the coordination among the databases and the distribution of data. Moreover, the distributed database does not pay attention to centralized control of the system, but pays attention to autonomy of each database node. In order to reduce the workload and the possibility of system errors when a programmer writes a program, the distributed database usually does not consider the distribution of data, so that the distribution of system data can always keep transparency, and the transparency can prevent the correctness of the program from being affected when the data is migrated, as if the data is not distributed when the program is written.
Furthermore, in a distributed database, data redundancy is a desirable feature, unlike a typical centralized database system. The data redundancy is introduced into the distributed database based on two considerations, the first is to copy data at the required database nodes for improving local applicability, and the second is because if a certain database node has a system error, the system can be continuously used by operating the copied data in other database nodes before the database node is repaired, so that the effectiveness of the system is improved.
Based on the above advantages of the distributed database, the embodiment of the present application may preferably select the distributed database as the database to be migrated.
The following describes a specific process for migrating data in a database.
Fig. 2 exemplarily shows a flowchart of a data migration method provided in an embodiment of the present application, and as shown in fig. 2, the method includes:
step 201, exporting the file to be migrated from the database.
For example, a file in a to-be-migrated database may be exported to a migration environment, which is an intermediate environment for data migration, and may be an open database composed of a plurality of import and export tools and transcoding tools.
In a possible implementation manner, before exporting a file in a database to be migrated, the data volume of the file may be determined, and when the data volume is smaller than a threshold, the file may be directly exported, and when the data volume is larger than the threshold, the file may be divided into several smaller files before exporting.
For example, assuming that the file to be migrated out of the database is a transaction flow data table with a large data volume, when the transaction flow data table is derived, the transaction flow data table may be first divided into a plurality of small transaction flow data tables according to the generation time, and then the plurality of small transaction flow data tables are derived in sequence. For example, a node may be divided every three days according to the generation time of the transaction running data table, an export sequence table ordered according to the forward or reverse order of the generation time is generated, when the database file export is performed, the transaction running data table of three days is exported each time according to the export sequence table, and after all the transaction running data tables of the three days are exported, the transaction running data tables of the next three days are exported according to the export sequence until all the transaction files are exported unknown.
Further illustratively, after dividing the transaction pipeline data table into a plurality of small transaction pipeline data tables by generation time, all of the small transaction pipeline data tables may also be derived in parallel. For example, if the transaction flow data table is divided into 100 small transaction flow data tables according to a rule that one node is divided every month, 100 processes can be called to export the 100 small transaction flow data tables at the same time, so that the export time of the whole database file is effectively shortened, and the efficiency of data migration is improved.
It should be understood that the above manner of dividing the data table according to the generation time is only an example, and in some other embodiments, the data table may also be divided according to other information, for example, the data table is divided according to a field of the data table, or the data table is divided according to a type of the data table, and the like, which is not specifically limited in this application.
In a possible implementation manner, before the step 201, resource allocation may be performed on the data migration. Illustratively, the file engine is defined to allocate resources as TEMP-DKB-SORT, the file in the export database is allocated with resources TEMP-DKB-UNLOAD, the file transcoding is allocated with resources as TEMP-DKB-FORMAT, and the file is transmitted to the open server and allocated with resources as TEMP-DKB-XCOM.
Step 202, splitting the exported file into at least two first subfiles.
In one possible implementation, after the files in the database are imported into the migration environment, the files can be split, so that one file is split into at least two subfiles. Still by way of example in the above step 201, after the transaction flow data table is divided into 100 small transaction flow data tables according to the generation time and the 100 small transaction flow data tables are exported to the migration environment, each small transaction flow data table in the migration environment may be further divided into smaller transaction flow data tables, for example, the 100 small transaction flow data tables are further divided into 600 first subfiles. The splitting manner of each small data table may refer to the dividing manner of the data table in step 201, which is not described in detail herein.
And 203, transcoding the at least two first subfiles in parallel to obtain at least two second subfiles, wherein the at least two second subfiles are matched with the file format of the database to be migrated.
Still referring to the above example in step 202, after 600 first subfiles are obtained, the 600 first subfiles may be transcoded at the same time, and the file format of the first database corresponding to each first subfile is converted into the file format of the second database to obtain 600 second subfiles. For example, if the file format stored in the first database is integer and the file format stored in the database to be migrated is short integer, the first subfile with the integer file format becomes the second subfile with the short integer file format after transcoding, and the first subfile is the same as the file format stored in the database to be migrated, so that the file format can be smoothly stored in the database to be migrated.
And step 204, combining at least two second subfiles to obtain a file to be imported.
Step 205, importing the file to be imported into the database to be migrated.
Fig. 3 exemplarily shows a schematic architecture diagram of a to-be-migrated database provided in the embodiment of the present application, as shown in fig. 3, the to-be-migrated database includes K sub-databases (K is a positive integer greater than or equal to 2), and the K sub-databases are respectively used to store files of different data types, such as data tables. Illustratively, the data table may be a traffic type data table, a parameter type data table, a data type data table, or the like.
In a possible implementation manner, in the step 203, when at least two second sub-files are merged, the transcoded second sub-files may be further classified and merged according to K data types to obtain files to be imported corresponding to the K data types, so that in the step 204, the files to be imported of the K different data types may be imported into the K sub-databases corresponding to the data types respectively according to the data types. Therefore, the file to be imported is imported into the database to be migrated by using the import tool, so that the import process is simpler and easier to operate. And moreover, the identification is distributed to the file to be imported according to the data type, so that the import tool can be accurately imported into the corresponding sub-database according to the identification, the file with the error can be positioned and imported, and the problem can be solved in time.
Illustratively, the import operation may also be performed in dependence on an import tool, which may be a database base tool (DBTool). When DBTool is used as an introduction tool, a specific introduction process may be performed according to the following steps one to three:
the method comprises the following steps: when the tool is used to manage a database, an Internet Protocol (IP) address of an import tool is configured to be the same as an IP address of a database to be migrated.
Step two: and allocating an identifier for the file to be imported corresponding to each data type in the K data types, wherein the identifier is uniquely corresponding to the sub-database corresponding to the data type. Illustratively, the identifier may be in the form of an identifier cluster, where each identifier cluster corresponds to a sub-database of a data type, and an identifier corresponding to the data type of the file to be imported is allocated to the file to be imported corresponding to each data type. For example, an identification cluster is allocated to a file to be imported of the parameter class, where the identification cluster corresponds to a sub-database for storing parameter class data, an identification cluster is allocated to a file to be imported of the service class, where the identification cluster corresponds to a sub-database for storing service class data, and so on.
Step three: and the control import tool imports the K files to be imported into the K sub-databases in the database to be migrated according to the identifiers of the K files to be imported. For example, still taking the file to be imported in the step two as an example, during the importing, the importing tool may import the file to be imported into the sub-database storing the parameter class data according to the identifier cluster of the parameter class file, import the file to be imported into the sub-database storing the service class data according to the identifier cluster of the service class file, and so on.
It should be noted that the process of importing the K import files of different data types into the K child databases may be executed serially or in parallel, so as to save the import time and improve the efficiency of the whole data migration.
In the embodiment of the application, when the file to be imported is imported for the first time, the whole file to be imported can be imported into the database to be migrated according to the first script. The first script is used for generating a plurality of table structures for the file to be imported, and the table structures are imported separately in the importing process. However, during the import process, an import error may also occur due to a fault of an import tool, a change of an import environment, and the like, and at this time, a part of the table structures in the file to be imported may have been successfully imported, and another part of the table structures may not have been successfully imported. Thus, after the repair is completed, if the import is continued according to the first script, the already imported parts are duplicated.
Based on this, in an optional implementation manner, a second script may be generated according to the table structure that is not successfully imported, and the part that is not successfully imported may be re-imported according to the second script. For example, assuming that 100 table structures are generated for a file to be imported by the first script, when an error is imported, 60 table structures are successfully imported, and two remaining 40 table structures are not successfully imported, at this time, the second script may be generated according to the remaining 40 table structures which are not successfully imported, and then the 40 table structures may be re-imported into the database to be imported according to the second script. In this way, by replacing the script for executing the import when the error is imported, the file to be imported in the full text does not need to be re-imported, and only the part which is not successfully imported needs to be re-imported, which is beneficial to improving the efficiency of file import.
Further illustratively, a record of the success of the import or the error of the import may also be written into the execution log. For example, for a table structure that is successfully imported, the last row of the execution log may record ". su ccess"; for the table structure of the introduced error, the last line of the execution log may record "failure".
In this embodiment of the present application, the data migration method may be used to migrate data in a large database, and in order to ensure completeness of migration, a scheme of gradual migration according to nodes is further provided in this embodiment of the present application, and this scheme is described in detail below.
Fig. 4 exemplarily shows a time node schematic diagram for data migration according to the data migration method illustrated in fig. 2 provided by the embodiment of the present application, and as shown in fig. 4, the whole migration process may include the following three stages:
in the first stage, according to the data migration method illustrated in fig. 2, the full files in the database to be migrated are first migrated to the database to be migrated, where the full files are all data files generated in the database to be migrated before the data migration is started. In addition, in the first time period of the full file migration process, the to-be-migrated database is still served to the outside, so that a new file is still generated in the to-be-migrated database, the file generated in the time period is called as a first incremental file, and the data volume of the first incremental file is much smaller than that of the full file.
And a second stage, according to the data migration method illustrated in fig. 2, migrating the first incremental file in the database to be migrated to the database to be migrated. Since the data volume of the first incremental file is much smaller than that of the full file, the time for migrating the first incremental file to the database to be migrated is also relatively short. In addition, in a second time period of the first incremental file migration process, the to-be-migrated database is still served to the outside, so that a new file is still generated in the to-be-migrated database, the file generated in the time period is called as a second incremental file, and the data volume of the second incremental file is much smaller than that of the first incremental file. And after the first incremental file is migrated, stopping the external service when the database to be migrated is migrated.
And step three, according to the data migration method illustrated in fig. 2, migrating the second incremental file in the database to be migrated to the database to be migrated. Since the external service of the to-be-migrated database is stopped after the migration of the first incremental file is completed, no new file is generated in the to-be-migrated database, that is, until the migration of the second incremental file is completed, no new file to be migrated is generated in the to-be-migrated database, it is determined that all files in the to-be-migrated database have been migrated to the to-be-migrated database, and the to-be-migrated database can be started to serve the external service at this time.
In a specific example, when the data volume of the full-volume file is expected to be 10T, the 10T full-volume file is migrated to the database to be migrated for 7 days, and after 7 days, the 10T full-volume file is migrated; within the 7 days, the database to be migrated still serves the outside, the generated first incremental file is counted to be 1T, 3 days are needed for migrating the 1T first incremental file to the database to be migrated, and after 3 days, the 1T first incremental file is migrated completely; at this time, the to-be-migrated database stops external service. And then, migrating the second incremental file generated in the 3 days to the database to be migrated, predicting that the second incremental file has 0.5T, and migrating the 0.5T second incremental file to the database to be migrated for 1 day, wherein after 1 day, the 0.5T second incremental file is completely migrated. And transferring all data from the database to be migrated to the database to be migrated, and starting the database to be migrated to serve the outside.
In the above embodiment, the file in the database to be migrated is exported, the exported file is split into at least two first subfiles, and then the at least two first subfiles are transcoded in parallel to obtain at least two second subfiles matched with the file format of the database to be migrated, the at least two second subfiles are further merged to obtain the file to be imported, and the file to be imported is imported into the database to be migrated. By splitting the files and then performing parallel transcoding, even if the data volume of the files in the database to be migrated is very large, the time consumed by file migration can be saved by simultaneously transcoding the split files, and the efficiency of data migration of a large database can be improved.
Based on the same technical concept, the embodiment of the present application further provides a data migration apparatus, and the data migration apparatus may execute the flow of the data migration method provided in the foregoing embodiment.
Fig. 5 exemplarily illustrates a schematic diagram of a data migration apparatus provided in an embodiment of the present application, and as shown in fig. 5, the apparatus includes: an export unit 501, configured to export a file in a database to be migrated; a splitting unit 502, which splits the exported file into at least two first subfiles; the transcoding unit 503 is configured to transcode the at least two first subfiles in parallel to obtain at least two second subfiles, where the at least two second subfiles are matched with the file format of the database to be migrated; a merging unit 504, configured to merge at least two second subfiles to obtain a file to be imported; an importing unit 505, configured to import the file to be imported into the database to be migrated.
One possible implementation manner is that the to-be-migrated database comprises K sub-databases corresponding to K data types, wherein K is an integer greater than or equal to 2; the merging unit 504 is further configured to merge at least two second subfiles according to the K data types, respectively, to obtain files to be imported corresponding to the K data types; the importing unit is further used for importing the files to be imported corresponding to the K data types into the K sub-databases corresponding to the data types respectively.
In one possible implementation manner, the importing unit 505 is further configured to: configuring an Internet protocol IP address of an import tool to be the same as an IP address of a database to be migrated; distributing an identifier for a file to be imported corresponding to each data type in the K data types, wherein the identifier is uniquely corresponding to a sub-database corresponding to the data type; and the control import tool imports the K files to be imported into the K sub-databases in the database to be migrated according to the identifiers of the K files to be imported.
In one possible implementation manner, the importing unit 505 is further configured to: importing the file to be imported into a database to be migrated according to the first script; when an error is introduced, generating a second script according to the part which is not successfully introduced in the file to be introduced; and according to the second script, the part which is not successfully imported is re-imported into the database to be migrated.
Based on the same technical concept, an embodiment of the present invention further provides a computing device, including: a memory for storing program instructions;
and the processor is used for calling the program instructions stored in the memory and executing the method illustrated in the figure 2 according to the obtained program.
Based on the same technical concept, the embodiment of the present invention also provides a computer-readable storage medium, which when running on a processor, implements the method as illustrated in fig. 2.
Based on the same technical concept, the embodiment of the present invention also provides a computer program product, which when running on a processor implements the method illustrated in fig. 2.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims (12)

1. A method of data migration, comprising:
exporting files to be migrated from the database;
splitting the exported file into at least two first subfiles;
transcoding the at least two first subfiles in parallel to obtain at least two second subfiles, wherein the at least two second subfiles are matched with the file format of the database to be migrated;
combining the at least two second subfiles to obtain a file to be imported;
and importing the file to be imported into the database to be migrated.
2. The method of claim 1, wherein the to-be-migrated database includes K sub-databases corresponding to K data types, where K is an integer greater than or equal to 2;
the merging the at least two second subfiles to obtain the file to be imported includes:
respectively merging the at least two second subfiles according to the K data types to obtain files to be imported, which correspond to the K data types respectively;
the importing the file to be imported into the database to be migrated comprises:
and importing the files to be imported corresponding to the K data types into K sub-databases corresponding to the data types respectively.
3. The method according to claim 2, wherein the importing the files to be imported corresponding to the K data types into the K sub databases corresponding to the data types respectively comprises:
configuring an Internet protocol IP address of an import tool to be the same as the IP address of the database to be migrated;
distributing an identifier for a file to be imported corresponding to each data type in the K data types, wherein the identifier is uniquely corresponding to a sub-database corresponding to the data type;
and controlling the importing tool to import the K files to be imported into the K sub-databases in the database to be migrated according to the identifiers of the K files to be imported.
4. The method of claim 1, wherein the importing the file to be imported into the database to be migrated comprises:
importing the file to be imported into the database to be migrated according to a first script;
when an error is introduced, generating a second script according to the part which is not successfully introduced in the file to be introduced;
and according to the second script, the part which is not successfully imported is imported into the database to be migrated again.
5. The method of any of claims 1 to 4, further comprising:
according to the data migration method, migrating the full amount of files in the database to be migrated to the database to be migrated;
after the migration of the full file is completed, according to the data migration method, migrating a first incremental file generated in the database to be migrated within a first time period from the beginning to the end of the migration of the full file to the database to be migrated;
after the migration of the first incremental file is completed, stopping the external service of the database to be migrated;
according to the data migration method, migrating a second incremental file generated in a second time period from the beginning of migrating the first incremental file to the end of migrating the first incremental file in the to-be-migrated database to the to-be-migrated database, wherein the duration of the second time period is less than that of the first time period;
and after the second incremental file is migrated, starting the external service of the database to be migrated.
6. A data migration apparatus, comprising:
the export unit is used for exporting the files in the database to be migrated;
the splitting unit is used for splitting the exported file into at least two first subfiles;
the transcoding unit is used for transcoding the at least two first sub-files in parallel to obtain at least two second sub-files, and the at least two second sub-files are matched with the file format of the database to be migrated;
the merging unit is used for merging the at least two second subfiles to obtain a file to be imported;
and the import unit is used for importing the file to be imported into the database to be migrated.
7. The apparatus of claim 6, wherein the to-be-migrated database includes K sub-databases corresponding to K data types, where K is an integer greater than or equal to 2;
the merging unit is further configured to merge the at least two second subfiles according to the K data types, respectively, to obtain files to be imported, which correspond to the K data types, respectively;
the importing unit is further configured to import the files to be imported corresponding to the K data types into the K sub databases corresponding to the data types, respectively.
8. The apparatus of claim 7, wherein the importing unit is further to:
configuring an Internet protocol IP address of an import tool to be the same as the IP address of the database to be migrated;
distributing an identifier for a file to be imported corresponding to each data type in the K data types, wherein the identifier is uniquely corresponding to a sub-database corresponding to the data type;
and controlling the importing tool to import the K files to be imported into the K sub-databases in the database to be migrated according to the identifiers of the K files to be imported.
9. The apparatus of claim 6, wherein the import unit is further to:
importing the file to be imported into the database to be migrated according to a first script;
when an error is introduced, generating a second script according to the part which is not successfully introduced in the file to be introduced;
and according to the second script, the part which is not successfully imported is imported into the database to be migrated again.
10. A computer-readable storage medium, characterized in that it stores a computer program which, when executed, performs the method according to any one of claims 1 to 5.
11. A computing device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory to execute the method of any of claims 1 to 5 in accordance with the obtained program.
12. A computer program product, characterized in that it implements the method of any of claims 1 to 5 when run on a processor.
CN202111493397.9A 2021-12-08 2021-12-08 Data migration method and device Pending CN114168545A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111493397.9A CN114168545A (en) 2021-12-08 2021-12-08 Data migration method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111493397.9A CN114168545A (en) 2021-12-08 2021-12-08 Data migration method and device

Publications (1)

Publication Number Publication Date
CN114168545A true CN114168545A (en) 2022-03-11

Family

ID=80484429

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111493397.9A Pending CN114168545A (en) 2021-12-08 2021-12-08 Data migration method and device

Country Status (1)

Country Link
CN (1) CN114168545A (en)

Similar Documents

Publication Publication Date Title
US9602422B2 (en) Implementing fixed points in network state updates using generation numbers
CN107958010B (en) Method and system for online data migration
CN109656911A (en) Distributed variable-frequencypump Database Systems and its data processing method
US9563697B1 (en) Calculating differences between datasets having differing numbers of partitions
CN107016016B (en) Data processing method and device
CN103309904A (en) Method and device for generating data warehouse ETL (Extraction, Transformation and Loading) codes
CN104272258A (en) System and method for supporting implicit versioning in a transactional middleware machine environment
CN106055670A (en) Inter-system data migration method and device
CN106339475A (en) Distributed storage system for mass data
CN101593294A (en) The method and system of dynamic-configuration workflow
EP3442201A1 (en) Cloud platform construction method and cloud platform
US20190004844A1 (en) Cloud platform construction method and cloud platform
CN114942965B (en) Method and system for accelerating synchronous operation of main database and standby database
CN107798120B (en) Data conversion method and device
CN112783868A (en) Distributed database table structure gray scale upgrading method, device and system
CN104182295A (en) Data backup method and data backup device
CN111190814A (en) Software test case generation method and device, storage medium and terminal
CN113157701A (en) Dual-activity mechanism deployment method and device of ORACLE database
CN111078371B (en) Branch task reconstruction method and device based on Flowable process engine
CN114168545A (en) Data migration method and device
CN116303348A (en) Method and system for online migration of DRDS (data distribution system) database and table division based on cap
CN113612818B (en) Industrial app release system of low-code platform
CN115587141A (en) Database synchronization method and device
CN115495265A (en) Method for improving kafka consumption capacity based on hadoop
JP2018180706A (en) Support device and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination