CN112131177A - Data migration method and device, storage medium and electronic equipment - Google Patents
Data migration method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112131177A CN112131177A CN202010818251.6A CN202010818251A CN112131177A CN 112131177 A CN112131177 A CN 112131177A CN 202010818251 A CN202010818251 A CN 202010818251A CN 112131177 A CN112131177 A CN 112131177A
- Authority
- CN
- China
- Prior art keywords
- data
- library
- target
- source
- data file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/151—Transformation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data migration method and device, a storage medium and electronic equipment. The method comprises the steps of sending a data export instruction to a source library to be migrated; receiving data exported by a source library and generating a source data file according to the exported data; transcoding the source data file to obtain a target data file; and loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library, so that the technical problem that data migration cannot be performed between heterogeneous databases in the related technology is solved, and data migration between the heterogeneous databases is realized.
Description
Technical Field
The present invention relates to the field of data migration, and in particular, to a data migration method and apparatus, a storage medium, and an electronic device.
Background
Currently, data migration generally aims at databases with the same structure, and no technical scheme for solving data migration between heterogeneous databases exists.
Disclosure of Invention
The embodiment of the invention provides a data migration method and device, a storage medium and electronic equipment, and aims to at least solve the technical problem of how to perform data migration on a heterogeneous database in the related art.
According to an aspect of an embodiment of the present invention, there is provided a data migration method, including:
sending a data export instruction to a source library to be migrated;
receiving data exported by the source library and generating a source data file according to the exported data;
transcoding the source data file to obtain a target data file;
loading the target data file to a target library, wherein the target library is a structurally different database than the source library.
According to another aspect of the embodiments of the present invention, there is also provided a data migration apparatus, including:
the sending module is used for sending a data export instruction to the source library to be migrated;
the receiving module is used for receiving the data exported by the source library and generating a source data file according to the exported data;
the transcoding module is used for transcoding the source data file to obtain a target data file;
and the loading module is used for loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library.
According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above data migration method when running.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the data migration method through the computer program.
In the embodiment of the invention, a data export instruction is sent to a source library to be migrated; receiving data exported by a source library and generating a source data file according to the exported data; transcoding the source data file to obtain a target data file; and loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library, so that the technical problem that data migration cannot be performed between heterogeneous databases in the related technology is solved, and data migration between the heterogeneous databases is realized.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow diagram of a method of data migration according to an embodiment of the present invention;
FIG. 2 is a block diagram of a data migration apparatus according to an embodiment of the present invention;
FIG. 3 is a flow diagram of a data migration method according to an example embodiment of the present invention;
fig. 4 is a schematic structural diagram of an alternative electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an aspect of the embodiments of the present invention, a data migration method is provided, and optionally, as an optional implementation manner, the data migration method in this embodiment may be executed by a computer program, and may be applied in a terminal or a server or a similar operation device or an electronic device. For example, the computing device or electronic device may include one or more processors (the processors may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory for storing data, and may further include a transmission device for communication functions and an input/output device. It may also include more or fewer components than previously described, or have a different configuration than that shown. The memory may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data migration method in the embodiment of the present invention, and the processor executes various functional applications and data processing by running the computer programs stored in the memory, so as to implement the method described above. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the mobile terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
Illustratively, as an alternative implementation manner, fig. 1 is a flowchart of a data migration method according to an embodiment of the present invention, and as shown in fig. 1, the data migration method includes:
step S101, sending a data export instruction to a source library to be migrated;
step S103, receiving the exported data of the source library and generating a source data file according to the exported data;
step S105, transcoding the source data file to obtain a target data file;
step S107, the target data file is loaded to a target library, wherein the target library is a database with a structure different from that of the source library.
Through the steps, a data export instruction is sent to the source library to be migrated; receiving data exported by a source library and generating a source data file according to the exported data; transcoding the source data file to obtain a target data file; and loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library, so that the technical problem that data migration cannot be performed between heterogeneous databases in the related technology is solved, and data migration between the heterogeneous databases is realized.
In an exemplary embodiment, the loading the target data file to the target repository includes: establishing a temporary table with the same table structure as the source library; migrating the source data file to the temporary table; and loading the temporary table to the target library.
In an exemplary embodiment, after loading the target data file to the target repository, the data migration method further comprises: sending export instructions to the source library and the target library respectively, wherein the export instructions are used for indicating the source library and the target library to export own data respectively; respectively receiving data exported by the source library and the target library to generate a source library data file and a target library data file, wherein the data formats of the source library data file and the target library data file are the same; and checking whether the data of the source library data file and the data of the target library data file are consistent.
In an exemplary embodiment, after the checking whether the data of the source repository data file and the target repository data file are consistent, the data migration method further includes: and determining the opening and closing of the access channels of the source library and the target library according to the verification result.
In an exemplary embodiment, determining the opening and closing of the access channels of the source library and the target library according to the checking result includes: and under the condition that the data of the source library data file and the data of the target library data file are verified to be consistent, closing the access channel of the source library and opening the access channel of the target library.
In an exemplary embodiment, after the checking whether the data of the source repository data file and the target repository data file are consistent, the data migration method further includes: and determining whether to block the data operation of the application server according to the verification result.
In an exemplary embodiment, verifying whether the data of the source repository data file and the target repository data file are consistent comprises at least one of: checking whether the data of the source database data file and the data of the target database data file are consistent or not through a data integrity algorithm; or, checking whether the data of the source database data file and the data of the target database data file are consistent through a file comparison algorithm; or, checking whether the data of the source library data file and the data of the target library data file are consistent through a timestamp judgment algorithm.
In this embodiment, a data migration apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted for brevity. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 2 is a block diagram of a data migration apparatus according to an embodiment of the present invention, as shown in fig. 2, the apparatus including:
a sending module 22, configured to send a data export instruction to a source library to be migrated;
a receiving module 24, configured to receive data exported by the source library and generate a source data file according to the exported data;
a transcoding module 26, configured to transcode the source data file to obtain a target data file;
a loading module 28, configured to load the target data file into a target library, where the target library is a database with a structure different from that of the source library.
Sending a data export instruction to a source library to be migrated through the module; receiving data exported by a source library and generating a source data file according to the exported data; transcoding the source data file to obtain a target data file; and loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library, so that the technical problem that data migration cannot be performed between heterogeneous databases in the related technology is solved, and data migration between the heterogeneous databases is realized.
In an exemplary embodiment, the load module 28 is configured to create a temporary table having the same table structure as the source library; migrating the source data file to the temporary table; and loading the temporary table to the target library.
In an exemplary embodiment, the data migration apparatus further includes: the verification module is used for respectively sending export instructions to the source library and the target library after the target data file is loaded to the target library, wherein the export instructions are respectively used for indicating the source library and the target library to export own data; respectively receiving data exported by the source library and the target library to generate a source library data file and a target library data file, wherein the data formats of the source library data file and the target library data file are the same; and checking whether the data of the source library data file and the data of the target library data file are consistent.
In an exemplary embodiment, the data migration apparatus further includes: and the opening and closing module is used for determining the opening and closing of the access channels of the source library and the target library according to the verification result after the data of the source library data file and the data of the target library data file are verified to be consistent.
In an exemplary embodiment, determining the opening and closing of the access channels of the source library and the target library according to the checking result includes: and under the condition that the data of the source library data file and the data of the target library data file are verified to be consistent, closing the access channel of the source library and opening the access channel of the target library.
In an exemplary embodiment, the data migration apparatus further includes: and the blocking module is used for determining whether to block the data operation of the application server according to a checking result after checking whether the data of the source library data file and the data of the target library data file are consistent.
In an exemplary embodiment, verifying whether the data of the source repository data file and the target repository data file are consistent comprises at least one of: checking whether the data of the source database data file and the data of the target database data file are consistent or not through a data integrity algorithm; or, checking whether the data of the source database data file and the data of the target database data file are consistent through a file comparison algorithm; or, checking whether the data of the source library data file and the data of the target library data file are consistent through a timestamp judgment algorithm.
It should be further noted that other embodiments of this embodiment are the same as those of the above embodiment, and are not described herein again.
Example embodiments
The following further explains embodiments of the present invention with reference to specific scenarios. The data migration method provided by the embodiment of the invention can properly solve the three main problems of the system for implementing data migration: firstly, the problem of completing data migration between heterogeneous databases in a non-shutdown state is solved; secondly, the problem of synchronously implementing data structure change in online data migration is solved; and thirdly, solving the problems of online data migration gray level and gradual switching according to flow.
The method provided by the embodiment of the invention realizes online data migration among heterogeneous databases and solves the problem that the online migration data structure cannot be changed in a large scale. The migration is accurate and efficient, and almost no sense is given to customers. The whole migration process can support gray scale and batch switching according to clients, can support real-time rollback, and provides powerful support for safe implementation of database switching.
The method is an online data migration method based on the time stamp. The method provided by the embodiment can be applied to the following migration implementation flows:
and (3) full migration: namely stock data migration, if the source base and the target base are isomorphic databases, the step can be implemented by a database mirror image backup and recovery mode; if the source library and the target library are heterogeneous databases, the method can be realized by methods of parallel exporting, importing scripts and the like. Typically some time before the system is brought into line.
Incremental migration: the step is a routine data migration step before the system is formally online, and is mainly used for adding incremental data to a target library at regular time to prevent the switching time from being overlong due to the fact that a large amount of data are accumulated on the formally online day. In order to realize incremental migration, an application system needs to perform certain treatment in advance, time stamp fields need to be maintained when data is added and modified in each table needing incremental migration, and it is guaranteed that a scene where data is not deleted in each table needing incremental migration or deleted data can be effectively traced.
Application online: and (4) putting on a new application program on the production day, realizing a public routing mechanism, and deciding whether to access the source library or the target library through the routing mechanism. Meanwhile, a public write-resistant mechanism is realized, and the application can temporarily prevent the operations of adding, deleting and changing the data of the migration table (or the client) when the write-resistant switch is switched on, so that conditions are created for the follow-up final leveling of the data.
And (3) write blocking: the method is realized by setting a write-resistant switch for the migration table (or the client), wherein the switch is turned on, and the data of the migration table (or the client) enters a static state.
And (3) blocking write migration: and data migration is carried out after the write-resistant switch is turned on, so that the target library data is leveled finally.
Stopping the write inhibition: the method is realized by setting a write-resistant switch for the migration table (or the client), and after the switch is closed, the data access of the application to the migration table (or the client) can be routed to the target library, and finally the online switching of the database system is completed.
Fig. 3 is a flow chart of a data migration method according to an exemplary embodiment of the present invention, as shown in fig. 3, in the method, a most core component is a "migration control component", which may be deployed separately on one or more migration servers that need to be able to remotely access a source repository, a target repository, and a REDIS cache cluster. The "migration control component" mainly accomplishes three main tasks, namely: data migration, data checking and data access control.
Data migration: and the migration control component remotely initiates a data export instruction to the source library at the migration server and generates a source data file at the migration server. Because the source database and the target database may be encoded differently, the source data files need to be transcoded uniformly by the data transcoding module. And the transcoded data file is launched by the migration control component to be remotely imported into a command and finally loaded into a target library. The whole data migration process supports the reconstruction of a target library data structure, and data extraction and data loading are realized by configuring corresponding SQL scripts.
Data checking: after the data is migrated and put in storage, the data check starts to work, and the module realizes end-to-end data check. And the migration control assembly remotely issues the same SQL to the source library and the target library to generate data files of the source library and the target library. And finally, the two data files are subjected to data consistency comparison through a series of processes such as md5, Linux diff, timestamp judgment and the like. The result of data check directly influences the selection of data route, if the data check finds inconsistent data, it indicates that data is lost or damaged in the migration process, we can choose not to modify the route switch, and when the problem is solved later, the route switching is performed after the data is migrated again.
Data access control: the method comprises setting of a write-resistant switch and a routing switch, and information of the two switches can be written into a redis cache in real time. The user of the write-resistant switch tells the application server whether the transaction can be carried out or not through addition and deletion; the routing switch tells the application server whether the table (or user) should access the source repository or the target repository. The two switches can be controlled by a manual script or a transfer controller. The database is truly migrated without halt through the two switches.
In the whole migration framework, certain transformation needs to be carried out on the application, the application needs to realize public write-blocking and routing logics, and the access logics of a source library and a target library are synchronously reserved in a data access layer, so that support is provided for realizing online real-time switching of a data table (user).
In an exemplary embodiment, in the data migration process in the method, if the structure of the target table changes on a large scale, the data extraction script of the source library is generally relatively complex, multiple table queries are often associated, the migration efficiency is relatively low, and the stable operation of the source library is easily affected. In this case, it may be considered that a temporary table with the same table structure as the source library is established in the target library, and during migration, data is migrated to the temporary table first, and then the target library is processed by using the efficient script.
By using the data migration method provided by the embodiment, under the conditions that the project source database is DB2 (the database is coded as GBK), the target database is MYSQL distributed database (the database is coded as UTF8MB4), the data volume is 2T, and the daily access frequency of the database is 5 hundred million times, the migration method is used, and under the condition of no service, the database access of 6000 million clients is successfully and smoothly switched to the distributed database according to the strategies of gray scale online and user batch flow switching. In the whole migration process, the accuracy of the migration data is 100%, the normal database reading operation of a user is not affected in the migration process, the influence time on the writing operation of the database is less than 3 minutes (the actual write-blocking transaction is less than 100), and the peak value of the TPS of the system after the migration is increased to 17000TPS from 5700 TPS.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
According to yet another aspect of the embodiment of the present invention, there is further provided an electronic device for implementing the data migration method, and fig. 4 is a schematic structural diagram of an alternative electronic device according to the embodiment of the present invention, as shown in fig. 4, the electronic device includes a memory 402 and a processor 404, the memory 402 stores a computer program, and the processor 404 is configured to execute the steps in any one of the method embodiments through the computer program.
Optionally, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of a computer network.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, sending a data export instruction to the source library to be migrated;
s2, receiving the data exported by the source library and generating a source data file according to the exported data;
s3, transcoding the source data file to obtain a target data file;
s4, loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library.
Through the steps, a data export instruction is sent to the source library to be migrated; receiving data exported by a source library and generating a source data file according to the exported data; transcoding the source data file to obtain a target data file; and loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library, so that the technical problem that data migration cannot be performed between heterogeneous databases in the related technology is solved, and data migration between the heterogeneous databases is realized.
Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 4 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, and a Mobile Internet Device (MID), a PAD, a server, etc. Fig. 4 is a diagram illustrating a structure of the electronic device and the electronic apparatus. For example, the electronic device, electronic equipment may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 4, or have a different configuration than shown in FIG. 4.
The memory 402 may be used to store software programs and modules, such as program instructions/modules corresponding to the data migration method and apparatus in the embodiment of the present invention, and the processor 404 executes various functional applications and data processing by running the software programs and modules stored in the memory 402, that is, implements the data migration method described above. The memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 402 may further include memory located remotely from the processor 404, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The storage 402 may be, but not limited to, specifically used for storing information such as sample characteristics of the item and the target virtual resource account number.
Optionally, the transmission device 406 is used for receiving or sending data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 406 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 406 is a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In addition, the electronic device further includes: a display 408 for displaying the information of the order to be processed; and a connection bus 410 for connecting the respective module parts in the above-described electronic apparatus.
In other embodiments, the terminal device or the server may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. Nodes can form a Peer-To-Peer (P2P, Peer To Peer) network, and any type of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the Peer-To-Peer network.
According to a further aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, sending a data export instruction to the source library to be migrated;
s2, receiving the data exported by the source library and generating a source data file according to the exported data;
s3, transcoding the source data file to obtain a target data file;
s4, loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library.
Through the steps, a data export instruction is sent to the source library to be migrated; receiving data exported by a source library and generating a source data file according to the exported data; transcoding the source data file to obtain a target data file; and loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library, so that the technical problem that data migration cannot be performed between heterogeneous databases in the related technology is solved, and data migration between the heterogeneous databases is realized.
Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in this application, it should be understood that the above-described apparatus embodiments are merely illustrative, for example, the division of the units is only one logical function division, and in actual implementation, there may be other division manners, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method of data migration, comprising:
sending a data export instruction to a source library to be migrated;
receiving data exported by the source library and generating a source data file according to the exported data;
transcoding the source data file to obtain a target data file;
loading the target data file to a target library, wherein the target library is a structurally different database than the source library.
2. The data migration method of claim 1, wherein the loading the target data file to a target repository comprises:
establishing a temporary table with the same table structure as the source library;
migrating the source data file to the temporary table;
loading the temporary table to the target library.
3. The data migration method of claim 1, wherein after loading the target data file to the target repository, the data migration method further comprises:
sending export instructions to the source library and the target library respectively, wherein the export instructions are used for indicating the source library and the target library to export own data respectively;
respectively receiving data exported by the source library and the target library to generate a source library data file and a target library data file, wherein the data formats of the source library data file and the target library data file are the same;
and checking whether the data of the source library data file and the data of the target library data file are consistent.
4. The data migration method according to claim 3, wherein after said checking whether the data of the source repository data file and the data of the target repository data file are consistent, the data migration method further comprises:
and determining the opening and closing of the access channels of the source library and the target library according to the verification result.
5. The data migration method according to claim 4, wherein the determining, according to the verification result, whether the access channels of the source repository and the target repository are opened or closed includes:
and under the condition that the data of the source library data file and the data of the target library data file are verified to be consistent, closing the access channel of the source library and opening the access channel of the target library.
6. The data migration method according to claim 3, wherein after said checking whether the data of the source repository data file and the data of the target repository data file are consistent, the data migration method further comprises:
and determining whether to block the data operation of the application server according to the verification result.
7. The data migration method according to any one of claims 3 to 6, wherein the checking whether the data of the source repository data file and the target repository data file are consistent comprises at least one of:
verifying whether the data of the source library data file and the data of the target library data file are consistent through a data integrity algorithm;
verifying whether the data of the source library data file and the data of the target library data file are consistent through a file comparison algorithm;
and verifying whether the data of the source library data file and the data of the target library data file are consistent or not through a timestamp judgment algorithm.
8. A data migration apparatus, comprising:
the sending module is used for sending a data export instruction to the source library to be migrated;
the receiving module is used for receiving the data exported by the source library and generating a source data file according to the exported data;
the transcoding module is used for transcoding the source data file to obtain a target data file;
and the loading module is used for loading the target data file to a target library, wherein the target library is a database with a structure different from that of the source library.
9. A computer-readable storage medium comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 7.
10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 7 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010818251.6A CN112131177A (en) | 2020-08-14 | 2020-08-14 | Data migration method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010818251.6A CN112131177A (en) | 2020-08-14 | 2020-08-14 | Data migration method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112131177A true CN112131177A (en) | 2020-12-25 |
Family
ID=73850825
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010818251.6A Pending CN112131177A (en) | 2020-08-14 | 2020-08-14 | Data migration method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131177A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055391A (en) * | 2021-03-25 | 2021-06-29 | 建信金融科技有限责任公司 | Method and device for policy configuration conversion during firewall replacement |
| CN113157670A (en) * | 2021-05-10 | 2021-07-23 | 中国工商银行股份有限公司 | Data migration method and system of database |
-
2020
- 2020-08-14 CN CN202010818251.6A patent/CN112131177A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055391A (en) * | 2021-03-25 | 2021-06-29 | 建信金融科技有限责任公司 | Method and device for policy configuration conversion during firewall replacement |
| CN113157670A (en) * | 2021-05-10 | 2021-07-23 | 中国工商银行股份有限公司 | Data migration method and system of database |
| CN113157670B (en) * | 2021-05-10 | 2024-02-09 | 中国工商银行股份有限公司 | Data migration method and system of database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110389900B (en) | Distributed database cluster testing method and device and storage medium | |
| JP6236533B2 (en) | Method and apparatus for creating differential update package, system differential update method and apparatus | |
| CN107205057A (en) | A kind of game resource update method, device and client | |
| CN107632827B (en) | Method and device for generating installation package of application | |
| CN108959385A (en) | Database portion arranging method, device, computer equipment and storage medium | |
| CN111064626B (en) | Configuration updating method, device, server and readable storage medium | |
| CN112131177A (en) | Data migration method and device, storage medium and electronic equipment | |
| CN103475721A (en) | System for updating digital assets and method thereof | |
| CN105320577B (en) | A kind of data backup and resume method, system and device | |
| CN108038153A (en) | The online data moving method and device of Hbase | |
| CN113127023B (en) | Service upgrading method, device and system | |
| CN112685391B (en) | Service data migration method and device, computer equipment and storage medium | |
| CN110442601A (en) | A kind of method and apparatus that Openstack mirror image data accelerates parallel | |
| CN104079610A (en) | Cloud server, and method and system for cloud synchronization of application software | |
| CN115129574A (en) | Code testing method and device | |
| CN111813346A (en) | Method, system, device and medium for establishing Ceph distributed storage based on cloud platform | |
| CN112905676A (en) | Data file importing method and device | |
| CN115129360A (en) | Code file management method and device, electronic equipment and storage medium | |
| CN108920936A (en) | A kind of method and apparatus that vmware certification is realized based on Purley platform | |
| CN114443294A (en) | Big data service component deployment method, system, terminal and storage medium | |
| CN114564211A (en) | Cluster deployment method, cluster deployment device, equipment and medium | |
| CN114675872A (en) | Data processing method, device and equipment for application program and storage medium | |
| CN105867961A (en) | System upgrading and degrading method and device | |
| CN115485677A (en) | Secure data replication in a distributed data storage environment | |
| CN112948354A (en) | Method and device for creating copy cluster, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |