CN114579534A - Data migration method and device and electronic equipment - Google Patents
Data migration method and device and electronic equipment Download PDFInfo
- Publication number
- CN114579534A CN114579534A CN202011304953.9A CN202011304953A CN114579534A CN 114579534 A CN114579534 A CN 114579534A CN 202011304953 A CN202011304953 A CN 202011304953A CN 114579534 A CN114579534 A CN 114579534A
- Authority
- CN
- China
- Prior art keywords
- data
- incremental data
- incremental
- moment
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data migration method, a data migration device and electronic equipment, and relates to the technical field of data migration, wherein the method comprises the steps of determining stock data and incremental data to be migrated in a source database; migrating the stock data to a target database; backing up the incremental data to a specified position until the moment when the stock data migration is completed; migrating the incremental data in the designated position to a target database; determining a third moment corresponding to the incremental data of the last backup, and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database. According to the embodiment of the invention, the incremental data is backed up simultaneously when the stock data is migrated, and the incremental data is successfully linked to the migration of the backup incremental data and other incremental data which are not backed up after the stock data is migrated, so that the incremental data can be completely migrated to the target database, and the success rate of data migration is improved.
Description
Technical Field
The present invention relates to the field of data migration technologies, and in particular, to a data migration method and apparatus, and an electronic device.
Background
MongoDB is a database based on distributed file storage, and aims to provide an extensible high-performance data storage solution for WEB applications. When MongoDB performs full-scale data migration, incremental migration realizes data synchronization by pulling an operation log (oplog). The oplog is limited space management realized by adopting a ring queue, when business writing is in a peak period, if stock data migration is slow, the earliest generated oplog may be covered by the latest generated oplog, and thus, when the stock data migration is finished, the incremental migration is restarted to find that the earlier generated oplog to be pulled is covered, so that a fault occurs in migration, and further the data migration fails.
On the whole, the existing montogdb stock migration method has the risk that an operation log which is not pulled yet is lost due to being covered, so that migration failure is caused, and the success rate of data migration of a database is reduced.
Disclosure of Invention
In view of this, the present invention provides a data migration method, an apparatus and an electronic device, which can alleviate the problem of data migration failure caused by the fact that an operation log that is not yet pulled is covered in a database migration process, and improve the success rate of data migration of a database.
In a first aspect, an embodiment of the present invention provides a data migration method, including: determining stock data and incremental data to be migrated in a source database; wherein the inventory data includes: storing the data of the source database before the first moment; the incremental data includes: storing the data of the source database at the first time and after the first time; migrating the stock data to a target database; backing up the incremental data to a specified position until a second moment is reached; wherein the second moment is the moment when the stock data migration is finished; migrating the incremental data in the designated position to a target database; determining a third moment corresponding to the last backed-up incremental data, and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database.
In an optional embodiment of the present invention, the migrating the inventory data to the target database includes: and pulling the stock data from the source database through a preset first process, and synchronizing the pulled stock data to the target database.
In an optional embodiment of the present invention, the step of pulling the inventory data from the source database through a preset first process includes: acquiring a logical database and a logical data table of the source database through a preset first process; and pulling the stock data according to the logic database and the logic data table.
In an optional embodiment of the present invention, the step of backing up the incremental data to the specified location until the second time is reached includes: and pulling the incremental data from the source database through a preset second process, and backing up the pulled incremental data to the local until a second moment is reached.
In an optional embodiment of the present invention, the step of pulling the incremental data from the source database through the preset second process includes: acquiring an operation log after the first moment from the source database through a preset second process; and pulling corresponding incremental data according to the acquired operation log.
In an optional embodiment of the present invention, the step of backing up the pulled incremental data to the local includes: caching the pulled incremental data into a local memory; monitoring the data volume of the cached incremental data in the local memory, and writing the cached incremental data into a local file when the data volume reaches a preset data volume threshold value.
In an optional embodiment of the present invention, the migrating the incremental data in the designated location to the target database includes: and sequentially reading the incremental data from the specified position according to the time sequence through a preset third process, and synchronizing the read incremental data to the target database.
In an alternative embodiment of the present invention, the method further includes: if receiving a first message notification sent by a process executing stock data migration, stopping backing up the incremental data to the specified position; wherein the first message notification is to indicate that the inventory data migration is complete.
In an optional embodiment of the present invention, the migrating the incremental data after the third time to the target database includes: and pulling the incremental data after the third moment from the source database by presetting a fourth process, and synchronizing the pulled incremental data to the target database.
In an alternative embodiment of the present invention, the method further includes: if a second message notification sent by the process executing the incremental data backup is received, starting the fourth process to synchronize the incremental data after the third moment to the target database; wherein the second message notification indicates that migration of the incremental data backed up in the designated location is complete.
In an optional embodiment of the present invention, the first time is a generation time of a latest oplog in the source database.
In a second aspect, an embodiment of the present invention further provides a data migration apparatus, including: the stock data and incremental data determining module is used for determining stock data and incremental data to be migrated in the source database; wherein the inventory data includes: storing data in a source database before a first moment; the incremental data includes: storing the data in the source database at the first time and after the first time; the stock data migration module is used for migrating the stock data to the target database; the incremental data backup module is used for backing up the incremental data to a specified position until reaching a second moment; wherein the second moment is the moment when the stock data migration is finished; the first incremental data migration module is used for migrating the incremental data in the designated position to the target database; the second incremental data migration module is used for determining a third moment corresponding to the last backed-up incremental data and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the data migration method.
In a fourth aspect, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to implement the data migration method described above.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a data migration method, a data migration device and electronic equipment, which are used for determining stock data and incremental data to be migrated in a source database; wherein the inventory data includes: storing the data of the source database before the first moment; the incremental data includes: storing the data in the source database at the first time and after the first time; migrating the stock data to a target database; backing up the incremental data to a specified position until a second moment is reached; wherein the second moment is the moment when the stock data migration is finished; migrating the incremental data in the designated position to a target database; determining a third moment corresponding to the incremental data of the last backup, and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database. In the method, when the stock data is migrated, the incremental data is backed up at the same time, and after the stock data is migrated, the migration to the backup incremental data and other incremental data which are not backed up is smoothly connected, so that the incremental data can be completely migrated to the target database, the problem that in the prior art, an operation log generated earlier is covered due to slow migration of the stock data, and further, the migration failure of part of the incremental data is caused is solved, and the success rate of data migration is improved.
Additional features and advantages of the disclosure will be set forth in the description which follows, or in part may be learned by the practice of the above-described techniques of the disclosure, or may be learned by practice of the disclosure.
In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of another data migration apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Icon: 21-stock data and incremental data determination module; 22-inventory data migration module; 23-an incremental data backup module; 24-a first incremental data migration module; 25-a second incremental data migration module; 31-a migration control module; 200-a data migration apparatus; 41-a processor; 42-a memory; 43-bus; 44-communication interface.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In view of the problem that incremental data migration fails due to the fact that an operation log which is not pulled yet is covered in an existing full data migration mode, the data migration method, the data migration device and the electronic device provided by the embodiment of the invention can be applied to various application scenarios where database migration is required. To facilitate understanding of the present embodiment, a detailed description will be given first of all of a data migration method disclosed in the present embodiment.
Referring to fig. 1, which is a schematic flow chart of a data migration method, as can be seen from fig. 1, the method includes the following steps:
step S102: determining stock data and incremental data to be migrated in a source database; wherein the inventory data includes: storing the data of the source database before the first moment; the incremental data includes: and storing the data of the source database at the first time and after the first time.
Here, the source database refers to a database that can directly provide the required original data or specific data in the data migration scenario, where the type of the source database may be SQL Server, PostgreSQL, mySQL, MongoDB, and the like, and is not limited herein. In this example, what corresponds to the source database is the target database, which refers to the database object to which the data is migrated, and the type of the target database is not limited.
In a scenario of data migration, stock data and incremental data are divided according to a time point defined by a user, for example, in this embodiment, a first time is taken as a divided time point, data stored in the source database before the first time is determined as stock data, data stored in the source database after the first time and the first time are determined as incremental data, and full-volume data refers to all data in the source database, including stock data and incremental data.
During the actual full data migration, the incremental data may change dynamically because the source database may continue to store new data. In one possible implementation, the first time may be a generation time of a latest operation log in the source database. For example, assuming that the full-volume data migration of the source database a is ready to be performed at the current time, the generation time t1 of the latest operation log in the source database at the current time may be obtained as the first time, so that the data stored in the source database before the time t1 is determined as stock data, and the data stored in the source database after the time t1 and the time t1 is determined as incremental data.
Step S104: migrating the stock data to a target database; backing up the incremental data to a specified position until a second moment is reached; wherein the second time is the time when the stock data migration is completed.
While migrating the inventory data to the target database, a backup of the incremental data is also performed, and the incremental data is backed up to a specified location, which may be a local storage, for example. Here, when the migration of the stock data is completed, the backup of the incremental data is stopped, that is, when the above-described second timing is reached, the backup of the incremental data to the specified position is stopped.
In one possible implementation, different processes may be provided for separately performing the stock data migration and the incremental data backup. For example, the stock data can be pulled from the source database by presetting a first process, and the pulled stock data is synchronized to the target database; and pulling the incremental data from the source database through a preset second process, and backing up the pulled incremental data to the local until a second moment is reached. When the incremental data is backed up, the incremental data stored earliest can be backed up first, and the incremental data stored later can be backed up later in sequence according to the time sequence.
Here, in the stock data migration, the synchronization of the pulled stock data to the target database may be realized as the following steps 11 to 12:
(11) acquiring a logical database and a logical data table of the source database through a preset first process;
(12) and pulling the stock data according to the Logical Databases (LDBs) and the Logical data tables.
The logical database provides another way to obtain data, and can read data from the database instead of Open SQL statements. The logical database is not a physical database capable of storing data, but a predefined database reading program used in the design of Advanced Business Application Programming (ABAP) report program.
Generally, a logical Database is mainly composed of three parts, namely, a Structure (Structure), a selection (Selections), and a Database program (Database program). The "structure" part determines the database tables and views from which data is extracted, and the hierarchical relationship between the tables and views (the hierarchy will determine the order of data reading). The "select" section defines a selection screen for the logical database, the layout of which is determined by the structure of the logical database, which is automatically embedded in the default selection screen once the logical database is linked to the reporting program. If the logical database is called by the function LDB _ PROCESS, the selection screen appears in the LDB _ PROCESS function as an interface parameter. And, the "database program" section contains the specific logic read from the database tables and the code to pass the read data to the logical database user, where the program is a collection of subroutines whose specific structure depends on the "structure" and "selection" of the logical database.
The logical data table is a virtual table, which may be mapped to a physical table, or may be composed of a plurality of physical tables, which may be from different data sources, and the logical data table corresponds to a view of the source database. Thus, based on the logical database and the logical data table of the source database, the required inventory data can be read from the source database.
In actual operation, when the incremental data is backed up to a specified position, the incremental data can be pulled from the source database through a preset second process, and the pulled incremental data is backed up locally until a second time is reached. Specifically, the incremental data may be backed up locally by the following steps 21-22:
(21) caching the pulled incremental data into a local memory;
(22) monitoring the data volume of the cached incremental data in the local memory, and writing the cached incremental data into a local file when the data volume reaches a preset data volume threshold value.
For example, if the data stored in the source database after time t1 and time t1 are incremental data, the second process sequentially pulls the incremental data in order from the data with the time stamp of time t1 and buffers the pulled incremental data in the local memory, and if the upper limit of the buffer is 30M, the buffered incremental data is written into the local file whenever the buffered incremental data in the local memory reaches 30M, and the process is repeated.
Here, the step of pulling the incremental data from the source database through the second process may be implemented by the following steps 31 to 32:
(31) acquiring an operation log after the first moment from the source database through a preset second process;
(32) and pulling corresponding incremental data according to the acquired operation log.
Here, the operation log (oplog) is an operation record for storing all data of a database (e.g., MongoDB, etc.), which is similar to the binlog log of mysql. Usually oplog is stored within oplog.rs set of local library, and when a log of the storage size is reached, the old record will be overwritten by the new one. Each oplog corresponds to an operation and also corresponds to a timestamp indicating the time of occurrence of the operation, e.g., for a logging operation of data B, the timestamp t2 in the corresponding oplog indicates: the time for storing the data B is t 2.
In addition, because the migration of the incremental data can be started after the migration of the stock data is finished, the backup of the incremental data is not needed any more, and in order to stop the backup of the incremental data in time, in one possible implementation, if the migration control center receives a first message notification sent by a process executing the migration of the stock data, the backup of the incremental data to the specified position is stopped; wherein the first message notification is to indicate inventory data migration is complete. For example, in this embodiment, assuming that the migration control center receives the first message notification sent by the first process, the migration control center may control the second process to stop backing up the incremental data.
Step S106: and migrating the incremental data in the designated position to a target database.
And stopping the backup of the incremental data at the moment because the stock data is migrated, thereby starting to migrate the incremental data at the specified position to the target database. In at least one possible embodiment, the incremental data may be sequentially read in time sequence from the above specified position by a preset third process, and the read incremental data may be synchronized to the target database.
Step S108: determining a third moment corresponding to the incremental data of the last backup, and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database.
In at least one possible implementation manner, the incremental data after the third time may be pulled from the source database by a preset fourth process, and the pulled incremental data is synchronized to the target database. Here, the third time may also be obtained according to an operation log (oplog) corresponding to the last backed-up incremental data, where a timestamp in the operation log indicates a time when the incremental data is stored in the source database, that is, the third time.
In an actual operation, in order to improve the efficiency of data migration, the migration of the incremental data that is not backed up needs to be started in time after the migration of the backed up incremental data is completed, and therefore, in a possible implementation, if the migration control center receives a second message notification sent by a process that executes the backup of the incremental data, the fourth process is started to synchronize the incremental data after the third time to the target database; wherein the second message notification indicates that migration of the incremental data backed up in the designated location is complete. For example, in this embodiment, assuming that the migration control center receives the second message notification sent by the third process, the fourth process is started to migrate the incremental data after the third time to the target database.
Thus, through the steps S102 to S108, it can be ensured that the incremental data is completely migrated to the target database, and a problem in the prior art that an operation log generated earlier is covered due to slow migration of stock data, thereby causing a failure in migration of part of the incremental data is avoided.
The data migration method provided by the embodiment of the invention determines stock data and incremental data to be migrated in a source database; wherein the inventory data includes: storing the data of the source database before the first moment; the incremental data includes: storing the data of the source database at the first time and after the first time; migrating the stock data to a target database; backing up the incremental data to a specified position until a second moment is reached; the second moment is the moment when the stock data migration is finished; migrating the incremental data in the designated position to a target database; determining a third moment corresponding to the incremental data of the last backup, and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database. In the method, incremental data are backed up simultaneously when stock data are migrated, and the incremental data are successfully linked to the migration of the backup incremental data and other incremental data which are not backed up after the stock data are migrated, so that the incremental data can be completely migrated to a target database, and the success rate of data migration is improved.
Corresponding to the data migration method shown in fig. 1, an embodiment of the present invention further provides a data migration apparatus, as shown in fig. 2, which is a schematic structural diagram of the apparatus, where the data migration apparatus 200 includes an inventory data and incremental data determining module 21, an inventory data migration module 22, an incremental data backup module 23, a first incremental data migration module 24, and a second incremental data migration module 25, where functions of the modules are as follows:
the stock data and incremental data determining module 21 is configured to determine stock data and incremental data to be migrated in the source database; wherein the inventory data includes: storing data in a source database before a first time; the incremental data includes: storing the data in the source database at the first time and after the first time;
an inventory data migration module 22 for migrating inventory data to a target database;
the incremental data backup module 23 is configured to backup the incremental data to the specified position until the second time is reached; wherein the second moment is the moment when the stock data migration is finished;
a first incremental data migration module 24, configured to migrate the incremental data in the designated location to the target database;
the second incremental data migration module 25 is configured to determine a third time corresponding to the last backed-up incremental data, and migrate the incremental data after the third time to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database.
In the data migration device 200, incremental data are backed up simultaneously when stock data are migrated, and after the stock data are migrated, migration to the backed-up incremental data and other incremental data not backed up is smoothly connected, so that the incremental data can be completely migrated to a target database, the problem that in the prior art, an operation log generated earlier is covered due to slow stock data migration, and further, partial incremental data migration fails is avoided, and the success rate of data migration is improved.
In one possible implementation, the inventory data migration module 22 is further configured to: and pulling the stock data from the source database through a preset first process, and synchronizing the pulled stock data to the target database.
In another possible implementation, the inventory data migration module 22 is further configured to: acquiring a logical database and a logical data table of the source database through a preset first process; and pulling the stock data according to the logic database and the logic data table.
In another possible embodiment, the incremental data backup module 23 is further configured to: and pulling the incremental data from the source database through a preset second process, and backing up the pulled incremental data to the local until a second moment is reached.
In another possible embodiment, the incremental data backup module 23 is further configured to: acquiring an operation log after the first moment from the source database through a preset second process; and pulling corresponding incremental data according to the acquired operation log.
In another possible embodiment, the incremental data backup module 23 is further configured to: caching the pulled incremental data into a local memory; monitoring the data volume of the cached incremental data in the local memory, and writing the cached incremental data into a local file when the data volume reaches a preset data volume threshold value.
In another possible implementation, the first incremental data migration module 24 is further configured to: and sequentially reading the incremental data from the specified position according to the time sequence through a preset third process, and synchronizing the read incremental data to the target database.
In another possible implementation, the apparatus further includes a migration control module configured to: if receiving a first message notification sent by a process executing stock data migration, stopping backing up the incremental data to the specified position; wherein the first message notification is to indicate that the inventory data migration is complete.
In another possible implementation, the second incremental data migration module 25 is further configured to: and pulling the incremental data after the third moment from the source database by presetting a fourth process, and synchronizing the pulled incremental data to the target database.
In another possible implementation, the migration control module is further configured to: if a second message notification sent by the process executing the incremental data backup is received, starting the fourth process to synchronize the incremental data after the third moment to the target database; wherein the second message notification indicates that migration of the incremental data backed up in the designated location is complete.
In another possible implementation, the first time is a generation time of a latest oplog in the source database.
The data migration apparatus provided in the embodiment of the present invention has the same implementation principle and technical effect as those of the foregoing data migration method embodiment, and for brief description, reference may be made to corresponding contents in the foregoing data migration method embodiment for a part of the embodiment of the data migration apparatus that is not mentioned.
In order to more clearly understand the data migration method provided by the foregoing embodiment, an application example of data migration in the montogdb is described below. Wherein the method is applied to the device shown in fig. 3, as can be seen from fig. 3, the device comprises: a migration control module 31, and a stock data migration module 22, an incremental data backup module 23, a first incremental data migration module 24, and a second incremental data migration module 25 respectively connected to the migration control module 31.
When the full-scale migration of the database of the source database DB is started, the method comprises the following steps:
(1) the migration control module 31 is started, and the migration control module 31 reads the latest time stamp t1 of the operation log (oplog) in the source database DB.
(2) The migration control module 31 starts the inventory data migration module 22, and the inventory data migration module 22 is responsible for synchronizing all inventory data to the target database. The inventory data migration module 22 first obtains all the logical database and logical data table information and then starts to synchronize all the existing data.
(3) At the same time, the migration control module 31 starts the incremental data backup module 23, and notifies the incremental data backup module 23 to cache the incremental data from the data corresponding to the time stamp of t 1. The incremental data backup module 23 is responsible for caching the pulled incremental data into a memory cache, wherein the upper limit of the cache is 50M, and the data in the cache is written into a local file whenever the 50M is pulled. Here, there may be a plurality of local files, and the generated file names may be named in an orderly increasing form.
(4) When the inventory data migration module 22 completes the inventory data migration, the inventory data migration module 22 notifies the migration control module 31, and the migration control module 31 starts the first incremental data migration module 24. The first incremental data migration module 24 is responsible for sequentially reading all local files and then synchronizing data in the files to the target database. When the first incremental data migration module 24 starts synchronizing the last of the local files, it notifies the migration control module 31, this message notification is labeled q1, and then starts synchronizing data again.
(5) When the migration control module 31 receives the notification q1 sent by the first incremental data migration module 24, it instructs the incremental data backup module 23 to stop backup.
(6) When the first incremental data migration module 24 completes the synchronization of the last file in the local backup files, the migration control module 31 is notified of the timestamp t2 of the oplog of the synchronized last incremental data, where the notification is labeled q2, and then the first incremental data migration module 24 automatically shuts down.
(7) When the migration control module 31 receives the notification q2 sent by the first incremental data migration module 24, it starts the first incremental data migration module 24, and notifies the first incremental data migration module 24 to start synchronizing the incremental data from the data corresponding to the time stamp of t 2. That is, the first incremental data migration module 24 pulls all the incremental data after the time stamp of t2, and directly synchronizes to the target database side until the migration is completed.
The data migration method provided by the embodiment of the invention has the same technical characteristics as the method in the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
An embodiment of the present invention further provides an electronic device, as shown in fig. 4, which is a schematic structural diagram of the electronic device, where the electronic device includes a processor 41 and a memory 42, the memory 42 stores machine executable instructions that can be executed by the processor 41, and the processor 41 executes the machine executable instructions to implement the data migration method.
In the embodiment shown in fig. 4, the electronic device further comprises a bus 43 and a communication interface 44, wherein the processor 41, the communication interface 44 and the memory 42 are connected by the bus.
The Memory 42 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 44 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
The processor 41 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 41. The Processor 41 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, etc. as is well known in the art. The storage medium is located in a memory, and the processor 41 reads information in the memory 42 and completes the steps of the data migration method of the foregoing embodiment in combination with hardware thereof.
An embodiment of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the data migration method, and specific implementation may refer to the foregoing method embodiment, and is not described herein again.
The data migration method, the data migration apparatus, and the computer program product of the electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the data migration method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (14)
1. A method of data migration, comprising:
determining stock data and incremental data to be migrated in a source database; wherein the inventory data comprises: storing the data of the source database before the first moment; the incremental data includes: the data stored in the source database at the first time and after the first time;
migrating the inventory data to a target database; backing up the incremental data to a specified position until a second moment is reached; wherein the second moment is the moment when the stock data migration is completed;
migrating the incremental data in the specified location to the target database;
determining a third moment corresponding to the last backed-up incremental data, and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database.
2. The data migration method of claim 1, wherein said step of migrating said inventory data to a target database comprises:
and pulling the stock data from the source database through a preset first process, and synchronizing the pulled stock data to a target database.
3. The data migration method according to claim 2, wherein the step of pulling the inventory data from the source database through a preset first process includes:
acquiring a logic database and a logic data table of the source database through a preset first process;
and pulling the stock data according to the logic database and the logic data table.
4. The method for data migration according to claim 1, wherein the step of backing up the incremental data to the designated location until the second time is reached comprises:
and pulling the incremental data from the source database through a preset second process, and backing up the pulled incremental data to the local until a second moment is reached.
5. The data migration method according to claim 4, wherein the step of pulling the incremental data from the source database through a preset second process comprises:
acquiring an operation log after the first time from the source database through a preset second process;
and pulling corresponding incremental data according to the acquired operation log.
6. The data migration method according to claim 4, wherein the step of backing up the pulled incremental data to the local comprises:
caching the pulled incremental data to a local memory;
monitoring the data volume of the cached incremental data in the local memory, and writing the cached incremental data into a local file when the data volume reaches a preset data volume threshold value.
7. The data migration method of claim 1, wherein said step of migrating said incremental data in said designated location to said target database comprises:
and sequentially reading the incremental data from the specified position according to a time sequence through a preset third process, and synchronizing the read incremental data to the target database.
8. The data migration method of claim 7, wherein the method further comprises:
if receiving a first message notification sent by a process executing stock data migration, stopping backing up the incremental data to the specified position; wherein the first message notification is to indicate that the inventory data migration is complete.
9. The data migration method according to claim 1, wherein the step of migrating the incremental data after the third time to the target database includes:
and pulling the incremental data after the third moment from the source database through a preset fourth process, and synchronizing the pulled incremental data to the target database.
10. The data migration method of claim 9, wherein the method further comprises:
if a second message notification sent by the process executing the incremental data backup is received, starting the fourth process to synchronize the incremental data after the third moment to the target database; wherein the second message notification is used to indicate that all incremental data backed up in the specified location have been migrated.
11. The data migration method according to any one of claims 1 to 9, wherein the first time is a generation time of a latest operation log in the source database.
12. A data migration apparatus, comprising:
the stock data and incremental data determining module is used for determining stock data and incremental data to be migrated in the source database; wherein the inventory data comprises: storing the data of the source database before the first moment; the incremental data includes: the data stored in the source database at the first time and after the first time;
the stock data migration module is used for migrating the stock data to a target database;
the incremental data backup module is used for backing up the incremental data to a specified position until a second moment is reached; wherein the second moment is the moment when the stock data migration is completed;
a first incremental data migration module to migrate the incremental data in the specified location to the target database;
the second incremental data migration module is used for determining a third moment corresponding to the last backed-up incremental data and migrating the incremental data after the third moment to the target database; and the third moment is the moment when the last backup incremental data is stored in the source database.
13. An electronic device, comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the data migration method of any one of claims 1 to 11.
14. A computer-readable storage medium having computer-executable instructions stored thereon which, when invoked and executed by a processor, cause the processor to implement the data migration method of any of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011304953.9A CN114579534A (en) | 2020-11-18 | 2020-11-18 | Data migration method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011304953.9A CN114579534A (en) | 2020-11-18 | 2020-11-18 | Data migration method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114579534A true CN114579534A (en) | 2022-06-03 |
Family
ID=81767315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011304953.9A Pending CN114579534A (en) | 2020-11-18 | 2020-11-18 | Data migration method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114579534A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116881371A (en) * | 2023-09-07 | 2023-10-13 | 北京逐风科技有限公司 | Data synchronization method, device, equipment and storage medium |
-
2020
- 2020-11-18 CN CN202011304953.9A patent/CN114579534A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116881371A (en) * | 2023-09-07 | 2023-10-13 | 北京逐风科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN116881371B (en) * | 2023-09-07 | 2023-11-14 | 北京逐风科技有限公司 | Data synchronization method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11245758B2 (en) | System and method for automatic cloud-based full-data restore to mobile devices | |
| US20190146946A1 (en) | Method and device for archiving block data of blockchain and method and device for querying the same | |
| US8838919B2 (en) | Controlling data lag in a replicated computer system | |
| US20180262566A1 (en) | Caching Method and System Based on Cache Cluster | |
| US11907561B2 (en) | Data backup method and apparatus | |
| WO2021016230A1 (en) | Disaster recovery systems and methods with low recovery point objectives | |
| CN111078667B (en) | Data migration method and related device | |
| CN111049928B (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| WO2020015366A1 (en) | Method and device for data migration | |
| CN105824846B (en) | Data migration method and device | |
| CN104156361A (en) | Method and system for achieving data synchronization | |
| CN113111129A (en) | Data synchronization method, device, equipment and storage medium | |
| CN112506710B (en) | Distributed file system data restoration method, device, equipment and storage medium | |
| CN114579534A (en) | Data migration method and device and electronic equipment | |
| CN113849352A (en) | Business data storage exception processing method and device and server | |
| CN111459913B (en) | Capacity expansion method and device of distributed database and electronic equipment | |
| CN107402841B (en) | Data restoration method and device for large-scale distributed file system | |
| US20190087104A1 (en) | Storage Scheme For A Distributed Storage System | |
| CN112187889A (en) | Data synchronization method, device and storage medium | |
| CN111382024B (en) | Database master-slave copy delay monitoring method, device and system | |
| US20230124036A1 (en) | In-place garbage collection for state machine replication | |
| CN109634930B (en) | Method and device for cleaning logs, storage medium and electronic device | |
| CN114328007B (en) | Container backup and restoration method, device and medium thereof | |
| CN112732370B (en) | Business process adjustment method and device | |
| US8775371B2 (en) | Synchronizing an auxiliary data system with a primary data system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |