CN113138975A - Data migration method and device - Google Patents

Data migration method and device Download PDF

Info

Publication number
CN113138975A
CN113138975A CN202110515775.2A CN202110515775A CN113138975A CN 113138975 A CN113138975 A CN 113138975A CN 202110515775 A CN202110515775 A CN 202110515775A CN 113138975 A CN113138975 A CN 113138975A
Authority
CN
China
Prior art keywords
data
database
change
version
version information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110515775.2A
Other languages
Chinese (zh)
Inventor
安勇聪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Tuoxian Technology Co Ltd
Original Assignee
Beijing Jingdong Tuoxian Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Tuoxian Technology Co Ltd filed Critical Beijing Jingdong Tuoxian Technology Co Ltd
Priority to CN202110515775.2A priority Critical patent/CN113138975A/en
Publication of CN113138975A publication Critical patent/CN113138975A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/219—Managing data history or versioning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data migration method and device, and relates to the technical field of computers. One embodiment of the method comprises: receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request; synchronizing the existing data from the original database to the new database according to the version information of the existing data; acquiring changed data corresponding to the data change event, and synchronizing the changed data from a database corresponding to the data change event to a target database according to version information of the changed data; if the database corresponding to the data change event is the original database, the target database is the new database, and if the database corresponding to the data change event is the new database, the target database is the original database. The implementation method does not need to stop migration, can avoid the condition that new data is covered, and ensures dynamic migration of the data and consistency of the data.

Description

Data migration method and device
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data migration method and apparatus.
Background
With the user data reaching a certain magnitude, the problem that the original database cannot support the performance requirements required by the existing service exists. In this regard, it is necessary to migrate user data to a new database that can be expanded in capacity. At present, the following method is mainly used for data migration.
Firstly, service stopping migration, namely sending out a service stopping announcement and service stopping duration in advance, suspending service and executing static data migration. However, the user will not be able to use the service for a while, it is not user friendly, and the service is lost when the service is stopped, and it is necessary to complete within a specified time, and there is a certain working pressure.
And secondly, recording data changes by adopting the message table, firstly migrating static data in the original database, and then synchronizing the data changes in the message table into the new database. With this method, when the user data is changed frequently, new data is overwritten.
And thirdly, simultaneously transferring the static data and synchronizing the changed data in the original database in the new database. The method is easy to cause the condition of migration failure, cannot sense the migration failure and brings difficulty to the verification of data consistency. And if the migration of the database fails, the original database cannot be switched back in time, a good rollback scheme is not provided, and the user experience is poor.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data migration method and apparatus, which do not need to stop migration, and can also avoid a situation that new data is covered, thereby ensuring dynamic migration of data and consistency of data.
To achieve the above object, according to an aspect of an embodiment of the present invention, a data migration method is provided.
The data migration method of the embodiment of the invention comprises the following steps: receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request; synchronizing the existing data from the original database to the new database according to the version information of the existing data; acquiring change data corresponding to a data change event, and synchronizing the change data from a database corresponding to the data change event to a target database according to version information of the change data; if the database corresponding to the data change event is the original database, the target database is the new database, and if the database corresponding to the data change event is the new database, the target database is the original database.
Optionally, before obtaining the changed data corresponding to the data change event, the method further includes: controlling a first switch and a second switch so as to send a data processing request to the original database or the new database; if the data processing request is sent to the original database, storing first change data corresponding to the data processing request to the original database, and generating a first data change event corresponding to the original database; and if the data processing request is sent to the new database, storing second change data corresponding to the data processing request to the new database, and generating a second data change event corresponding to the new database.
Optionally, the storing the first change data corresponding to the data processing request to the original database includes: judging whether the original database stores low version data corresponding to the first changed data or not; if so, updating the low version data corresponding to the first changed data stored in the original database by using the first changed data, and upgrading the version information of the low version data to obtain the version information of the first changed data; if not, storing the first change data into the original database, and setting the version information of the first change data as an initial version.
Optionally, the storing the second change data corresponding to the data processing request to the new database includes: judging whether the new database stores low version data corresponding to the second change data; if so, updating the low version data corresponding to the second changed data stored in the new database by using the second changed data, and upgrading the version information of the low version data to obtain the version information of the second changed data; if not, storing the second change data into the new database, and setting the version information of the second change data.
Optionally, the setting version information of the second change data includes: judging whether the original database stores low version data corresponding to the second changed data or not; if so, upgrading the version information of the low version data to obtain the version information of the second changed data; and if not, setting the version information of the second change data as the initial version.
Optionally, the obtaining of the change data corresponding to the data change event, and synchronizing the change data from the database corresponding to the data change event to the target database according to the version information of the change data includes: acquiring first changed data by monitoring a first data change event corresponding to the original database, and then synchronizing the first changed data from the original database to the new database according to version information of the first changed data; and acquiring second changed data by monitoring a second data change event corresponding to the new database, and then synchronizing the second changed data from the new database to the original database according to the version information of the second changed data.
Optionally, the synchronizing the first changed data from the original database to the new database according to the version information of the first changed data includes: judging whether the new database stores low version data corresponding to the first changed data or not according to the version information of the first changed data; if so, updating low version data corresponding to the first change data stored in the new database by using the first change data; and if not, storing the first change data into the new database.
Optionally, the synchronizing the second changed data from the new database to the original database according to the version information of the second changed data includes: judging whether the original database stores low version data corresponding to the second changed data or not according to the version information of the second changed data; if so, updating low version data corresponding to the second change data stored in the original database by using the second change data; and if not, storing the second change data into the original database.
Optionally, the synchronizing the existing data from the original database to the new database according to the version information of the existing data includes: judging whether the new database stores low-version data corresponding to the existing data or not according to the version information of the existing data; if so, updating the low version data corresponding to the existing data stored in the new database by using the existing data; and if not, storing the existing data into the new database.
Optionally, the method further comprises: if the data synchronization result is failure, storing the data with the failure of synchronization into a message table; and performing synchronous compensation processing on the data which fails to be synchronized through a timing task, and deleting the data which fails to be synchronized from the message table if the synchronous compensation is successful.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a data migration apparatus.
The data migration device of the embodiment of the invention comprises: the acquisition module is used for receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request; the first migration module is used for synchronizing the existing data from the original database to the new database according to the version information of the existing data; the second migration module is used for acquiring change data corresponding to a data change event and synchronizing the change data from a database corresponding to the data change event to a target database according to version information of the change data; if the database corresponding to the data change event is the original database, the target database is the new database, and if the database corresponding to the data change event is the new database, the target database is the original database.
Optionally, the apparatus further comprises a control module configured to: controlling a first switch and a second switch so as to send a data processing request to the original database or the new database; if the data processing request is sent to the original database, storing first change data corresponding to the data processing request to the original database, and generating a first data change event corresponding to the original database; and if the data processing request is sent to the new database, storing second change data corresponding to the data processing request to the new database, and generating a second data change event corresponding to the new database.
Optionally, the control module is further configured to: judging whether the original database stores low version data corresponding to the first changed data or not; if so, updating the low version data corresponding to the first changed data stored in the original database by using the first changed data, and upgrading the version information of the low version data to obtain the version information of the first changed data; if not, storing the first change data into the original database, and setting the version information of the first change data as an initial version.
Optionally, the control module is further configured to: judging whether the new database stores low version data corresponding to the second change data; if so, updating the low version data corresponding to the second changed data stored in the new database by using the second changed data, and upgrading the version information of the low version data to obtain the version information of the second changed data; if not, storing the second change data into the new database, and setting the version information of the second change data.
Optionally, the control module is further configured to: judging whether the original database stores low version data corresponding to the second changed data or not; if so, upgrading the version information of the low version data to obtain the version information of the second changed data; and if not, setting the version information of the second change data as the initial version.
Optionally, the second migration module is configured to: acquiring first changed data by monitoring a first data change event corresponding to the original database, and then synchronizing the first changed data from the original database to the new database according to version information of the first changed data; and acquiring second changed data by monitoring a second data change event corresponding to the new database, and then synchronizing the second changed data from the new database to the original database according to the version information of the second changed data.
Optionally, the second migration module is configured to: judging whether the new database stores low version data corresponding to the first changed data or not according to the version information of the first changed data; if so, updating low version data corresponding to the first change data stored in the new database by using the first change data; and if not, storing the first change data into the new database.
Optionally, the second migration module is configured to: judging whether the original database stores low version data corresponding to the second changed data or not according to the version information of the second changed data; if so, updating low version data corresponding to the second change data stored in the original database by using the second change data; and if not, storing the second change data into the original database.
Optionally, the first migration module is configured to: judging whether the new database stores low-version data corresponding to the existing data or not according to the version information of the existing data; if so, updating the low version data corresponding to the existing data stored in the new database by using the existing data; and if not, storing the existing data into the new database.
Optionally, the apparatus further comprises a compensation module for: if the data synchronization result is failure, storing the data with the failure of synchronization into a message table; and performing synchronous compensation processing on the data which fails to be synchronized through a timing task, and deleting the data which fails to be synchronized from the message table if the synchronous compensation is successful.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus.
An electronic device of an embodiment of the present invention includes: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by one or more processors, the one or more processors realize the data migration method of the embodiment of the invention.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer-readable medium.
A computer-readable medium of an embodiment of the present invention has a computer program stored thereon, and the program, when executed by a processor, implements a data migration method of an embodiment of the present invention.
One embodiment of the above invention has the following advantages or benefits: the version information of the data is used for representing the updating condition of the data, so that the data of the latest version can be migrated without stopping the migration, and the condition that new data is covered can be avoided; and the data change event is monitored, the changed data can be accurately synchronized according to the version information of the changed data, and the dynamic migration of the data and the consistency of the data are ensured. In addition, the data processing request is dynamically controlled by the switch, so that the original database can be switched back in time under the condition that the database is failed to migrate, a rollback scheme is provided, and the user experience is better. In addition, the message table is adopted to record the data which fails in synchronization, and then the data which fails in synchronization can be subjected to synchronous compensation processing through the timing task, so that the consistency of the data is further ensured.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main steps of a data migration method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main processes of a data migration method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main process of synchronizing data from an original database to a new database;
FIG. 4 is a schematic diagram of the main process of synchronizing data from a new database to an original database;
FIG. 5 is a schematic diagram of the main modules of a data migration apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of main steps of a data migration method according to an embodiment of the present invention.
As shown in fig. 1, the main steps of the data migration method may include:
step S101, receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request;
step S102, synchronizing the existing data from the original database to the new database according to the version information of the existing data;
and step S103, acquiring the changed data corresponding to the data change event, and synchronizing the changed data from the database corresponding to the data change event to the target database according to the version information of the changed data.
In step S101, the data migration request refers to a request for migrating data from an original database to a new database. Therefore, the original database and the new database can be obtained through the data migration request, and the existing data stored in the original database, namely the data currently stored in the original database, can be obtained.
Fig. 2 is a schematic diagram of the main processes of the data migration method according to the embodiment of the present invention. As shown in fig. 2, different data migration means may be employed for the existing data and the changed data. Next, how to perform data synchronization on existing data and changed data, respectively, will be discussed.
In step S102, for the existing data, the existing data may be synchronized from the original database to the new database according to the version information of the existing data.
The version information may be understood as attribute information of the data, and the update condition of the data may be represented by the version information. If the version information of the data a can be set to 1, if the data a is updated subsequently, the version information of the update is set to 2. In the embodiment of the invention, the version information of the introduced data represents the updating condition of the data, so that the data of the latest version can be migrated, and the condition that new data is covered is avoided. Furthermore, the data version information is adopted to represent the update condition of the data, not the time information of the data. This is because it is considered that the general time is in the order of seconds, and there may be a case where data is updated many times within one second, and therefore if the time information is used to identify the update of data, there is a case where the latest data cannot be distinguished, and a problem occurs in that new data is overwritten.
Further, synchronizing the existing data from the original database to the new database according to the version information of the existing data may include: judging whether the new database stores low version data corresponding to the existing data or not according to the version information of the existing data; if so, updating the low version data corresponding to the existing data stored in the new database by using the existing data; and if not, storing the existing data into the new database.
Before data migration is performed, existing data may be marked, version information of the existing data is determined, and then the existing data is synchronized from the original database to the new database according to the version information of the existing data. Specifically, whether the new database stores low-version data corresponding to the existing data is judged. If the data is stored, the low version data is deleted, and the existing data is stored in a new database. And if not, directly storing the existing data into the new database. Therefore, before existing data is synchronized each time, whether the existing data is the highest version or not is judged, and if yes, the existing data is synchronized, so that coverage of new data can be prevented, and the accuracy of the data is guaranteed.
In step S103, first, the changed data corresponding to the data change event is acquired, and then, after the changed data is acquired, the changed data is synchronized from the database corresponding to the data change event to the target database according to the version information of the changed data.
And if the database corresponding to the data change event is the original database, the target database is the new database. That is, changing the data in the original database, i.e., the changed data in the original database, requires synchronizing the changed data from the original database to the new database. And if the database corresponding to the data change event is a new database, the target database is the original database. That is, changing data in the new database, i.e., changing data in the new database, requires synchronizing the changed data from the new database to the original database.
Before data migration, processing a data request by an original database, namely selecting the original database as a data source; in the data migration process, the original database and the new database can process data requests, namely the original database and/or the new database can be selected as data sources; after the data migration, the data request is processed by the new database, i.e., the new database is selected as the data source. The data change event is an event generated by changing data in the data migration process, and then the data change event is analyzed to obtain changed data, and further the changed data is synchronously processed.
As an embodiment of the present invention, before obtaining the change data corresponding to the data change event, the data migration method may further include: controlling the first switch and the second switch so as to send the data processing request to the original database or the new database; if the data processing request is sent to the original database, storing first change data corresponding to the data processing request to the original database, and generating a first data change event corresponding to the original database; and if the data processing request is sent to the new database, storing second change data corresponding to the data processing request to the new database, and generating a second data change event corresponding to the new database.
User data may update existing data at any time due to scene needs, and the update frequency of data changes more frequently with the requirements of a particular scene. In contrast, in the data migration process, the data source is dynamically selected through the switch, so that the dynamic smooth migration of the data is ensured. As shown in fig. 2, by controlling the first switch and the second switch, a percentage stream-cut of data requests may be achieved. If the personal identification code of the user is processed by using the Hash algorithm, a matching numerical value is obtained, which is equivalent to converting characters into numerical values, and then the numerical values are subjected to modular segmentation flow. For example, the personal identification code of the user is "abc", the personal identification code is "123" after the hash algorithm processing, 1000 modulo and 10% cut flow are performed, that is, the data processing request corresponding to the user whose modulo result is within 100 is sent to the new database, and the others are sent to the original database, so that the percentage cut flow can be realized. It can be seen that by controlling the first switch and the second switch, a percentage cut flow can be achieved, so that each data processing request can be sent to the original database or the new database.
It should be noted that the purpose of data migration is to replace the original database with a new database, so the cut-stream can be made gradually from small batch to full volume. For example, before data migration, data processing requests are all sent to an original database, and the original database is used as a data source; when data migration is started, 10% of data processing requests are sent to a new database, and 90% of data processing requests are sent to an original database; after the time t1 of data migration, 30% of data processing requests are sent to a new database, and 70% of data processing requests are sent to an original database; after the time t2 of data migration, 70% of data processing requests are sent to a new database, and 30% of data processing requests are sent to an original database; after the time t3 of data migration is carried out, or after existing data stored in the original database are migrated, data processing requests are all sent to the new database, and the new database serves as a data source.
In addition, if the data processing request is sent to the original database, the data in the original database is changed, and the first changed data corresponding to the data processing request can be stored in the original database to generate a first data change event corresponding to the original database. If the data processing request is sent to the new database, the data in the new database is changed, second changed data corresponding to the data processing request can be stored in the new database, and a second data change event corresponding to the new database is generated.
Further, storing the first change data corresponding to the data processing request to the original database may include: judging whether the original database stores low version data corresponding to the first change data; if so, updating the low-version data corresponding to the first changed data stored in the original database by using the first changed data, and upgrading the version information of the low-version data to obtain the version information of the first changed data; if not, storing the first change data into the original database, and setting the version information of the first change data as the initial version.
If the original database does not store the low version data corresponding to the first changed data, it indicates that the first changed data is the first-appearing data, and the first changed data can be stored in the original database, and the version information of the first changed data is set as the initial version. For example, if the first changed data is the data of the user S1, and the original database is obtained through query that the data of the user S1 is not stored, the first changed data is directly stored in the original database, and the version of the first changed data may be set to 1. On the other hand, if the original database stores the low version data corresponding to the first changed data, the stored low version data may be replaced by the first changed data, and the version information of the low version data may be updated to obtain the version information of the first changed data. For example, if the first changed data is data of the user S2, the original database is queried to store the low version data of the user S2, and the version is 2, the first changed data may be stored in the original database instead of the low version data, and the version of the first changed data is set to 3.
Similarly, storing the second change data corresponding to the data processing request to the new database includes: judging whether the new database stores low version data corresponding to the second changed data; if so, updating the low-version data corresponding to the second changed data stored in the new database by using the second changed data, and upgrading the version information of the low-version data to obtain the version information of the second changed data; if not, storing the second changed data into a new database, and setting version information of the second changed data.
If the low version data corresponding to the second changed data is stored in the new database, the stored low version data can be replaced by the second changed data, and the version information of the low version data is upgraded to obtain the version information of the second changed data. On the other hand, if the new database does not store the low version data corresponding to the second changed data, it indicates that the second changed data is the first-appearing data, and the second changed data may be stored in the new database, and the version information of the second changed data may be set as the initial version.
As an embodiment of the present invention, setting version information of the second change data may include: judging whether the original database stores low version data corresponding to the second changed data; if so, upgrading the version information of the low version data to obtain the version information of the second changed data; if not, setting the version information of the second change data as the initial version. That is, if the new database does not store the low version data corresponding to the second changed data, it is also necessary to determine whether the original database stores the low version data corresponding to the second changed data. If the original database does not store the low version data corresponding to the second changed data, the second changed data is shown to be firstly appeared, and the version information of the second changed data can be directly set as the initial version. If the original database stores the low version data corresponding to the second changed data, the version information of the low version data can be upgraded, and the obtained upgraded version information is the version information of the second changed data. It should be noted that, the version information of the low-version data is upgraded here, but the low-version data and the version information corresponding to the low-version data are still stored in the original database. And updating the stored low-version data and the version information corresponding to the low-version data only after the second change data is synchronized to the original database.
As an embodiment of the present invention, in step S103, acquiring the change data corresponding to the data change event, and synchronizing the change data from the database corresponding to the data change event to the target database according to the version information of the change data may include:
(1) the method comprises the steps of monitoring a first data change event corresponding to an original database, obtaining first change data, and synchronizing the first change data from the original database to a new database according to version information of the first change data.
As described above, if the data processing request is sent to the original database, the data in the original database is updated, and the first data change event is generated. At this time, the first data change event may be monitored, so as to obtain first change data corresponding to the first data change event, and then the first change data may be synchronized from the original database to the new database according to version information of the first change data.
Further, synchronizing the first changed data from the original database to the new database according to the version information of the first changed data may include: judging whether the new database stores low version data corresponding to the first changed data or not according to the version information of the first changed data; if so, updating the low version data corresponding to the first change data stored in the new database by using the first change data; and if not, storing the first change data into the new database.
Before the first changed data is synchronized from the original database to the new database, whether the low version data corresponding to the first changed data is stored in the new database may be determined according to the version information of the first changed data. If so, the low version data is deleted and the first change data is stored in the new database. If not, the first change data is stored directly into the new database.
(2) And acquiring second changed data by monitoring a second data change event corresponding to the new database, and then synchronizing the second changed data from the new database to the original database according to the version information of the second changed data.
As already explained above, if the data processing request is sent to the new database, the data in the new database is updated, and the second data change event is generated. At this time, the second data change event may be monitored, so as to obtain second change data corresponding to the second data change event, and then the second change data may be synchronized from the new database to the original database according to version information of the second change data.
Further, synchronizing the second changed data from the new database to the original database according to the version information of the second changed data may include: judging whether the original database stores low version data corresponding to the second changed data or not according to the version information of the second changed data; if so, updating the low version data corresponding to the second change data stored in the original database by using the second change data; and if not, storing the second changed data into the original database.
Before synchronizing the second changed data from the new database to the original database, whether the original database stores the low version data corresponding to the second changed data may be determined according to the version information of the second changed data. If so, deleting the low version data and storing the second change data in the original database. And if not, directly storing the second change data into the original database.
In addition, as an embodiment of the present invention, the data migration method may further include: if the data synchronization result is failure, storing the data with the failure of synchronization into a message table; and performing synchronous compensation processing on the data which fails to be synchronized through the timing task, and deleting the data which fails to be synchronized from the message table if the synchronous compensation is successful.
If the data synchronization fails, the data with the failed synchronization needs to be recorded, and then the synchronization compensation processing of the data is carried out through the timing task. In the synchronization compensation process, for data which fails to be synchronized, the latest data is pulled from the source library again, and then the pulled data is synchronized to the target library. If existing data is synchronized from an original database to a new database, but the synchronization fails, the latest data of the existing data needs to be pulled from the original database and then synchronized to the new database. Considering that data update is frequent, there are situations that synchronization of some data fails and the data is updated subsequently, and therefore the latest data of the data needs to be pulled for synchronization. In addition, after the data synchronization compensation is successful, the data which fails to be synchronized needs to be deleted, otherwise, the data may be subjected to the synchronization compensation again. In addition, in the embodiment of the present invention, the number of retries of the data, that is, the number of synchronization compensation processes, may also be recorded. When the number of times of synchronous compensation processing reaches the retry upper limit, alarm processing can be carried out so as to facilitate the follow-up manual intervention and repair processing.
In summary, in the data migration method according to the embodiment of the present invention, the existing data may be synchronized from the original database to the new database, the first changed data may be synchronized from the original database to the new database, and the second changed data may be synchronized from the new database to the original database. Fig. 3 is a schematic diagram showing a main process of synchronizing data from an original database to a new database, and fig. 4 is a schematic diagram showing a main process of synchronizing data from the new database to the original database.
As shown in fig. 3, if the second switch is turned off, the data processing request can be sent to the original database, and the first data change event is generated, so that it is necessary to synchronize the first changed data from the original database to the new database. In addition, as can be seen from fig. 3, existing data in the original database is synchronized from the original database to the new database. If the data fails to be synchronized from the original database to the new database, the local message table 1 is adopted to record the data which fails to be synchronized, so that the subsequent synchronization compensation processing is performed. Before the first changed data is synchronized to the new database from the original database, whether the low version data corresponding to the first changed data is stored in the new database needs to be judged, and if yes, the low version data is deleted. Similarly, before existing data is synchronized to the new database from the original database, it is necessary to determine whether the new database stores low-version data corresponding to the existing data, and if so, delete the low-version data.
As shown in fig. 4, if the first switch is turned off and the data processing request can be sent to the new database, the second data change event is generated, and therefore, the second changed data needs to be synchronized from the new database to the original database. And if the data fails to be synchronized from the new database to the original database, recording the data which fails to be synchronized by adopting the local message table 2 so as to carry out synchronous compensation processing subsequently. Before the second changed data is synchronized to the original database from the new database, whether the original database stores the low-version data corresponding to the second changed data needs to be judged, and if yes, the low-version data is deleted.
According to the data migration method provided by the embodiment of the invention, the version information of the data is used for representing the updating condition of the data, so that the data of the latest version can be migrated without stopping the migration, and the condition that new data is covered can be avoided; and the data change event is monitored, the changed data can be accurately synchronized according to the version information of the changed data, and the dynamic migration of the data and the consistency of the data are ensured. In addition, the data processing request is dynamically controlled by the switch, so that the original database can be switched back in time under the condition that the database is failed to migrate, a rollback scheme is provided, and the user experience is better. In addition, the message table is adopted to record the data which fails in synchronization, and then the data which fails in synchronization can be subjected to synchronous compensation processing through the timing task, so that the consistency of the data is further ensured.
FIG. 5 is a schematic diagram of the main modules of a data migration apparatus according to an embodiment of the present invention. As shown in fig. 5, the main modules of the data migration apparatus 500 may include: an acquisition module 501, a first migration module 502, and a second migration module 503.
The obtaining module 501 may be configured to: receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request; the first migration module 502 may be to: synchronizing the existing data from the original database to the new database according to the version information of the existing data; the second migration module 503 may be configured to: and acquiring the changed data corresponding to the data change event, and synchronizing the changed data from the database corresponding to the data change event to the target database according to the version information of the changed data. If the database corresponding to the data change event is the original database, the target database is a new database; and if the database corresponding to the data change event is a new database, the target database is the original database.
As can be seen in FIG. 5, the data migration apparatus 500 may further include a control module 504. The control module 504 may be configured to: controlling the first switch and the second switch so as to send the data processing request to the original database or the new database; if the data processing request is sent to the original database, storing first change data corresponding to the data processing request to the original database, and generating a first data change event corresponding to the original database; and if the data processing request is sent to the new database, storing second change data corresponding to the data processing request to the new database, and generating a second data change event corresponding to the new database.
As an embodiment of the present invention, the control module 504 may further be configured to: judging whether the original database stores low version data corresponding to the first change data; if so, updating the low-version data corresponding to the first changed data stored in the original database by using the first changed data, and upgrading the version information of the low-version data to obtain the version information of the first changed data; if not, storing the first change data into the original database, and setting the version information of the first change data as the initial version.
As an embodiment of the present invention, the control module 504 may further be configured to: judging whether the new database stores low version data corresponding to the second changed data; if so, updating the low-version data corresponding to the second changed data stored in the new database by using the second changed data, and upgrading the version information of the low-version data to obtain the version information of the second changed data; if not, storing the second changed data into a new database, and setting version information of the second changed data.
As an embodiment of the present invention, the control module 504 may further be configured to: judging whether the original database stores low version data corresponding to the second changed data; if so, upgrading the version information of the low version data to obtain the version information of the second changed data; if not, setting the version information of the second change data as the initial version.
As an embodiment of the present invention, the second migration module 503 may further be configured to: acquiring first changed data by monitoring a first data change event corresponding to an original database, and then synchronizing the first changed data from the original database to a new database according to version information of the first changed data; and acquiring second changed data by monitoring a second data change event corresponding to the new database, and then synchronizing the second changed data from the new database to the original database according to the version information of the second changed data.
As an embodiment of the present invention, the second migration module 503 may further be configured to: judging whether the new database stores low version data corresponding to the first changed data or not according to the version information of the first changed data; if so, updating the low version data corresponding to the first change data stored in the new database by using the first change data; and if not, storing the first change data into the new database.
As an embodiment of the present invention, the second migration module 503 may further be configured to: judging whether the original database stores low version data corresponding to the second changed data or not according to the version information of the second changed data; if so, updating the low version data corresponding to the second change data stored in the original database by using the second change data; and if not, storing the second changed data into the original database.
As an embodiment of the present invention, the first migration module 502 may further be configured to: judging whether the new database stores low version data corresponding to the existing data or not according to the version information of the existing data; if so, updating the low version data corresponding to the existing data stored in the new database by using the existing data; and if not, storing the existing data into the new database.
As shown in FIG. 5, the data migration apparatus 500 may further include a compensation module 505. The compensation module 505 is operable to: if the data synchronization result is failure, storing the data with the failure of synchronization into a message table; and performing synchronous compensation processing on the data which fails to be synchronized through the timing task, and deleting the data which fails to be synchronized from the message table if the synchronous compensation is successful.
According to the data migration device provided by the embodiment of the invention, the version information of the data is used for representing the updating condition of the data, so that the data of the latest version can be migrated without stopping the migration, and the condition that new data is covered can be avoided; and the data change event is monitored, the changed data can be accurately synchronized according to the version information of the changed data, and the dynamic migration of the data and the consistency of the data are ensured. In addition, the data processing request is dynamically controlled by the switch, so that the original database can be switched back in time under the condition that the database is failed to migrate, a rollback scheme is provided, and the user experience is better. In addition, the message table is adopted to record the data which fails in synchronization, and then the data which fails in synchronization can be subjected to synchronous compensation processing through the timing task, so that the consistency of the data is further ensured.
Fig. 6 illustrates an exemplary system architecture 600 to which the data migration method or the data migration apparatus of the embodiments of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 605 may be a server providing various services, for example, a background management server (for example only) providing support during data migration by the user using the terminal devices 601, 602, and 603; as another example, server 605 may perform data migration in accordance with embodiments of the present invention.
It should be noted that the data migration method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data migration apparatus is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes an acquisition module, a first migration module, and a second migration module. The names of the modules do not limit the modules themselves under certain conditions, for example, the obtaining module may also be described as a "module that receives a data migration request and obtains existing data stored in an original database, a new database and an original database according to the data migration request".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request; synchronizing the existing data from the original database to the new database according to the version information of the existing data; acquiring changed data corresponding to the data change event, and synchronizing the changed data from a database corresponding to the data change event to a target database according to version information of the changed data; if the database corresponding to the data change event is the original database, the target database is the new database, and if the database corresponding to the data change event is the new database, the target database is the original database.
According to the technical scheme of the embodiment of the invention, the version information of the data is used for representing the updating condition of the data, so that the data of the latest version can be migrated without stopping the migration, and the condition that new data is covered can be avoided; and the data change event is monitored, the changed data can be accurately synchronized according to the version information of the changed data, and the dynamic migration of the data and the consistency of the data are ensured. In addition, the data processing request is dynamically controlled by the switch, so that the original database can be switched back in time under the condition that the database is failed to migrate, a rollback scheme is provided, and the user experience is better. In addition, the message table is adopted to record the data which fails in synchronization, and then the data which fails in synchronization can be subjected to synchronous compensation processing through the timing task, so that the consistency of the data is further ensured.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (13)

1. A method of data migration, comprising:
receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request;
synchronizing the existing data from the original database to the new database according to the version information of the existing data;
acquiring change data corresponding to a data change event, and synchronizing the change data from a database corresponding to the data change event to a target database according to version information of the change data; wherein,
and if the database corresponding to the data change event is the original database, the target database is the new database, and if the database corresponding to the data change event is the new database, the target database is the original database.
2. The method of claim 1, wherein prior to obtaining change data corresponding to a data change event, the method further comprises:
controlling a first switch and a second switch so as to send a data processing request to the original database or the new database;
if the data processing request is sent to the original database, storing first change data corresponding to the data processing request to the original database, and generating a first data change event corresponding to the original database;
and if the data processing request is sent to the new database, storing second change data corresponding to the data processing request to the new database, and generating a second data change event corresponding to the new database.
3. The method of claim 2, wherein storing the first change data corresponding to the data processing request to the original database comprises:
judging whether the original database stores low version data corresponding to the first changed data or not;
if so, updating the low version data corresponding to the first changed data stored in the original database by using the first changed data, and upgrading the version information of the low version data to obtain the version information of the first changed data;
if not, storing the first change data into the original database, and setting the version information of the first change data as an initial version.
4. The method according to claim 2, wherein the storing the second change data corresponding to the data processing request to the new database includes:
judging whether the new database stores low version data corresponding to the second change data;
if so, updating the low version data corresponding to the second changed data stored in the new database by using the second changed data, and upgrading the version information of the low version data to obtain the version information of the second changed data;
if not, storing the second change data into the new database, and setting the version information of the second change data.
5. The method according to claim 4, wherein the setting of version information of the second change data includes:
judging whether the original database stores low version data corresponding to the second changed data or not;
if so, upgrading the version information of the low version data to obtain the version information of the second changed data;
and if not, setting the version information of the second change data as the initial version.
6. The method according to any one of claims 2 to 5, wherein the acquiring change data corresponding to a data change event, and synchronizing the change data from the database corresponding to the data change event to a target database according to version information of the change data, comprises:
acquiring first changed data by monitoring a first data change event corresponding to the original database, and then synchronizing the first changed data from the original database to the new database according to version information of the first changed data;
and acquiring second changed data by monitoring a second data change event corresponding to the new database, and then synchronizing the second changed data from the new database to the original database according to the version information of the second changed data.
7. The method of claim 6, wherein synchronizing the first change data from the original database to the new database according to version information of the first change data comprises:
judging whether the new database stores low version data corresponding to the first changed data or not according to the version information of the first changed data;
if so, updating low version data corresponding to the first change data stored in the new database by using the first change data;
and if not, storing the first change data into the new database.
8. The method of claim 6, wherein synchronizing the second change data from the new database to the original database according to version information of the second change data comprises:
judging whether the original database stores low version data corresponding to the second changed data or not according to the version information of the second changed data;
if so, updating low version data corresponding to the second change data stored in the original database by using the second change data;
and if not, storing the second change data into the original database.
9. The method of claim 1, wherein synchronizing the existing data from the original database to the new database according to the version information of the existing data comprises:
judging whether the new database stores low-version data corresponding to the existing data or not according to the version information of the existing data;
if so, updating the low version data corresponding to the existing data stored in the new database by using the existing data;
and if not, storing the existing data into the new database.
10. The method of claim 1, further comprising:
if the data synchronization result is failure, storing the data with the failure of synchronization into a message table;
and performing synchronous compensation processing on the data which fails to be synchronized through a timing task, and deleting the data which fails to be synchronized from the message table if the synchronous compensation is successful.
11. A data migration apparatus, comprising:
the acquisition module is used for receiving a data migration request, and acquiring an original database, a new database and existing data stored in the original database according to the data migration request;
the first migration module is used for synchronizing the existing data from the original database to the new database according to the version information of the existing data;
the second migration module is used for acquiring change data corresponding to a data change event and synchronizing the change data from a database corresponding to the data change event to a target database according to version information of the change data; wherein,
and if the database corresponding to the data change event is the original database, the target database is the new database, and if the database corresponding to the data change event is the new database, the target database is the original database.
12. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-10.
13. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-10.
CN202110515775.2A 2021-05-12 2021-05-12 Data migration method and device Pending CN113138975A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110515775.2A CN113138975A (en) 2021-05-12 2021-05-12 Data migration method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110515775.2A CN113138975A (en) 2021-05-12 2021-05-12 Data migration method and device

Publications (1)

Publication Number Publication Date
CN113138975A true CN113138975A (en) 2021-07-20

Family

ID=76816909

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110515775.2A Pending CN113138975A (en) 2021-05-12 2021-05-12 Data migration method and device

Country Status (1)

Country Link
CN (1) CN113138975A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113626412A (en) * 2021-08-13 2021-11-09 徐世亮 Database migration method and device, readable storage medium and computer equipment
CN114579624A (en) * 2022-02-25 2022-06-03 阿里巴巴（中国）有限公司 Data processing method, device, storage medium and computer program product

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113626412A (en) * 2021-08-13 2021-11-09 徐世亮 Database migration method and device, readable storage medium and computer equipment
CN114579624A (en) * 2022-02-25 2022-06-03 阿里巴巴（中国）有限公司 Data processing method, device, storage medium and computer program product

Similar Documents

Publication Publication Date Title
CN107656748B (en) Application publishing method and device
CN112965761B (en) Data processing method, system, electronic equipment and storage medium
CN113138975A (en) Data migration method and device
CN111782235B (en) Data upgrading and inquiring method and device
CN111290871A (en) Method and device for acquiring crash information of application program
CN111666134A (en) Method and system for scheduling distributed tasks
CN112084254A (en) Data synchronization method and system
CN107818027B (en) Method and device for switching main name node and standby name node and distributed system
CN113742376A (en) Data synchronization method, first server and data synchronization system
CN111208974A (en) Component management method and device
CN112751693B (en) Data processing method and device of distributed storage system and electronic equipment
JP2023169287A (en) Memory efficient software patching for updating applications on computing devices
CN116737662A (en) Method, device, electronic equipment and storage medium for processing business data
CN115167874B (en) Automatic driving software mirror image deployment method and device, electronic equipment and readable medium
CN111767495A (en) Method and system for synthesizing webpage
CN113760469A (en) Distributed computing method and device
CN110851192B (en) Method and device for responding to degraded switch configuration
CN114281786B (en) File processing method, device, electronic equipment and storage medium
CN113805858A (en) Method and device for continuously deploying software developed by scripting language
CN112732728A (en) Data synchronization method and system
CN113127443A (en) Method and device for updating cache data
CN111176679A (en) Application program updating method and device
CN111142921A (en) Software upgrading method and device
CN113094211B (en) Backup data processing method and device
CN114979187B (en) Data processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination