CN114116676A - Data migration method and device, electronic equipment and computer readable storage medium - Google Patents
Data migration method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN114116676A CN114116676A CN202111481239.1A CN202111481239A CN114116676A CN 114116676 A CN114116676 A CN 114116676A CN 202111481239 A CN202111481239 A CN 202111481239A CN 114116676 A CN114116676 A CN 114116676A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- source
- database
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 170
- 230000005012 migration Effects 0.000 title claims abstract description 170
- 238000000034 method Methods 0.000 title claims abstract description 100
- 230000008569 process Effects 0.000 claims abstract description 57
- 238000012545 processing Methods 0.000 claims description 30
- 230000002159 abnormal effect Effects 0.000 claims description 29
- 230000004044 response Effects 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 7
- 230000001915 proofreading effect Effects 0.000 claims description 6
- 230000001502 supplementing effect Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data migration method and device, electronic equipment and a computer readable storage medium, and relates to the field of databases. In the embodiment of the application, when a first target operation request aiming at first source data is received in the data migration process, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
Description
Technical Field
The present application relates to the field of database technologies, and in particular, to a data migration method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
With the rapid development of the internet industry, mass data is generally stored in a database of an internet service, and the mass data generally needs to be migrated between the databases and other processing operations. For example, in a data migration operation between a new system database and an old system database, since an old system cannot be well designed on a system architecture when a previous service is quickly iterated, the maintenance difficulty of a subsequent project is large, and the performance and the expansibility of the system cannot meet the requirement of supporting quick development of the service, the old system needs to be reconstructed. As an important resource in a database, data needs to be migrated from an old system to a new system while the data structure changes with the upgrade and reconstruction of the database system. However, in the related art, due to the problems of inconsistency between source data and target data and the like in the data migration operation process, the data migration operation affects normal use of the database system by a user.
Disclosure of Invention
The purpose of the present application is to solve at least one of the above technical drawbacks, in particular, the technical drawback that the data migration process affects the normal use of the database by the user.
According to an aspect of the present application, there is provided a data migration method, including:
receiving a first target operation request aiming at first source data in the process of executing the data migration operation;
executing a first data operation corresponding to the first target operation request on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation;
the source database corresponds to a first service logic, and the target database corresponds to a second service logic.
Optionally, the executing, in the target database of the data migration operation, a second data operation corresponding to the first target operation request on the first target data corresponding to the first source data includes:
determining a target position of target data corresponding to the first source data in the target database in a data table of the target database according to a preset corresponding relation;
and executing a second data operation corresponding to the first target operation request on the first target data of the target position.
Optionally, the method further includes:
receiving indication information indicating that the data migration operation is completed;
receiving a second target operation request aiming at the first target data in response to the indication information;
executing a third data operation corresponding to the second target operation request on the first target data in the target database; and executing a fourth data operation corresponding to the second target operation request on the first source data corresponding to the first target data in the source database.
Optionally, the method further includes:
after receiving indication information indicating that the data migration operation is completed, checking first processing data, which is obtained after the first data operation is executed, in the source database with second processing data, which is obtained after the second data operation is executed, in the target database;
and if the first processing data and the second processing data are consistent in proofreading, switching a data source of data read by a user to the target database.
Optionally, the method further includes:
stopping receiving a data operation request for the source data in the source database after a preset condition is met, wherein the preset condition comprises: and receiving error reporting information indicating that the operation of the target database is abnormal, wherein the error reporting information is smaller than a first preset value in a preset test period.
Optionally, after the first processed data and the second processed data are collated, the method further includes:
if abnormal data exists in the target database, supplementing the target database with the target data corresponding to the data position of the abnormal data, or re-executing the data migration operation; wherein the exception data comprises: and the data field corresponding to the data position of the abnormal data is empty data, or the data field corresponding to the data position of the abnormal data is inconsistent with the first processing data.
Optionally, after the data source for reading data by the user is switched to the target database,
and receiving indication information of abnormal operation of the target database, and switching a data source of data read by a user into the source database.
Optionally, the first data operation, the second data operation, the third data operation, and the fourth data operation respectively include at least one of the following:
adding data;
modifying the data;
and deleting the data.
According to another aspect of the present application, there is provided a data migration apparatus, including:
the receiving module is used for receiving a first target operation request aiming at first source data in the process of executing the data migration operation;
a data operation module, configured to execute, in a source database of the data migration operation, a first data operation corresponding to the first target operation request on the first source data; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation;
the source database corresponds to a first service logic, and the target database corresponds to a second service logic.
According to another aspect of the present application, there is provided an electronic device including:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: the data migration method according to the first aspect of the embodiment of the present application is executed.
For example, in a third aspect of the present application, there is provided a computing device comprising: the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data migration method as shown in the first aspect of the application.
According to yet another aspect of the present application, there is provided a computer readable storage medium, which when executed by a processor implements the data migration method of the first aspect of the present application.
For example, in a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the data migration method shown in the first aspect of the present application.
According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the various alternative implementations of the first aspect described above.
The beneficial effect that technical scheme that this application provided brought is:
in the embodiment of the application, when a first target operation request aiming at first source data is received in a data migration process, a first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments of the present application will be briefly described below.
Fig. 1 is a schematic flowchart of a data migration method according to an embodiment of the present application;
fig. 2 is a schematic flow timing diagram of a data migration method according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device for data migration according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
With the upgrade and reconfiguration of database systems, data needs to be migrated from the old system to the new system while the data structure changes. In the related art, the data migration operation process is not strictly designed, and the normal use of the database system by a user in the data migration process cannot be realized.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
The data migration method provided by the embodiment of the application can be executed by computer equipment, and the computer equipment includes but is not limited to a terminal or a server. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, middleware service, a domain name service, a security service, a CDN, a big data and artificial intelligence platform, and the like. The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.
The embodiment of the present application provides a possible implementation manner, and as shown in fig. 1, a flowchart of a data processing method is provided, where the scheme may be executed by any electronic device, and optionally may be executed at a server side or a terminal device, and for convenience of description, the method provided in the embodiment of the present application is described below with a server as an execution subject.
The method and the device for data migration can be applied to the field of databases, and when a first target operation request aiming at first source data is received in the process of data migration, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal use of the database by the user is not influenced in the data migration operation process, and the user experience is improved.
As shown in fig. 1, the method may include the steps of:
s101: in performing a data migration operation, a first target operation request for first source data is received.
The method and the device for migrating the data can be applied to the field of databases, and the data to be migrated is migrated to a target database from a source database in the data migration process; for example, taking the method applied to a heterogeneous database as an example, the original service system (hereinafter referred to as the original system) has problems of old service logic and complex and disordered service data in a long-time operation process, which results in low operation efficiency of the original system, and the like. In view of various problems of the original system, the service logic needs to be updated and a new system needs to be rebuilt, and in the replacement process of the new system and the old system, in order to ensure that the data of the two systems are consistent, data migration operation needs to be performed between the two systems.
Specifically, the data migration operation includes an operation of migrating data from a source database of the original system to a target database of the new system.
The source database corresponds to a first service logic of an original system, namely an original service logic; the target database corresponds to a second business logic of the new system, i.e., the updated new business logic. Due to the change of the service logics respectively corresponding to the source database and the target database, the source data can be reclassified according to the actual service requirement, and the reclassified source data is migrated to a data table (hereinafter referred to as a target data table) in the target database.
In the embodiment of the present application, the target data table may include a pre-established data table, and the pre-established data table may be established according to actual service requirements. Before and after data migration is performed on source data, the source position of each source data in the source data table in the source database is different from the target position in the target data table corresponding to the source data.
Alternatively, the data migration operation may be performed after the new business logic is built and the new system is tested.
Specifically, the data migration operation may be performed based on the data migration instruction; the data migration instruction may include source database information and target database information. Optionally, the source database information may include an IP (Internet Protocol Address) Address of the source database, a port number of the source database, a source information table corresponding to the source data to be migrated (the source information table may include a source field corresponding to the source data to be migrated), a flow limit value of a data volume to be migrated in a unit time period when data migration is performed, start time of data migration, and the like.
The target database information may include an IP address of the target database, a write address of source data to be migrated in the target database, a flow limit value of a data volume to be migrated in a unit time period when data migration is performed, and the like.
In the data migration process, because the source data volume is large, the continuous data migration may affect the normal operation of the source database; for example, the source database operation efficiency is reduced. Therefore, in order to reduce the influence of the data migration process on the source database, the source data may be cached first.
In addition, when the data migration operation is executed, the source data may be divided into a plurality of subtasks according to each service module to which the source data belongs, and the data migration may be executed for each subtask. For example, the source data relates to data of 3 service modules, and the source data corresponding to each service module can be regarded as a subtask.
Further, in the embodiment of the present application, in the process of executing the data migration operation, the source database is still running; in this case, the source database may still be read and written, i.e. a first target operation request for the first source data may be received by the original system.
Wherein the first source data comprises any source data in the source database. The first target operation request represents a request to perform a first data operation on first source data. The first data operation may include operations to add data, modify data, delete data, and so on.
The first target operation request may include a request issued by a user to perform operations such as adding data, modifying data, deleting data, and the like on the first source data; in addition, the first target operation request may further include a request for performing operations such as adding data, modifying data, deleting data and the like on the first source data, which is automatically issued by the system in response to a preset trigger condition.
It can be understood that, in the process of the data migration operation, because the source data in the source database is not yet completely migrated to the target database, in the process of executing the data migration operation, the new system corresponding to the target database is not yet online used, and the data read and written by the read and write operations (the read and write operations may include operations of reading data, adding data, modifying data, deleting data, and the like) on the data, which are automatically triggered by the user or the system, is still the source data in the read and write source database.
S102: executing a first data operation corresponding to the first target operation request on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation;
the source database corresponds to a first service logic, and the target database corresponds to a second service logic.
In this embodiment, the source database and the target database correspond to different data logics, that is, the source database corresponds to a first service logic, and the target database corresponds to a second service logic. The first service logic is the service logic applied to the source database, and the second service logic is the service logic applied to the target database; the first service logic and the second service logic may respectively include different function instructions, service interfaces, and data read-write operation permissions.
After receiving a first target operation request aiming at first source data, responding to the first target operation request, and executing a first data operation corresponding to the first target operation request on the first source data.
It can be understood that, in the embodiment of the present application, in the process of executing the data migration operation, the use of the original system corresponding to the source database may not be stopped; that is, in the data migration operation process, the user can still perform read-write operation on the source data in the source database. In order to ensure that the data of the source database and the data of the target database are kept consistent, so as to achieve that when a user uses a new system corresponding to the target database after the data migration operation is completed, the user does not influence normal reading and writing of the data in the target database.
Specifically, the target position of the target data corresponding to the first source data in the data table of the target database may be determined according to a preset corresponding relationship; then, a second data operation corresponding to the first target operation request is executed on the first target data of the target position.
The second data operation may specifically include operations of adding data, modifying data, deleting data, and the like. In order to ensure that the data of the source database and the target database are consistent, the second data operation is the operation corresponding to the first data operation.
In the embodiment of the application, when a first target operation request aiming at first source data is received in the data migration process, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
In an optional embodiment, the performing, in the target database of the data migration operation, a second data operation corresponding to the first target operation request on the first target data corresponding to the first source data includes:
determining a target position of target data corresponding to the first source data in the target database in a data table of the target database according to a preset corresponding relation;
and executing a second data operation corresponding to the first target operation request on the first target data of the target position.
Specifically, in the embodiment of the present application, since the data table of the source data in the source database has a situation of unreasonable design, when the target database is constructed, the data table of the target data in the target database may be reestablished. In addition, the corresponding relationship between the source data and the target data can be established according to the position of the source data in the data table of the source database and the position of the target data in the data table of the target database. For example, for a first source data a, it is in the first row and the first column in table one of the source database; after the data migration is performed, the first target data a (a may be the same data as a, or a may be data after the data processing is performed on a) corresponding to the first source data a is in the first column of the second row in table three of the target database, and so on.
Further, a target position of target data corresponding to the first source data in the target database in a data table of the target database may be determined according to a preset corresponding relationship; and executing a second data operation corresponding to the first target operation request on the first target data of the target position.
The second data operation may specifically include operations of adding data, modifying data, deleting data, and the like. It is understood that, in order to ensure that the data of the source database and the target database are consistent, the second data operation is the operation corresponding to the first data operation. Specifically, when the second data operation is performed, the following two cases can be classified:
the first condition is as follows: during the data migration operation, if data migration has been performed on the first source data before receiving the first target operation request for the first source data, the second data operation may be to update the first target data corresponding to the first source data in the target database, for example, the same data operation as the first data operation may be performed on the first target data. As an example, in an actual scenario, the first data operation is to modify the data field of the first source data from "on" to "off"; accordingly, the second data operation may be to change the data field of the first target data after the migration operation from "on" to "off".
Case two: in the data migration operation process, if data migration is not performed on the first source data before the first target operation request for the first source data is received, the second data operation may be to write, in the target database, first target data corresponding to first processing data obtained after the first data operation is performed on the first source data. As an example, in an actual scenario, the first data operation is to modify the data field of the first source data from "on" to "off"; accordingly, the second data operation may be writing the data field "off" directly at the corresponding location of the first target data in the data table of the target database.
In an optional embodiment, the method further comprises:
receiving indication information indicating that the data migration operation is completed;
receiving a second target operation request aiming at the first target data in response to the indication information;
executing a third data operation corresponding to the second target operation request on the first target data in the target database; and executing a fourth data operation corresponding to the second target operation request on the first source data corresponding to the first target data in the source database.
Specifically, after receiving indication information indicating that the data migration operation is completed, that is, after the data migration operation is completed, the authority to write the data operation may be switched to write the data operation into the target database, that is, the data write operation is completed in the target database, where the data write operation may include operations of adding data, modifying data, deleting data, and the like.
In this embodiment of the present application, after the permission of the data writing operation is switched to the target database, a second target operation request for the first target data may be received.
The second target operation request may include a request sent by a user to perform operations such as adding data, modifying data, deleting data and the like on the first target data; in addition, the second target operation request may further include a request for automatically sending, by the system, an operation of adding data, modifying data, deleting data, and the like to the first target data in response to a preset trigger condition.
It can be understood that after the data migration operation is completed, since all the source data in the source database is migrated to the target database, the new system corresponding to the target database may be used online, and the write operation to the data, which is automatically triggered by the user or the system, may be writing to a data table in the target database.
In the embodiment of the application, after the data migration operation is completed, a write operation may be performed on the target database, and at this time, since the new system corresponding to the target database is at an initial operation stage, in order to ensure that when a problem occurs in the new system, the new system can be smoothly switched back to the original system corresponding to the source database so as not to affect the user experience, when a second target operation request for the first target data is received, a third data operation corresponding to the second target operation request may be performed on the first target data in the target database; and executing a fourth data operation corresponding to the second target operation request on the first source data corresponding to the first target data in the source database.
The third data operation and the fourth data operation may specifically include operations of adding data, modifying data, deleting data, and the like. It is understood that, in order to ensure that the data of the source database and the data of the target database are consistent, the third data operation is an operation corresponding to the fourth data operation. For example, the third data operation and the fourth data operation may be the same data operation performed on the first target data and the first source data, respectively. As an example, in an actual scenario, the third data operation may be to modify the data field of the first target data from "on" to "off"; accordingly, the fourth data operation may be to modify the data field of the first source data from "on" to "off".
In an optional embodiment, the method further comprises:
after receiving indication information indicating that the data migration operation is completed, checking first processing data, which is obtained after the first data operation is executed, in the source database with second processing data, which is obtained after the second data operation is executed, in the target database;
and if the first processing data and the second processing data are consistent in proofreading, switching a data source of data read by a user to the target database.
After the data migration operation is completed, in order to ensure that the data in the target database is consistent with the data in the source database, the first processing data in the source database after the first data operation is performed may also be collated with the second processing data in the target database after the second data operation is performed.
The first processing data comprises data obtained after first data operation is carried out on the first source data, and the second processing data comprises data obtained after second data operation is carried out on the first target data.
When the first processing data is consistent with the second processing data in proofreading, a data source of data read by a user can be switched to the target database. It can be understood that after the data migration operation is completed, and the data in the target database is checked to be consistent with the data in the source database, the permission of the user to read the data may be switched to read the target database.
In an optional embodiment, the method further comprises:
stopping receiving a data operation request aiming at the source data in the source database after a preset condition is met; the preset conditions include: and receiving error reporting information indicating that the operation of the target database is abnormal, wherein the error reporting information is smaller than a first preset value in a preset test period.
In this embodiment, in a preset test period, for example, within 10 days, if the received error report information indicating that the target database runs abnormally is smaller than the first preset value, the data write permission to the source database may be closed, that is, the data operation request for the source data in the source database may be stopped from being received.
It can be understood that, in the embodiment of the present application, in the preset test period, when the error reporting information of the abnormal operation of the target database is smaller than the first preset value, the data writing permission to the source database may be closed, that is, the operation of the source database may be stopped.
In an optional embodiment, after the first processed data and the second processed data are collated, the method further comprises:
if abnormal data exists in the target database, supplementing the target database with the target data corresponding to the data position of the abnormal data, or re-executing the data migration operation; wherein the exception data comprises: and the data field corresponding to the data position of the abnormal data is empty data, or the data field corresponding to the data position of the abnormal data is inconsistent with the first processing data.
Specifically, in the process of checking the first processed data and the second processed data, if the second processed data in the target database is inconsistent with the first processed data in the source database, or if a data field at a corresponding position in a data table of the target database is empty, that is, the source data to be migrated is not successfully migrated, the target data corresponding to the data position of the abnormal data may be supplemented in the target database, or the data migration operation may be re-executed.
In an alternative embodiment, after the data source of the user read data is switched to the target database,
and receiving indication information of abnormal operation of the target database, and switching a data source of data read by a user into the source database.
On the basis of the above embodiment, since the second target operation request for the first target data is received in response to the indication information when the indication information indicating that the data migration operation is completed is received; executing a third data operation corresponding to the second target operation request on the first target data in the target database; and executing a fourth data operation corresponding to the second target operation request on the first source data corresponding to the first target data in the source database. That is to say, when the write operation is performed on the target database, the corresponding operation is also performed on the source database, so that the data in the target database and the data in the source database are kept consistent, and thus, when the indication information of the abnormal operation of the target database is received, the data source from which the user reads the data can be switched to the source database, thereby realizing that the use experience of the user is not affected.
By way of example, with reference to fig. 2, a data migration method according to an embodiment of the present application is described:
in the embodiment of the application, the problems of low operation efficiency and the like of the original system are caused by the old service logic and the complex and disordered service data of the original system in the long-time operation process. For example, the original system may include a reach system that can provide push messages and reminder messages to users, and the system is inefficient to operate because the reach system is more oriented to merchant users and the accumulated data of the system during long-time operation is complex and disordered.
In view of various problems of the original system, the service logic needs to be updated and a new system needs to be rebuilt, and in the replacement process of the new system and the old system, in order to ensure that the data of the two systems are consistent, data migration operation needs to be performed between the two systems.
Because data service needs to be provided to the user continuously, during the data migration operation, the user still needs to perform data read-write operation through the source database of the original system, that is, during the data migration operation, the data read-written by the user is still the source data in the read-write source database.
In the process of executing data migration operation, when a first target operation request aiming at first source data is received, on one hand, first data operation corresponding to the first target operation request is executed on the first source data; on the other hand, in the target database of the data migration operation, a second data operation corresponding to the first target operation request is executed on first target data corresponding to the first source data.
After the data migration operation is completed, a target database of a new system is used for a user to perform data reading and writing operation, and when a second target operation request aiming at the first target data is received, on one hand, a third data operation corresponding to the second target operation request is executed on the first target data; on the other hand, in the source database, a fourth data operation corresponding to the second target operation request is executed on the first source data corresponding to the first target data.
In addition, after the data migration operation is completed, the source data in the source database and the target data in the target database can be calibrated, and if the source data and the target data are calibrated consistently, the permission of the user for reading the data can be switched to the target database, that is, the data source of the user for reading the data is switched to the target database. If the source data and the target data are inconsistent, such as the case of unsuccessful migration of part of the source data, the target data can be complemented or the data migration operation of all the source data can be re-executed.
And if the data source is switched to the source database, receiving indication information of abnormal operation of the target database, and switching the data source of the data read by the user to the source database.
If the error report information of the abnormal operation of the target database is not received in the preset test period, the read-write permission of the source database can be closed, namely, the data operation request aiming at the source data in the source database is stopped from being received.
In the embodiment of the application, when a first target operation request aiming at first source data is received in the data migration process, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
An embodiment of the present application provides a data migration apparatus, and as shown in fig. 3, the data migration apparatus 30 may include: a receiving module 301, a first data operation module 302, wherein,
a receiving module 301, configured to receive a first target operation request for first source data in a process of performing a data migration operation;
a first data operation module 302, configured to execute, in the source database of the data migration operation, a first data operation corresponding to the first target operation request on the first source data; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation;
the source database corresponds to a first service logic, and the target database corresponds to a second service logic.
In an optional embodiment, the data operation module is specifically configured to determine, according to a preset correspondence, a target position of target data in the target database, where the target data corresponds to the first source data, in a data table of the target database;
and executing a second data operation corresponding to the first target operation request on the first target data of the target position.
In an optional embodiment, the apparatus further comprises:
the second data operation module is used for receiving indication information indicating that the data migration operation is completed;
receiving a second target operation request aiming at the first target data in response to the indication information;
executing a third data operation corresponding to the second target operation request on the first target data in the target database; and executing a fourth data operation corresponding to the second target operation request on the first source data corresponding to the first target data in the source database.
In an optional embodiment, the apparatus further comprises:
a proofreading module, configured to, after receiving indication information indicating that the data migration operation is completed, proofread first processing data in the source database after the first data operation is performed and second processing data in the target database after the second data operation is performed;
and if the first processing data and the second processing data are consistent in proofreading, switching a data source of data read by a user to the target database.
In an optional embodiment, the apparatus further comprises:
a closing module, configured to stop receiving a data operation request for source data in the source database after a preset condition is met, where the preset condition includes: and receiving error reporting information indicating that the operation of the target database is abnormal, wherein the error reporting information is smaller than a first preset value in a preset test period.
In an optional embodiment, the apparatus further comprises:
a supplement module, configured to, after the first processed data and the second processed data are collated, if there is abnormal data, supplement, in the target database, target data corresponding to a data location of the abnormal data, or re-execute the data migration operation; wherein the exception data comprises: and the data field corresponding to the data position of the abnormal data is empty data, or the data field corresponding to the data position of the abnormal data is inconsistent with the first processing data.
In an optional embodiment, the apparatus further comprises:
a switching module, configured to, after the data source that reads data by a user is switched to the target database,
and receiving indication information of abnormal operation of the target database, and switching a data source of data read by a user into the source database.
In an optional embodiment, the first data operation, the second data operation, the third data operation, and the fourth data operation each comprise at least one of:
adding data;
modifying the data;
and deleting the data.
The data migration apparatus of this embodiment can execute the data migration method shown in the above embodiments of this application, and the implementation principles thereof are similar and will not be described herein again.
In the embodiment of the application, when a first target operation request aiming at first source data is received in the data migration process, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
An embodiment of the present application provides an electronic device, including: a memory and a processor; at least one program stored in the memory for execution by the processor, which when executed by the processor, implements: in the embodiment of the application, when a first target operation request aiming at first source data is received in the data migration process, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
In an alternative embodiment, an electronic device is provided, as shown in fig. 4, the electronic device 4000 shown in fig. 4 comprising: a processor 4001 and a memory 4003. Processor 4001 is coupled to memory 4003, such as via bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as transmission of data and/or reception of data. In addition, the transceiver 4004 is not limited to one in practical applications, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.
The Processor 4001 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 4001 may also be a combination that performs a computational function, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
The Memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic Disc storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.
The memory 4003 is used for storing application program codes (computer programs) for executing the present scheme, and is controlled by the processor 4001 to execute. Processor 4001 is configured to execute application code stored in memory 4003 to implement what is shown in the foregoing method embodiments.
Among them, electronic devices include but are not limited to: mobile phones, notebook computers, multimedia players, desktop computers, and the like.
The present application provides a computer-readable storage medium, on which a computer program is stored, which, when running on a computer, enables the computer to execute the corresponding content in the foregoing method embodiments.
In the embodiment of the application, when a first target operation request aiming at first source data is received in the data migration process, first data operation corresponding to the first target operation request is executed on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation, so that the data of the source database and the data of the target database are kept consistent in the data migration operation process; therefore, in the data migration operation process, a user can normally read and write the source database; after the data migration operation is completed, the user can still normally read and write the target database, so that the normal reading and writing of the data of the user are not influenced in the data migration operation process, and the user experience is improved.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method of data migration, comprising:
receiving a first target operation request aiming at first source data in the process of executing the data migration operation;
executing a first data operation corresponding to the first target operation request on the first source data in a source database of the data migration operation; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation;
the source database corresponds to a first service logic, and the target database corresponds to a second service logic.
2. The method according to claim 1, wherein the performing, in the target database of the data migration operation, the second data operation corresponding to the first target operation request on the first target data corresponding to the first source data includes:
determining a target position of first target data corresponding to the first source data in a data table of the target database in the target database according to a preset corresponding relation;
and executing a second data operation corresponding to the first target operation request on the first target data of the target position.
3. The data migration method according to claim 1, wherein the method further comprises:
receiving indication information indicating that the data migration operation is completed;
receiving a second target operation request aiming at the first target data in response to the indication information;
executing a third data operation corresponding to the second target operation request on the first target data in the target database; and executing a fourth data operation corresponding to the second target operation request on the first source data corresponding to the first target data in the source database.
4. The data migration method according to claim 3, wherein the method further comprises:
after receiving indication information indicating that the data migration operation is completed, checking first processing data, which is obtained after the first data operation is executed, in the source database with second processing data, which is obtained after the second data operation is executed, in the target database;
and if the first processing data and the second processing data are consistent in proofreading, switching a data source of data read by a user to the target database.
5. The data migration method according to claim 3, wherein the method further comprises:
stopping receiving a data operation request for the source data in the source database after a preset condition is met, wherein the preset condition comprises: and receiving error reporting information indicating that the operation of the target database is abnormal, wherein the error reporting information is smaller than a first preset value in a preset test period.
6. The method of data migration according to claim 4, wherein after the first processed data is collated with the second processed data, the method further comprises:
if abnormal data exists in the target database, supplementing the target database with the target data corresponding to the data position of the abnormal data, or re-executing the data migration operation; wherein the exception data comprises: and the data field corresponding to the data position of the abnormal data is empty data, or the data field corresponding to the data position of the abnormal data is inconsistent with the first processing data.
7. The data migration method according to claim 4, wherein after switching the data source from which the user reads the data to the target database,
and receiving indication information of abnormal operation of the target database, and switching a data source of data read by a user into the source database.
8. The data migration method according to any one of claims 3 to 7, wherein the first data operation, the second data operation, the third data operation, and the fourth data operation respectively comprise at least one of:
adding data;
modifying the data;
and deleting the data.
9. A data migration apparatus, comprising:
the receiving module is used for receiving a first target operation request aiming at first source data in the process of executing the data migration operation;
a first data operation module, configured to execute, in a source database of the data migration operation, a first data operation corresponding to the first target operation request on the first source data; executing a second data operation corresponding to the first target operation request on first target data corresponding to the first source data in a target database of the data migration operation;
the source database corresponds to a first service logic, and the target database corresponds to a second service logic.
10. An electronic device, characterized in that the electronic device comprises:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: performing the data migration method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111481239.1A CN114116676A (en) | 2021-12-06 | 2021-12-06 | Data migration method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111481239.1A CN114116676A (en) | 2021-12-06 | 2021-12-06 | Data migration method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114116676A true CN114116676A (en) | 2022-03-01 |
Family
ID=80367304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111481239.1A Pending CN114116676A (en) | 2021-12-06 | 2021-12-06 | Data migration method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114116676A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116860716A (en) * | 2023-06-16 | 2023-10-10 | 北京原点数安科技有限公司 | Data migration method, device and storage medium |
-
2021
- 2021-12-06 CN CN202111481239.1A patent/CN114116676A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116860716A (en) * | 2023-06-16 | 2023-10-10 | 北京原点数安科技有限公司 | Data migration method, device and storage medium |
| CN116860716B (en) * | 2023-06-16 | 2023-12-22 | 北京原点数安科技有限公司 | Data migration method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8725684B1 (en) | Synchronizing data stores | |
| US10970311B2 (en) | Scalable snapshot isolation on non-transactional NoSQL | |
| CN108599973B (en) | Log association method, device and equipment | |
| CN108287708B (en) | Data processing method and device, server and computer readable storage medium | |
| CN108959407B (en) | Strong consistency writing method of data and terminal equipment | |
| CN107016016B (en) | Data processing method and device | |
| US10394775B2 (en) | Order constraint for transaction processing with snapshot isolation on non-transactional NoSQL servers | |
| US11061727B1 (en) | Preventing performance degradation over time in a cache | |
| CN111506580B (en) | Transaction storage method based on centralized block chain type account book | |
| CN107277022B (en) | Process marking method and device | |
| CN112930530A (en) | Client application for network application execution | |
| CN110019538B (en) | Data table switching method and device | |
| CN111414207A (en) | Page construction method, device and equipment | |
| US9378230B1 (en) | Ensuring availability of data in a set being uncorrelated over time | |
| CN114116676A (en) | Data migration method and device, electronic equipment and computer readable storage medium | |
| US12001450B2 (en) | Distributed table storage processing method, device and system | |
| CN112860720B (en) | Storage capacity updating method and device | |
| CN113553214A (en) | Idempotent checking method and device | |
| CN117435569A (en) | Dynamic capacity expansion method, device, equipment, medium and program product for cache system | |
| WO2024036829A1 (en) | Data fusion method and apparatus, and device and storage medium | |
| CN112491943A (en) | Data request method, device, storage medium and electronic equipment | |
| CN108959405B (en) | Strong consistency reading method of data and terminal equipment | |
| CN111427868B (en) | Processing method and device for operation request in database migration and electronic equipment | |
| CN113568890A (en) | Data migration method and device, electronic equipment and computer readable storage medium | |
| US12039378B2 (en) | In-band modification of event notification preferences for server events |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |