CN118132536A - Data migration method, device, equipment and storage medium - Google Patents
Data migration method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN118132536A CN118132536A CN202410073907.4A CN202410073907A CN118132536A CN 118132536 A CN118132536 A CN 118132536A CN 202410073907 A CN202410073907 A CN 202410073907A CN 118132536 A CN118132536 A CN 118132536A
- Authority
- CN
- China
- Prior art keywords
- data
- operation statement
- data change
- statement
- synchronization table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 230000005012 migration Effects 0.000 title claims abstract description 55
- 238000013508 migration Methods 0.000 title claims abstract description 55
- 230000008859 change Effects 0.000 claims abstract description 160
- 238000006243 chemical reaction Methods 0.000 claims description 28
- 238000012545 processing Methods 0.000 claims description 21
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 15
- 230000005540 biological transmission Effects 0.000 claims description 8
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 22
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data migration method, a device, equipment and a storage medium, belonging to the technical field of data migration, wherein the method comprises the following steps: acquiring at least one data operation statement from a transaction of a source centralized database; updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table; and migrating the data operation statement in the target data change synchronization table to the target distributed database. The invention does not need manual participation in the whole process, reduces the manual operation steps of data migration from the centralized database to the distributed database, improves the migration efficiency of data migration from the centralized database to the distributed database, and shortens the downtime.
Description
Technical Field
The present invention relates to the field of data migration technologies, and in particular, to a data migration method, apparatus, device, and storage medium.
Background
In the process of rapidly developing internet services, the traditional centralized database has the bottleneck of single-library capacity and processing capacity, and even if strategies such as library division, table division and the like are adopted, the requirements of expansibility and service continuity of complex services cannot be met, and data needs to be migrated.
However, in the conventional data migration method, for example, the data migration method based on the data migration tool has limited types of source databases which can be supported, for example, the manual data migration method is based on that the external service of the source database is required to be stopped before the data migration of the database is performed, and the service is started after the data migration of the database is completed, so that the manual operation steps are more and the downtime is long.
Disclosure of Invention
The invention provides a data migration method, a device, equipment and a storage medium, which are used for improving the migration efficiency of data migration from a centralized database to a distributed database and shortening the downtime.
According to an aspect of the present invention, there is provided a data migration method, the method comprising:
Acquiring at least one data operation statement from a transaction of a source centralized database;
updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table;
and migrating the data operation statement in the target data change synchronization table to the target distributed database.
According to another aspect of the present invention, there is provided a data migration apparatus, the apparatus comprising:
The data operation statement acquisition module is used for acquiring at least one data operation statement from the transaction of the source centralized database;
The target data change synchronization table determining module is used for updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table;
And the data migration module is used for migrating the data operation statement in the target data change synchronization table to the target distributed database.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data migration method of any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to perform the data migration method of any one of the embodiments of the present invention.
According to the technical scheme, at least one data operation statement is obtained from the transaction of the source centralized database; updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table; and migrating the data operation statement in the target data change synchronization table to the target distributed database. According to the technical scheme, the initial data change synchronization table is updated to obtain the target data change synchronization table for storing the final data operation statement to be migrated, and then the data operation statement in the target data change synchronization table is migrated to the target distributed database, so that manual participation is not needed in the whole process, manual operation steps for migrating data from the centralized database to the distributed database are reduced, migration efficiency of migrating data from the centralized database to the distributed database is improved, and downtime is shortened.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data migration method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data migration method according to a second embodiment of the present invention;
Fig. 3 is a schematic structural diagram of a data migration apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing a data migration method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "source," "target," "initial," "first," and "second," etc. in the description and claims of the present invention and in the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In addition, it should be noted that, in the technical solution of the present invention, the processes of collection, storage, use, processing, transmission, provision, disclosure, etc. of the related data operation sentences all conform to the regulations of the related laws and regulations, and do not violate the public order harmony.
Example 1
Fig. 1 is a flowchart of a data migration method according to an embodiment of the present invention, where the method may be performed by a data migration device, and the device may be implemented in hardware and/or software, and may be configured in an electronic device. As shown in fig. 1, the method includes:
s101, acquiring at least one data operation statement from a transaction of a source centralized database.
The source centralized database refers to a centralized database which needs to perform data migration. A transaction, also known as a database transaction, refers to a sequence of database operations that access and potentially manipulate various data items. It should be noted that, one database transaction includes at least one data operation statement. The data operation statement refers to a statement for describing what operation is performed on data in the source centralized database; alternatively, the data manipulation statement may take the form of an SQL statement.
Specifically, at least one data operation statement may be obtained from a transaction of a source centralized database by an interceptor.
S102, updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table.
The initial data change synchronization table refers to a blank data table without stored data; optionally, the initial data change synchronization table may include the following fields: table name, whether to process, data operation statement, processed data operation statement, and data writing state. Wherein, the table names refer to the table names involved in the data operation statement. It should be noted that, in the process of migrating data from the source centralized database to the target distributed database, only the table name of one data table is referred to by default one data operation statement. The data writing state is a state in which the data operation statement is written into the target distributed database. The target distributed database refers to a distributed database that receives the migrated data in the source centralized database.
Specifically, at least one data operation statement can be written into the initial data change synchronization table in sequence according to the sequence generated by at least one data operation statement, so as to obtain an intermediate data change synchronization table; and then, according to the table structure difference between the source centralized database and the target distributed database, carrying out data processing on the data operation statement to be processed in the intermediate data change synchronization table to obtain a target data change synchronization table. The intermediate data change synchronization table refers to an initial data change synchronization table in which data operation statements are written. The data operation statement to be processed refers to a data operation statement which needs to be processed in the intermediate data change synchronization table. The target data change synchronization table refers to a data table for storing the final data operation statement to be migrated.
S103, migrating the data operation statement in the target data change synchronization table to the target distributed database.
The target data change synchronization table comprises unprocessed data operation sentences and processed data operation sentences.
Specifically, for each data operation statement in the target data change synchronization table, reading a value of a processing field corresponding to the data operation statement; if the value of the processing field is not, migrating the data operation statement to a target distributed database; if the value of the processing field is yes, migrating the processed data operation statement corresponding to the data operation statement to the target distributed database.
Optionally, after migrating the data operation statement in the target data change synchronization table to the target distributed database, updating the data writing state of the data operation statement in the target data change synchronization table; the data writing state comprises an unwritten state, a written state and a writing failure state.
Specifically, for each data operation statement in the target data change synchronization table, after the data operation statement is successfully migrated to the target distributed database, the data writing state of the data operation statement in the target data change synchronization table is updated from the unwritten state to the written state, so that the data operation statement in the target data change synchronization table is prevented from being repeatedly written into the target distributed database, and the data operation statement in the target data change synchronization table is prevented from being missed to be written.
According to the technical scheme, at least one data operation statement is obtained from the transaction of the source centralized database; updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table; and migrating the data operation statement in the target data change synchronization table to the target distributed database. According to the technical scheme, the initial data change synchronization table is updated to obtain the target data change synchronization table for storing the final data operation statement to be migrated, and then the data operation statement in the target data change synchronization table is migrated to the target distributed database, so that manual participation is not needed in the whole process, manual operation steps for migrating data from the centralized database to the distributed database are reduced, migration efficiency of migrating data from the centralized database to the distributed database is improved, and downtime is shortened.
On the basis of the above embodiment, as an alternative manner of the embodiment of the present invention, a data operation statement in which the data writing state is the writing failure state in the target data change synchronization table may be recorded in the data exception table, and the writing failure reason of the data operation statement in which each data writing state is the writing failure state is recorded in the data exception table, and the data exception table is sent to the relevant staff, so as to help the relevant staff to process the data operation statement in the data exception table in time, thereby improving the accuracy of the data in the data migration process. The data exception table is used for storing data operation sentences which are abnormal in the data migration process and recording the reasons of the abnormality of the data operation sentences.
Example two
Fig. 2 is a flowchart of a data migration method according to a second embodiment of the present invention, where an alternative implementation manner is provided by further optimizing "update an initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table" based on the above embodiment. In the embodiments of the present invention, parts not described in detail may be referred to for related expressions of other embodiments. As shown in fig. 2, the method includes:
S201, at least one data operation statement is obtained from the transaction of the source centralized database.
S202, sequentially storing at least one data operation statement into the data change stream water meter according to the generation sequence of the at least one data operation statement.
The data change flow water meter is used for recording data related to data operation sentences in the source centralized database; optionally, the data change stream meter may include the following fields: first data write time, table name, data operation statement and statement transmission state. The first data writing time is the time of writing data operation statement into the data change flow water meter. The table name refers to the table name referred to in the data operation statement. It should be noted that, in the process of migrating data from the source centralized database to the target distributed database, only the table name of one data table is referred to by default one data operation statement. Statement send status is the status of the data manipulation statement sent from the data change flow table to the message queue.
It can be understood that, according to the generating sequence of at least one data operation statement, at least one data operation statement is sequentially stored in the data change flow table, so that the storage sequence of the data operation statement in the data change flow table can be ensured to be the same as the data operation statement sequence of the database transaction record in the source centralized database.
S203, sending the data operation statement in the data change stream water meter to a message queue.
Specifically, according to the first data writing time in the data change flow water meter, the statement to be sent can be read from the data change flow water meter; and transmitting the statement to be transmitted to the message queue according to the table names in the statement to be transmitted based on the mapping relation between the table names and the message subject. The statement to be sent is a data operation statement to be sent to the message queue in the data change flow table. It should be noted that one data table corresponds to one message topic in the message queue, and different data tables correspond to different message topics in the message queue.
If there are 5 data operation sentences in the data change flow water meter, the sentences to be sent are read from the data change flow water meter according to the sequence of the first data writing time of the 5 data operation sentences in the data change flow water meter until all the 5 data operation sentences are read, and the sentences to be sent are stopped from being read from the data change flow water meter. Then, for each statement to be sent, based on the mapping relation between the table names and the message topics, taking the table names in the statement to be sent as indexes, and acquiring the target message topics corresponding to the statement to be sent from a preset topic mapping table; and sending the statement to be sent to the destination message subject of the message queue. The topic mapping table refers to a data table for having a mapping relationship between a table name of the data table and a topic of a message in the message queue. The target message theme refers to a message type to which the statement to be sent belongs and is used for reflecting the approximate storage position of the statement to be sent in the message queue.
Optionally, after the data operation statement in the data change flow table is sent to the message queue, the statement sending state of the data operation statement in the data change flow table is updated; wherein, the statement transmission state includes an unsent state, a transmitted state, and a transmission failure state. In the initial case, the statement sending state of each data operation statement in the data change stream water meter is the unsent state.
Specifically, for each data operation statement in the data change flow table, if the data operation statement is successfully sent to the message queue, the statement sending state of the data operation statement is updated from the unsent state to the sent state, so that repeated sending or omission of the data operation statement in the data change flow table is avoided, and each data operation statement in the data change flow table is ensured to be sent to the message queue.
Optionally, the data operation statement with the statement sending state in the data change stream water meter being the sending failure state can be recorded in the data exception table, the sending failure reason of the data operation statement with each statement sending state being the sending failure state is recorded in the data exception table, and the data exception table is sent to related staff, so that the related staff can be helped to process the data operation statement in the data exception table in time, and the accuracy of data in the data migration process is improved. The data exception table is used for storing data operation sentences which are abnormal in the data migration process and recording the reasons of the abnormality of the data operation sentences.
S204, storing the data operation statement in the message queue into the initial data change synchronization table to obtain an intermediate data change synchronization table.
The intermediate data change synchronization table refers to an initial data change synchronization table in which data operation statements are written. The initial data change synchronization table refers to a blank data table in which data is not stored; optionally, the initial data change synchronization table may include the following fields: table name, whether to process, data operation statement, processed data operation statement, and data writing state. Wherein, the table names refer to the table names involved in the data operation statement. It should be noted that, in the process of migrating data from the source centralized database to the target distributed database, only the table name of one data table is referred to by default one data operation statement. The data writing state is a state in which the data operation statement is written into the target distributed database. It should be noted that, the data writing state of each data operation statement in the intermediate data change synchronization table is an unwritten state, the value of the processed field corresponding to each data operation statement is a null value, and the value of the processed data operation statement field corresponding to each data operation statement is a null value.
Specifically, according to the sequence of message topics in the message queue, the data operation sentences in the message queue can be sequentially stored in the initial data change synchronization table to obtain an intermediate data change synchronization table.
If there are 3 message topics, namely, message topic 1, message topic 2 and message topic 3 in the message queue, the data operation sentences in message topic 1, message topic 2 and message topic 3 are sequentially stored in the initial data change synchronization table from small to large according to the size sequence of the numbers of the 3 message topics, so as to obtain an intermediate data change synchronization table, and ensure that each data operation sentence in the message queue is stored in the initial data change synchronization table.
Optionally, for each data operation statement in the message queue, after the data operation statement is stored in the initial data change synchronization table, a message consumption offset corresponding to the data operation statement is fed back to the message queue, so as to ensure that each data operation statement in the message queue can be processed. The message consumption offset refers to the position of the consumer consuming the message in the message queue.
S205, according to the data conversion mapping table, data conversion processing is sequentially carried out on the data operation sentences in the intermediate data change synchronization table, and the target data change synchronization table is obtained.
The data conversion mapping table is used for storing data conversion rules; the data conversion rule is determined according to the difference of the table structures in the source centralized database and the target distributed database. It can be understood that the data conversion rule is flexibly customized according to the difference of the table structures of the source centralized database and the target distributed database, so that the problem of incompatibility of heterogeneous tables in the data migration process is solved.
Specifically, according to the second data writing time in the intermediate data change synchronization table, the operation statement to be updated can be read from the intermediate data change synchronization table; determining a target data conversion rule corresponding to the operation statement to be updated from the data conversion mapping table according to the table name of the operation statement to be updated; performing data conversion processing on the operation statement to be updated according to the target data conversion rule to obtain an update operation statement of the operation statement to be updated; and updating the intermediate data change synchronization table according to the update operation statement to obtain the target data change synchronization table.
The second data writing time refers to the time when the data operation statement in the message queue is written into the initial data change synchronization table. The operation statement to be updated refers to a data operation statement which needs to be subjected to data conversion processing in the intermediate data change synchronization table. The target data conversion rule refers to a data conversion rule corresponding to the operation statement to be updated. The update operation statement is a data operation statement obtained by performing data conversion processing on the operation statement to be updated.
Optionally, updating the intermediate data change synchronization table according to the update operation statement to obtain the target data change synchronization table may be: and updating the value of the processed data operation statement field corresponding to the operation statement to be updated into the operation statement to be updated of each operation statement to be updated in the intermediate data change synchronization table, and updating the value of the processed field corresponding to the operation statement to be updated into yes or not, so that the target data change synchronization table is obtained.
According to the technical scheme provided by the embodiment of the invention, the specific method for determining the target data change synchronization table is provided, the data migration from the source centralized database to the target distributed database is completed by means of the target data change synchronization table, manual participation is not needed in the whole process, manual operation steps for migrating data from the centralized database to the distributed database are reduced, migration efficiency of migrating data from the centralized database to the distributed database is improved, and downtime is shortened.
Example III
Fig. 3 is a schematic structural diagram of a data migration apparatus according to a third embodiment of the present invention, where the embodiment is applicable to a case of migrating data in a centralized database to a distributed database, and the apparatus may be implemented in a form of hardware and/or software and may be configured in an electronic device. As shown in fig. 3, the apparatus includes:
a data operation statement obtaining module 301, configured to obtain at least one data operation statement from a transaction in a source centralized database;
The target data change synchronization table determining module 302 is configured to update the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table;
the data migration module 303 is configured to migrate the data operation statement in the target data change synchronization table to the target distributed database.
According to the technical scheme, at least one data operation statement is obtained from the transaction of the source centralized database; updating the initial data change synchronization table according to at least one data operation statement to obtain a target data change synchronization table; and migrating the data operation statement in the target data change synchronization table to the target distributed database. According to the technical scheme, the initial data change synchronization table is updated to obtain the target data change synchronization table for storing the final data operation statement to be migrated, and then the data operation statement in the target data change synchronization table is migrated to the target distributed database, so that manual participation is not needed in the whole process, manual operation steps for migrating data from the centralized database to the distributed database are reduced, migration efficiency of migrating data from the centralized database to the distributed database is improved, and downtime is shortened.
Optionally, the target data change synchronization table determining module 302 includes:
the data operation statement storage unit is used for sequentially storing at least one data operation statement into the data change stream water meter according to the generation sequence of the at least one data operation statement;
the data operation statement sending unit is used for sending the data operation statement in the data change flow water meter to the message queue;
The intermediate data change synchronization table determining unit is used for storing the data operation statement in the message queue into the initial data change synchronization table to obtain an intermediate data change synchronization table;
and the target data change synchronization table determining unit is used for sequentially carrying out data conversion processing on the data operation sentences in the intermediate data change synchronization table according to the data conversion mapping table to obtain a target data change synchronization table.
Optionally, the data operation statement sending unit is specifically configured to:
Reading the statement to be sent from the data change flow water meter according to the first data writing time in the data change flow water meter;
and transmitting the statement to be transmitted to the message queue according to the table names in the statement to be transmitted based on the mapping relation between the table names and the message subject.
Optionally, the intermediate data change synchronization table determining unit is specifically configured to:
and according to the sequence of the message subjects in the message queue, sequentially storing the data operation sentences in the message queue into the initial data change synchronization table to obtain an intermediate data change synchronization table.
Optionally, the target data change synchronization table determining unit is specifically configured to:
reading an operation statement to be updated from the intermediate data change synchronization table according to the second data writing time in the intermediate data change synchronization table;
determining a target data conversion rule corresponding to the operation statement to be updated from the data conversion mapping table according to the table name of the operation statement to be updated;
Performing data conversion processing on the operation statement to be updated according to the target data conversion rule to obtain an update operation statement of the operation statement to be updated;
and updating the intermediate data change synchronization table according to the update operation statement to obtain the target data change synchronization table.
Optionally, the apparatus further comprises:
The statement sending state updating module is used for updating the statement sending state of the data operation statement in the data change flow table after the data operation statement in the data change flow table is sent to the message queue; wherein, the statement transmission state includes an unsent state, a transmitted state, and a transmission failure state.
Optionally, the apparatus further comprises:
The data writing state updating module is used for updating the data writing state of the data operation statement in the target data change synchronous table after the data operation statement in the target data change synchronous table is migrated to the target distributed database; the data writing state comprises an unwritten state, a written state and a writing failure state.
The data migration device provided by the embodiment of the invention can execute the data migration method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the data migration methods.
Example IV
Fig. 4 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM12 and the RAM13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the data migration method.
In some embodiments, the data migration method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM12 and/or the communication unit 19. When the computer program is loaded into RAM13 and executed by processor 11, one or more steps of the data migration method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data migration method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1.A method of data migration, comprising:
Acquiring at least one data operation statement from a transaction of a source centralized database;
updating the initial data change synchronization table according to the at least one data operation statement to obtain a target data change synchronization table;
and migrating the data operation statement in the target data change synchronization table to a target distributed database.
2. The method of claim 1, wherein updating the initial data change synchronization table according to the at least one data manipulation statement to obtain the target data change synchronization table comprises:
Sequentially storing the at least one data operation statement into a data change flow water meter according to the generation sequence of the at least one data operation statement;
Transmitting the data operation statement in the data change flow water meter to a message queue;
storing the data operation statement in the message queue into an initial data change synchronization table to obtain an intermediate data change synchronization table;
And according to the data conversion mapping table, sequentially performing data conversion processing on the data operation sentences in the intermediate data change synchronization table to obtain a target data change synchronization table.
3. The method of claim 2, wherein said sending the data operation statement in the data change stream meter to a message queue comprises:
Reading a statement to be sent from the data change flow water meter according to the first data writing time in the data change flow water meter;
And transmitting the statement to be transmitted to a message queue according to the table names in the statement to be transmitted based on the mapping relation between the table names and the message subject.
4. The method of claim 2, wherein storing the data operation statement in the message queue in the initial data change synchronization table results in an intermediate data change synchronization table, comprising:
And according to the sequence of the message subjects in the message queue, sequentially storing the data operation sentences in the message queue into an initial data change synchronization table to obtain an intermediate data change synchronization table.
5. The method according to claim 2, wherein the sequentially performing data conversion processing on the data operation statements in the intermediate data change synchronization table according to the data conversion mapping table to obtain a target data change synchronization table includes:
reading an operation statement to be updated from the intermediate data change synchronization table according to the second data writing time in the intermediate data change synchronization table;
determining a target data conversion rule corresponding to the operation statement to be updated from a data conversion mapping table according to the table name of the operation statement to be updated;
performing data conversion processing on the operation statement to be updated according to the target data conversion rule to obtain an update operation statement of the operation statement to be updated;
And updating the intermediate data change synchronization table according to the update operation statement to obtain a target data change synchronization table.
6. The method of claim 2, further comprising, after said sending the data operation statement in the data change stream water table to a message queue:
updating the statement sending state of the data operation statement in the data change flow table; wherein the statement transmission state includes an unsent state, a transmitted state, and a transmission failure state.
7. The method of claim 1, further comprising, after migrating the data operation statement in the target data change synchronization table to a target distributed database:
updating the data writing state of the data operation statement in the target data change synchronous table; the data writing state comprises an unwritten state, a written state and a writing failure state.
8. A data migration apparatus, comprising:
The data operation statement acquisition module is used for acquiring at least one data operation statement from the transaction of the source centralized database;
The target data change synchronization table determining module is used for updating the initial data change synchronization table according to the at least one data operation statement to obtain a target data change synchronization table;
And the data migration module is used for migrating the data operation statement in the target data change synchronization table to a target distributed database.
9. An electronic device, the electronic device comprising:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data migration method of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the data migration method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410073907.4A CN118132536A (en) | 2024-01-18 | 2024-01-18 | Data migration method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410073907.4A CN118132536A (en) | 2024-01-18 | 2024-01-18 | Data migration method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118132536A true CN118132536A (en) | 2024-06-04 |
Family
ID=91241315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410073907.4A Pending CN118132536A (en) | 2024-01-18 | 2024-01-18 | Data migration method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118132536A (en) |
-
2024
- 2024-01-18 CN CN202410073907.4A patent/CN118132536A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113392974B (en) | Model training method, device, electronic equipment and storage medium | |
| CN113193947B (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| EP4145298A1 (en) | Method and apparatus for snapshotting metadata | |
| CN116307672A (en) | Fault diagnosis method, device, electronic equipment and medium | |
| CN115186738B (en) | Model training method, device and storage medium | |
| CN116185389A (en) | Code generation method and device, electronic equipment and medium | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN118132536A (en) | Data migration method, device, equipment and storage medium | |
| CN115167855A (en) | Front-end page generation method, device and equipment applied to matching transaction system | |
| CN114896322A (en) | JSON (Java Server object notation) -based configuration data conversion method, device, equipment and medium | |
| CN118503298B (en) | Data processing method, device, equipment and storage medium based on double caches | |
| CN118012897A (en) | Heterogeneous database grammar conversion method, device, equipment and storage medium | |
| CN117270862A (en) | Software pluggable method, device, equipment and medium based on dynamic compiling | |
| CN117331994A (en) | Real-time processing method and device for data, electronic equipment and storage medium | |
| CN115983222A (en) | EasyExcel-based file data reading method, device, equipment and medium | |
| CN114706578A (en) | Data processing method, device, equipment and medium | |
| CN117520403A (en) | Data query method based on synchronous pipeline Gremlin language | |
| CN115344646A (en) | Distributed transaction compensation method, device, equipment and medium | |
| CN118467631A (en) | Data synchronization method, device, equipment and storage medium | |
| CN115687529A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN116302371A (en) | Transaction visibility judging method, device, equipment and storage medium | |
| CN118132533A (en) | Data processing method, device, equipment and storage medium | |
| CN115454977A (en) | Data migration method, device, equipment and storage medium | |
| CN114638935A (en) | Method and device for generating dimension monitoring task and monitoring data quality | |
| CN117234509A (en) | Routing management method, device and equipment of front-end page and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |