CN116561102A

CN116561102A - Data bidirectional migration method, device, equipment, medium and program product

Info

Publication number: CN116561102A
Application number: CN202310540570.9A
Authority: CN
Inventors: 容达锋; 潘洪添; 周烨; 何思阳
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-05-15
Filing date: 2023-05-15
Publication date: 2023-08-08

Abstract

The disclosure provides a data bidirectional migration method, which can be applied to the technical field of information security. The data bidirectional migration method comprises the following steps: monitoring whether the log information in the source database changes or not; under the condition that the log information changes, obtaining change data under different data tables in the source end database, wherein the change data comprises a field of a flag bit; changing the field of the flag bit in the change data; converting the change data into change sentences, wherein the change sentences are executable in at least a target-end database; and executing the migration operation with the target-side database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value. The present disclosure also provides a data bi-directional migration apparatus, device, storage medium, and program product.

Description

Data bidirectional migration method, device, equipment, medium and program product

Technical Field

The present disclosure relates to the field of information security technology, and more particularly, to a data bidirectional migration method, apparatus, device, medium, and program product.

Background

In the field of data migration, the migration is mainly unidirectional, i.e. the migration from a source database is immediately switched to a target database. However, the target end database has a functional problem, or the target end database builds a cluster to have batch downtime, so that incremental data cannot be backed up or the source end database cannot be cut back.

In addition, when the target database is abnormal or the migration tool is in a position error in the migration process, data loss or repeated migration is easy to occur, and the synchronism of the data in the two databases is difficult to ensure.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a data bi-directional migration method, apparatus, device, medium, and program product that improve migration security and that are capable of being performed in a non-shutdown state.

According to a first aspect of the present disclosure, there is provided a data bidirectional migration method, including: monitoring whether the log information in the source database changes or not; under the condition that the log information changes, obtaining change data under different data tables in the source end database, wherein the change data comprises a field of a flag bit; changing the field of the flag bit in the change data; converting the change data into change sentences, wherein the change sentences are executable in at least a target-end database; and executing the migration operation with the target-side database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value.

According to an embodiment of the present disclosure, the obtaining the change data under different data tables in the source database includes: acquiring a data table related to the change data; creating a theme, a segmentation channel and a consumer model based on a message middleware respectively, wherein the theme corresponds to the data table one by one, the consumer model corresponds to the data table one by one, and the theme and the segmentation channel are in one-to-many relation; storing the change data into the theme according to the one-to-one correspondence between the theme and the data table; and pulling the change data by lot based on the consumer model.

According to an embodiment of the disclosure, the batch includes a first batch, and after the migration operation with the target database is performed based on the change statement, the method further includes: based on the returned migration result of the partition channel, checking whether a plurality of topics are migrated successfully in the first batch, wherein the topics comprise a first topic, and if any one of the migration results in the first topic is failure, judging that the migration of the first topic is failure; under the condition that the first theme is successfully migrated, recording a first offset of a last migration result of the first theme, and submitting the first offset after adding one process; and under the condition that the migration of the first theme fails, recording a second offset of a migration result of the migration failure in the partition channel, and submitting the second offset.

According to an embodiment of the present disclosure, the partition path includes at least a first path, and the checking whether migration of a plurality of topics in the first batch is successful based on a returned migration result of the partition path includes: traversing a plurality of migration results for the first channel, and judging whether the migration results fail or not; and under the condition that the migration results are judged to fail by traversing the plurality of migration results, judging that the migration of the current theme where the migration results are located fails, stopping traversing the plurality of partition channels in the current theme and entering the next theme.

According to an embodiment of the present disclosure, the change statement is executable in the source database, and after the converting the change data into the change statement, the method further includes: and backing up the change statement to the source database.

According to an embodiment of the disclosure, the source database includes an Oracle database and a MySQL database, the target database includes an Oracle database and a MySQL database, and the obtaining the change data under different data tables in the source database under the condition that the log information changes includes: analyzing the materialized view log for the Oracle database to obtain a binary code of a change object; inquiring the binary code of the change object, and positioning a main key field, a common field and a value of the change data; packaging and encapsulating the primary key field, the common field and the value to obtain the change data; analyzing a log file for the MySQL database to obtain a change time node time stamp; querying an SQL statement based on the time node timestamp; and packaging the SQL statement to obtain the change data.

In a second aspect of the present disclosure, there is provided a data bidirectional migration apparatus, including: the log monitoring module is used for monitoring whether the log information in the source end database changes or not; the change data acquisition module is used for acquiring change data under different data tables in the source database under the condition that the log information changes, wherein the change data comprises a field of a flag bit; the flag bit changing module is used for changing the field of the flag bit in the change data; the statement conversion module is used for converting the change data into change statements, wherein the change statements are executable in at least a target-end database; and the migration module is used for executing the migration operation with the target-end database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value.

According to an embodiment of the disclosure, the change data obtaining module is configured to obtain a data table related to the change data; creating a theme, a segmentation channel and a consumer model based on a message middleware respectively, wherein the theme corresponds to the data table one by one, the consumer model corresponds to the data table one by one, and the theme and the segmentation channel are in one-to-many relation; storing the change data into the theme according to the one-to-one correspondence between the theme and the data table; and pulling the change data by lot based on the consumer model.

According to an embodiment of the disclosure, the batch includes a first batch, and the apparatus further includes a result returning module, configured to check whether migration of a plurality of topics in the first batch is successful based on a returned migration result of the partition channel, where the plurality of topics include a first topic, and if any one of the migration results in the first topic is failure, determine that migration of the first topic fails; under the condition that the first theme is successfully migrated, recording a first offset of a last migration result of the first theme, and submitting the first offset after adding one process; and under the condition that the migration of the first theme fails, recording a second offset of a migration result of the migration failure in the partition channel, and submitting the second offset.

According to the embodiment of the disclosure, the partition channel at least comprises a first channel, and the result returning module is used for traversing a plurality of migration results for the first channel and judging whether the migration results fail or not; and under the condition that the migration results are judged to fail by traversing the plurality of migration results, judging that the migration of the current theme where the migration results are located fails, stopping traversing the plurality of partition channels in the current theme and entering the next theme.

The embodiment of the disclosure further comprises a backup module, configured to backup the change statement to the source database.

According to the embodiment of the disclosure, the source database comprises an Oracle database and a MySQL database, the target database comprises an Oracle database and a MySQL database, and the change data acquisition module is used for analyzing a materialized view log for the Oracle database to obtain a change object binary code; inquiring the binary code of the change object, and positioning a main key field, a common field and a value of the change data; packaging and encapsulating the primary key field, the common field and the value to obtain the change data; analyzing a log file for the MySQL database to obtain a change time node time stamp; querying an SQL statement based on the time node timestamp; and packaging the SQL statement to obtain the change data.

In a third aspect of the present disclosure, there is provided an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data bi-directional migration method described above.

In a fourth aspect of the present disclosure, there is also provided a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the data bi-directional migration method described above.

In a fifth aspect of the present disclosure, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the data bi-directional migration method described above.

In the embodiments of the present disclosure, the following advantageous effects may be achieved: 1. by identifying the data source of the changed data, bidirectional synchronous migration is realized between a source end database and a target end database generating service change, and the instantaneity of the bidirectional migration of the data is ensured; 2. by setting the flag bit, the state of data change is determined, effective migration of the data is ensured, and the situation that the same changed data is circularly and bidirectionally repeatedly migrated in different databases under the bidirectional migration scene is avoided.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a data bi-directional migration method according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data bi-directional migration method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of altered data acquisition in accordance with an embodiment of the disclosure;

FIG. 4 schematically illustrates a flow chart of a data bi-directional migration method according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a migration result determination method according to an embodiment of the present disclosure;

FIG. 6 schematically illustrates a full flow diagram of a data bi-directional migration method according to an embodiment of the present disclosure;

FIG. 7 schematically illustrates a block diagram of a data bi-directional migration apparatus according to an embodiment of the present disclosure; and

fig. 8 schematically illustrates a block diagram of an electronic device adapted to implement a data bi-directional migration method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Before the embodiments of the present disclosure are disclosed in detail, key technical terms related to the embodiments of the present disclosure are described one by one, as follows:

Batch: the method is used for improving the efficiency of the message middleware, and the messages are written into the message middleware in batches.

Subject matter: messages are classified by topic, which may be divided into several split channels.

Offset amount: the index representing the message acquired by the message middleware at a timing is a value that is specifically shifted with respect to the previous index position.

Because one end database is changed, single incremental data is repeatedly written into the source end database and the target end database to form dead-loop change, and the incremental data is repeatedly written into the double-end database. And in addition, the DML condition of the increment data repeatedly performed after restarting is possibly caused by downtime of the source end database or downtime of the target end database or unexpected stop of the migration program in the data migration process. When the situation occurs in the scene of the migration of the incremental data, the incremental data to be migrated needs to be preprocessed, the execution position of the error data is subjected to fault positioning, the error data is discarded, and the re-migration processing is performed, so that the situation that the database data of the target end and the database data of the source end are not synchronized is prevented from occurring due to the fact that the migration data is recorded in error.

In order to solve the technical problems in the prior art, an embodiment of the present disclosure provides a data bidirectional migration method, which monitors whether log information in a source database changes; under the condition that the log information changes, obtaining change data under different data tables in the source end database, wherein the change data comprises a field of a flag bit; changing the field of the flag bit in the change data; converting the change data into change sentences, wherein the change sentences are executable in at least a target-end database; and executing the migration operation with the target-side database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value.

Fig. 1 schematically illustrates an application scenario diagram of a data bi-directional migration method according to an embodiment of the present disclosure.

As shown in fig. 1, an application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that, the data bidirectional migration method provided by the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the data bi-directional migration apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 105. The data bi-directional migration method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the data bidirectional migration apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The data bidirectional migration method of the disclosed embodiment will be described in detail below with reference to fig. 2 to 6 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a data bi-directional migration method according to an embodiment of the present disclosure.

As shown in fig. 2, the data bi-directional migration method of this embodiment includes operations S210 to S250, and the data bi-directional migration method may be performed by the server 105.

In operation S210, it is monitored whether log information in the source database is changed.

Specifically, the embodiment of the disclosure can be applied to realizing bidirectional migration of data under the condition that the source end database and the target end database are not stopped. The source database and the target database may be databases of the same environment, for example, the source database and the target database are both Oracle databases, or the source database and the target database pass through MySQL databases; the source database and the target database may be databases of different environments, for example, the source database and the target database are respectively an Oracle database and a MySQL database, or the source database and the target database are respectively a MySQL database and an Oracle database.

Specifically, the bidirectional migration includes bidirectional incremental data migration and decremental data migration, for example, adding data to the source database, and accordingly, migrating the incremental data to the target database, for example, when the source database deletes the data. It will be appreciated that the source database in operation S210 refers to the original end of the data modification, rather than a specific database, for example, when the Oracle database and the MySQL database are the source database and the target database, the Oracle database is the source database when the Oracle database changes, and when the MySQL database changes, the Oracle database is the source database.

In operation S220, under the condition that the log information changes, change data under different data tables in the source database is obtained, where the change data includes a field of a flag bit.

Specifically, taking a database of different environments as an example under an incremental migration scene:

and for the Oracle database, before starting incremental data migration, creating a materialized view and a materialized view log according to a main key field of a table to be migrated under the Oracle database. And if the Oracle database table is subjected to data change, analyzing materialized view log file information, locating and changing the primary key field, the common field and the corresponding value of the incremental data by inquiring the binary code of the change object, and re-packaging the information into a JSON format for recording.

For the MySQL database, the MySQL database table is subjected to data change, a bin log file (binary log file) is monitored, and according to the change time node time stamp, an SQL statement corresponding to the change data is inquired and obtained.

In operation S230, a field of a flag bit in the change data is changed.

Specifically, if the Oracle database and the MySQL database have incremental data at the same time in a period of time, the corresponding default flag bit value is changed according to the incremental data in the Oracle database and the MySQL database at the same time. And encapsulating the change data for changing the field value of the flag bit.

In operation S240, the change data is converted into a change statement, wherein the change statement is executable at least in the target database.

After the marker bit is judged, the target-end database and the source-end database are written in two directions, so that double-data source data synchronous backup is completed, and data safety is improved.

Specifically, an executable SQL statement for the source database and the target database is generated. For example, the SQL statement may be an UPDATE statement. Researchers have found that SQL statements can be used both in the Oracle database and in the MySQL database, and therefore, SQL statements can be used for the change statement.

In operation S250, if the field of the flag bit of the change statement is a non-default value, a migration operation with the target database is performed based on the change statement.

Specifically, the Oracle database and the MySQL database respectively and correspondingly judge the written SQL statement, analyze the field value of the flag bit, and if the flag bit field value is judged to be changed, the change statement generated by the default Oracle database does not need to be repeatedly migrated to the Oracle database, and the change statement generated by the MySQL database does not need to be repeatedly migrated to the MySQL database. And using the persistent layer framework to drop libraries or perform other business processing. Of course, in the scene of needing to carry out backup, SQL sentences can be written into the Oracle database and the MySQL database simultaneously to finish backup, so that when unknown errors occur in one end of the database to cause downtime, quick switching back can be carried out.

The source end database and the target end database are independent and the writing-in and writing-out of data are frequent, so that message middleware can be set for frequent bidirectional synchronous migration operation between the two databases.

Fig. 3 schematically illustrates a flow chart of a method of altering data acquisition according to an embodiment of the disclosure.

As shown in fig. 3, the modification data acquisition method of this embodiment includes operations S310 to S340, and operations S310 to S340 may at least partially execute operation S220 described above.

In operation S310, a data table to which the change data relates is acquired.

The database contains a plurality of data tables, and the operations performed on different data tables generate change data, that is, the change data acquired through the change of the log can locate the data table where the change data is located.

In operation S320, a topic, a partition lane, and a consumer model are created based on the message middleware, wherein the topic corresponds to the data table one by one, the consumer model corresponds to the data table one by one, and the topic corresponds to the partition lane one by one.

It can be understood that the topics belong to logic concepts, and the topics and the data table have a certain corresponding relationship, and are generally divided according to business logic, so that the topics and the data table have a one-to-one corresponding relationship. For example, a data table records data of "service 1", and then the table corresponds to the topic "service 1".

In operation S330, the change data is stored in the theme according to the one-to-one correspondence between the theme and the data table.

In operation S340, the change data is pulled by lot based on the consumer model.

Specifically, according to the number of tables to be changed of the source end database and the target end database, respectively creating a corresponding Topic theme and a corresponding Partition channel Partition by using a message middleware, and sending repackaging information corresponding to incremental data of the source end database and the target end database to the Topic theme of the corresponding message middleware; creating a corresponding number of Consumer models Consumers according to the number of message middleware topics; and (3) regularly (or quantitatively) pulling messages in batches, distinguishing incremental data change information of the source database table and the target database table according to different Topic subjects serving as key values, and ensuring that information to be consumed in the Partition channel under the same Topic is the same data source database ordering table.

It can be understood that the split channel can be regarded as a channel, the channel is serial in migration, and in the process of performing one theme migration, one theme can realize character migration according to serial channels of a plurality of split channels.

In the embodiment of the disclosure, the bidirectional synchronous migration can be realized on the source end database and the target end database generating the service change through the message middleware, so that the interaction of service data is realized under the decoupling condition between different databases.

Fig. 4 schematically illustrates a flow chart of a data bi-directional migration method according to an embodiment of the present disclosure.

As shown in fig. 4, the data bi-directional migration method of this embodiment includes operations S410 to S430, and the operations S410 to S430 are performed after the operation S260.

According to an embodiment of the disclosure, wherein the batch comprises a first batch.

It is understood that the first batch refers to any batch.

In operation S410, based on the returned migration result of the partition channel, it is checked whether migration of a plurality of topics in the first batch is successful, where the plurality of topics includes a first topic, and if any one of the migration results in the first topic is failure, it is determined that migration of the first topic fails.

In operation S420, in the case that the migration of the first theme is successful, a first offset of a last migration result of the first theme is recorded, and the first offset is submitted after being added with a process. .

In operation S430, in case that the migration of the first theme fails, a second offset of the migration result of the migration failure in the split channel is recorded, and the second offset is submitted.

It will be appreciated that the first offset corresponds to the offset in the case of success and the second offset corresponds to the offset in the case of failure.

And obtaining the writing results of the persistent layer frameworks of all threads in the same batch, if network delay occurs in the data migration process or the message middleware accidentally stops running due to the actual production environment problem, sequentially and circularly traversing all returned results according to the Topic theme (the theme generally corresponds to a data table) as distinguishing information, and monitoring the returned information of all results.

Specifically, when the return values of the Partition channel messages under all the different topics of the batch are traversed, the offset value of the last message in the different topics of the batch is recorded, and the offset value added by 1 to the Partition channel offset under the Topic where the imported data error does not appear is submitted at the same time, so that the Partition under the Topic where the imported data error occurs needs to submit the message offset value with the error.

And similarly, after the data of each batch is consumed, the offset of the message which can possibly generate errors in the data migration process can be monitored, recorded and submitted to the message middleware, and meanwhile error reporting processing is carried out. The ordering and the uniqueness of the incremental data of the database are guaranteed.

Fig. 5 schematically illustrates a flowchart of a migration result judgment method according to an embodiment of the present disclosure.

As shown in fig. 5, the migration result judging method of this embodiment includes operations S510 to S520, and the operations S510 to S520 may at least partially execute the above-described operation S410.

And under the condition that the migration of the first theme fails, recording a second offset of a migration result of the migration failure in the partition channel, submitting the second offset, traversing a plurality of migration results for the first channel, and judging whether the migration result fails or not in operation S510.

In operation S520, if the traversing multiple migration results determines that the migration result fails, it is determined that the migration of the current topic where the migration result is located fails, and the traversing of multiple partition channels in the current topic is stopped and the next topic is entered.

Specifically, if the return result of the Partition split channel under a single Topic is false failure, the offset of the consumption failure message is recorded, and meanwhile, the cycle traversal of the Topic message of the subject is stopped, and the next traversal of the Topic subject message is performed. Therefore, the resources of the Partition channels can be effectively utilized, and the maximization of the utilization of the Partition channels is ensured.

Fig. 6 schematically illustrates a full flow chart of a data bi-directional migration method according to an embodiment of the present disclosure.

As shown in fig. 6, the data bidirectional migration method includes: under the condition that the Oracle database form and/or the MySQL database form are changed, analyzing the log to obtain change data; creating Topic corresponding to Oracle/MySQL; pulling the message in Oracle/MySQL to Topic; judging whether the flag bit is a default value or not; if yes, modifying the flag bit, assembling SQL sentences, and writing into a MySQL/Oracle database; otherwise, judging whether the data is backup data, and falling into a database without writing back; judging whether the writing is successful or not according to the return value; if yes, recording the written data position; otherwise, the error location is recorded and the offset is committed, while the data migration is interrupted.

Based on the data bidirectional migration method, the disclosure also provides a data bidirectional migration device. The device will be described in detail below in connection with fig. 7.

Fig. 7 schematically illustrates a block diagram of a data bi-directional migration apparatus according to an embodiment of the present disclosure.

As shown in fig. 7, the data bi-directional migration apparatus 700 of this embodiment includes a log monitoring module 710, a change data obtaining module 720, a flag bit changing module 730, a statement converting module 740, and a migration module 750.

The log monitoring module 710 is configured to monitor whether log information in the source database changes. In an embodiment, the log monitoring module 710 may be configured to perform the operation S210 described above, which is not described herein.

The change data obtaining module 720 is configured to obtain change data under different data tables in the source database when the log information changes, where the change data includes a field of a flag bit. In an embodiment, the change data obtaining module 720 may be configured to perform the operation S220 described above, which is not described herein.

The flag bit changing module 730 is configured to change a field of a flag bit in the change data. In an embodiment, the flag changing module 730 may be used to perform the operation S230 described above, which is not described herein.

The statement conversion module 740 is configured to convert the change data into a change statement, where the change statement is at least executable in the target database. In an embodiment, the sentence conversion module 740 may be used to perform the operation S240 described above, which is not described herein.

The migration module 750 is configured to execute a migration operation with the target database based on the change statement if a field of a flag bit of the change statement is a non-default value. In an embodiment, the migration module 750 may be configured to perform the operation S250 described above, which is not described herein.

Any of the log monitoring module 710, the change data obtaining module 720, the flag bit changing module 730, the statement converting module 740, and the migration module 750 may be combined in one module to be implemented, or any of the modules may be split into a plurality of modules according to an embodiment of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the log monitoring module 710, the change data acquisition module 720, the flag bit modification module 730, the statement conversion module 740, and the migration module 750 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of any of the three. Alternatively, at least one of the log monitoring module 710, the change data acquisition module 720, the flag bit modification module 730, the statement conversion module 740, and the migration module 750 may be at least partially implemented as a computer program module, which when executed, may perform the corresponding functions.

As shown in fig. 8, an electronic device 800 according to an embodiment of the present disclosure includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 801 may also include on-board memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the disclosure.

In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 802 and/or the RAM 803. Note that the program may be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 800 may also include an input/output (I/O) interface 805, the input/output (I/O) interface 805 also being connected to the bus 804. The electronic device 800 may also include one or more of the following components connected to the I/O interface 805: an input portion 806 including a keyboard, mouse, etc.; an output portion 807 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 808 including a hard disk or the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. The drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as needed so that a computer program read out therefrom is mounted into the storage section 808 as needed.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 802 and/or RAM 803 and/or one or more memories other than ROM 802 and RAM 803 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code, when executed in a computer system, causes the computer system to perform the methods provided by embodiments of the present disclosure.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 801. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed, and downloaded and installed in the form of a signal on a network medium, and/or from a removable medium 811 via a communication portion 809. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 809, and/or installed from the removable media 811. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 801. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A method of bi-directional migration of data, comprising:

monitoring whether the log information in the source database changes or not;

Under the condition that the log information changes, obtaining change data under different data tables in the source end database, wherein the change data comprises a field of a flag bit;

changing the field of the flag bit in the change data;

converting the change data into change sentences, wherein the change sentences are executable in at least a target-end database; and

and executing migration operation with the target-side database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value.

2. The method of claim 1, wherein the obtaining the change data under the different data tables in the source database comprises:

acquiring a data table related to the change data;

creating a theme, a segmentation channel and a consumer model based on a message middleware respectively, wherein the theme corresponds to the data table one by one, the consumer model corresponds to the data table one by one, and the theme and the segmentation channel are in one-to-many relation;

storing the change data into the theme according to the one-to-one correspondence between the theme and the data table; and

and pulling the change data by batch based on the consumer model.

3. The method of claim 2, wherein the batch comprises a first batch,

after the migration operation with the target-side database is performed based on the change statement, the method further includes:

based on the returned migration result of the partition channel, checking whether a plurality of topics are migrated successfully in the first batch, wherein the topics comprise a first topic, and if any one of the migration results in the first topic is failure, judging that the migration of the first topic is failure;

under the condition that the first theme is successfully migrated, recording a first offset of a last migration result of the first theme, and submitting the first offset after adding one process; and

and under the condition that the migration of the first theme fails, recording a second offset of a migration result of the migration failure in the partition channel, and submitting the second offset.

4. The method of claim 3, wherein the partitioned channels include at least a first channel,

based on the returned migration result of the partition channel, checking whether the migration of the plurality of topics in the first batch is successful, including:

Traversing a plurality of migration results for the first channel, and judging whether the migration results fail or not; and

and under the condition that the migration results are judged to fail by traversing the plurality of migration results, judging that the migration of the current theme where the migration results are located fails, stopping traversing the plurality of partition channels in the current theme and entering the next theme.

5. The method of claim 1, wherein the change statement is executable in the source database,

after the converting the change data into the change statement, the method further includes: and backing up the change statement to the source database.

6. The method of any one of claims 1-5, wherein the source database comprises an Oracle database and a MySQL database, the target database comprises an Oracle database and a MySQL database,

under the condition that the log information changes, acquiring the change data under different data tables in the source database comprises the following steps:

analyzing the materialized view log for the Oracle database to obtain a binary code of a change object;

inquiring the binary code of the change object, and positioning a main key field, a common field and a value of the change data;

Packaging and encapsulating the primary key field, the common field and the value to obtain the change data;

analyzing a log file for the MySQL database to obtain a change time node time stamp;

querying an SQL statement based on the time node timestamp; and

and packaging the SQL statement to obtain the change data.

7. A data bi-directional migration apparatus comprising:

the log monitoring module is used for monitoring whether the log information in the source end database changes or not;

the change data acquisition module is used for acquiring change data under different data tables in the source database under the condition that the log information changes, wherein the change data comprises a field of a flag bit;

the flag bit changing module is used for changing the field of the flag bit in the change data;

the statement conversion module is used for converting the change data into change statements, wherein the change statements are executable in at least a target-end database; and

and the migration module is used for executing migration operation with the target-end database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value.

8. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-6.

9. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.