CN113094434A

CN113094434A - Database synchronization method, system, device, electronic equipment and medium

Info

Publication number: CN113094434A
Application number: CN202110379979.8A
Authority: CN
Inventors: 陈磊; 陈雪峰
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-04-08
Filing date: 2021-04-08
Publication date: 2021-07-09

Abstract

The disclosure provides a database synchronization method, and belongs to the technical field of databases. The method comprises the following steps: responding to a user request of data operation received by any one of the M databases, and acquiring the user request; obtaining a data synchronization message based on the user request, wherein the data synchronization message comprises identification information used for indicating data operated by the user request and direction information used for indicating the synchronization direction of the data between the databases; pushing the data synchronization message to a distributed message queue; subscribing data synchronization messages from a distributed message queue; and after subscribing the data synchronization message, acquiring the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information. The present disclosure also provides a database synchronization system and apparatus, an electronic device, and a computer-readable storage medium.

Description

Database synchronization method, system, device, electronic equipment and medium

Technical Field

The present disclosure belongs to the field of database technologies, and in particular, to a database synchronization method, system, device, electronic device, and medium.

Background

Under a plurality of scenes that a database needs to be replaced when a system carries out technical transformation, and the like, two or more sets of clusters are generally operated in parallel. The databases of different clusters need to perform quasi-real-time data synchronization so as to meet the requirement that data between users on two or more clusters can interact with each other during a parallel period, and when a user is switched from one cluster to another cluster for use, the continuity of services can be ensured.

In the related art, each type of database basically has a respective data synchronization mechanism, but generally depends on the core technology of the database itself, and is limited to the mutual synchronization between homogeneous databases or between a few databases. When the database of the cluster requiring data synchronization is a heterogeneous database, the source database is often required to export data to be synchronized into a file, and then the file is transmitted to the server of the target database through a certain external tool to be reprocessed and imported. The whole process has the problems of low treatment time efficiency, difficulty in treating the relation among multiple tables and the like.

Disclosure of Invention

In view of the above, the embodiments of the present disclosure provide a database synchronization method, system and apparatus, an electronic device, and a computer-readable storage medium.

One aspect of the disclosed embodiments provides a database synchronization method. The method comprises the following steps: responding to a user request of data operation received by any one of M databases, and acquiring the user request, wherein M is an integer greater than or equal to 1; obtaining a data synchronization message based on the user request, wherein the data synchronization message comprises identification information used for indicating data operated by the user request and direction information used for indicating a synchronization direction of the data between databases; pushing the data synchronization message to a distributed message queue; subscribing to the data synchronization message from the distributed message queue; and after subscribing the data synchronization message, acquiring the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information.

According to an embodiment of the present disclosure, the obtaining a user request in response to any one of the M databases receiving a user request for data operation includes: in response to receiving the user request, identifying a transaction name in the user request, the transaction name indicating a business function performed by the user request; and when the transaction name belongs to a preset transaction name set, acquiring the user request, wherein the transaction name set is a preset transaction name set needing data synchronization.

According to an embodiment of the present disclosure, the data synchronization message further includes the transaction name. Wherein the obtaining of the data synchronization message based on the user request comprises: extracting the transaction name and the identification information from the user request; obtaining the direction information according to the data synchronization configuration of the database operated by the user request; and packaging the transaction name, the identification information and the direction information to obtain the data synchronization message.

According to an embodiment of the present disclosure, the obtaining the complete data indicated by the identification information from the source database indicated by the direction information includes: acquiring information of N data tables related to the transaction indicated by the transaction name; querying data from the N data tables in the source database to obtain the complete data, specifically including: acquiring data indicated by the identification information from each data table in the source database by using a query statement for acquiring a data manipulation result from each data table in the source database to obtain local data corresponding to each data table, wherein N data tables correspond to N local data; and combining the N pieces of local data to obtain the complete data.

According to an embodiment of the present disclosure, the writing the complete data to the target database indicated by the direction information includes: inquiring data from N data tables in the target database to obtain check data; comparing the complete data with the check data to obtain a comparison result; and writing the complete data to the target database when the comparison result indicates that the complete data does not violate the database constraint of the target database. Wherein, querying data from the N data tables in the target database to obtain the check data includes: acquiring data indicated by the identification information from each data table in the target database by using a query statement for acquiring a data manipulation result from each data table in the target database to obtain local check data corresponding to each data table, wherein N data tables correspond to N local check data; and merging the N local check data to obtain the check data.

According to an embodiment of the present disclosure, the writing the complete data to the target database indicated by the direction information includes: assembling the replay execution statement for data synchronization in the target database according to all field data in the complete data to form a data manipulation statement; and submitting the data manipulation statement to the target database for execution.

According to the embodiment of the disclosure, when N is larger than 1, data is inquired from S data tables concurrently, wherein S is an integer and S is more than or equal to 2 and less than or equal to N.

According to an embodiment of the present disclosure, the method further comprises: presetting the information of N data tables related to the transaction indicated by the transaction name; presetting query statements for respectively acquiring data manipulation results from each data table in the source database and the target database; and presetting a replay execution statement of the target database, wherein the replay execution statement is used for manipulating data in the target database.

According to an embodiment of the present disclosure, M is greater than 1, and the source database and the target database both belong to M databases.

In another aspect of the disclosed embodiments, a database synchronization system is provided. The system includes an interceptor, a distributed message queue, and a data synchronization cluster. The interceptor is configured to: responding to a user request of data operation received by any one of M databases, and acquiring the user request, wherein M is an integer greater than or equal to 1; obtaining a data synchronization message based on the user request, wherein the data synchronization message comprises identification information used for indicating data operated by the user request and direction information used for indicating a synchronization direction of the data between databases; and pushing the data synchronization message to the distributed message queue. The distributed message queue is configured to receive the data synchronization message and notify the data synchronization cluster to consume the data synchronization message. The data synchronization cluster is to: subscribing to the data synchronization message from the distributed message queue; and after subscribing the data synchronization message, acquiring the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information.

The embodiment of the disclosure provides a database synchronization device. The device comprises a request acquisition module, a synchronous message acquisition module, a pushing module, a subscription module and a data synchronization module. The request acquisition module is used for responding to a user request of data operation received by any one of M databases and acquiring the user request, wherein M is an integer larger than or equal to 1. The synchronous message obtaining module is used for obtaining a data synchronous message based on the user request, wherein the data synchronous message comprises identification information used for indicating data operated by the user request and direction information used for indicating a synchronous direction of the data between databases. And the pushing module is used for pushing the data synchronization message to a distributed message queue. And the subscription module is used for subscribing the data synchronization message from the distributed message queue. And the data synchronization module is used for acquiring the complete data indicated by the identification information from the source database indicated by the direction information after subscribing the data synchronization message, and writing the complete data into the target database indicated by the direction information.

According to an embodiment of the present disclosure, the apparatus further comprises a setup module. The setting module is used for: presetting the information of N data tables related to the transaction indicated by the transaction name; presetting query statements for respectively acquiring data manipulation results from each data table in the source database and the target database; and presetting a replay execution statement of the target database, wherein the replay execution statement is used for manipulating data in the target database.

In another aspect of the disclosed embodiments, an electronic device is provided. The electronic device includes one or more memories, and one or more processors. The memory stores executable instructions. The processor executes the executable instructions to implement the method as described above.

Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.

Another aspect of embodiments of the present disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.

One or more of the above-described embodiments may provide the following advantages or benefits: the operation of the database and the data synchronization are decoupled by using the distributed message queue, so that the availability of external services of a source database and a target database for data synchronization is not dependent on each other, and the efficiency of data synchronization is improved.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario of a data synchronization method and system according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a method of data synchronization according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow diagram for obtaining a data synchronization message according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow diagram for writing to a target database after subscribing to a data synchronization message according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow diagram for writing to a target database after subscribing to a data synchronization message, according to another embodiment of the present disclosure;

FIG. 6 schematically shows a block diagram of a data synchronization apparatus according to an embodiment of the present disclosure; and

fig. 7 schematically shows a block diagram of an electronic device adapted to implement the data synchronization method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

Embodiments of the present disclosure provide a database synchronization method, system, apparatus, electronic device, and computer-readable storage medium. Specifically, the method comprises the steps of firstly responding to a user request of data operation received by any one of M databases, and obtaining the user request, wherein M is an integer greater than or equal to 1; then, based on the user request, obtaining a data synchronization message, wherein the data synchronization message comprises identification information used for indicating data operated by the user request and direction information used for indicating the synchronization direction of the data among the databases; next, pushing the data synchronization message to a distributed message queue; then subscribing data synchronization information from the distributed information queue; and after subscribing to the data synchronization message, acquiring the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information.

In this way, the database synchronization can be carried out without depending on the synchronization mechanism of the database or the log format, so that the method can be applied to the data synchronization among a wide range of relational databases. Moreover, the embodiment of the present disclosure decouples the operation and data synchronization of the databases by using the distributed message queues, so that the embodiment is not only suitable for data synchronization between homogeneous databases, but also suitable for data synchronization between heterogeneous databases, and further enables the availability of external services of the source database and the target database for data synchronization to be independent of each other, thereby not affecting the continuity of the service.

It should be noted that the data synchronization method, system and apparatus determined in the embodiments of the present disclosure may be used in the financial field, and may also be used in any field other than the financial field, and the present disclosure does not limit the application field.

Fig. 1 schematically illustrates an application scenario 100 of a data synchronization method and system according to an embodiment of the present disclosure.

As shown in fig. 1, the application scenario 100 may include an old system 10, a new system 20, and a data synchronization system according to an embodiment of the present disclosure. Wherein the data synchronization system can achieve bidirectional synchronization of data (also referred to as double write synchronization) between the old system 10 and the new system 20.

In particular, the cluster of old application servers 11 and old application database 12 represents the old system 10. The cluster of new application servers 21 and new application database 22 represents the new system 20 after reconstruction or transformation.

The data synchronization system may include

interceptors

101A, 101B, distributed message queues 102, and data synchronization clusters 103.

Wherein the

interceptors

101A and 101B may be provided in the old system 10 and the new system 20, respectively. Each interceptor may be used to intercept user requests of the system in which it resides. Specifically, the interceptor 101A is configured to receive a user request for a data operation by the old system 10, obtain the user request, encapsulate a data synchronization message (e.g., one of the data synchronization messages 1, 2, 3.) based on the user request, and then push the data synchronization message to the distributed message queue 102. Accordingly, the interceptor 101B is configured to receive a user request for a data operation by the new system 20, obtain a data synchronization message (e.g., one of the data synchronization messages 1, 2, 3.) according to the user request, and then push the data synchronization message to the distributed message queue 102.

The distributed message queue 102 is used to receive and store the data synchronization messages pushed by the

interceptors

101A, 101B, and notify the data synchronization cluster 103 to consume the data synchronization messages.

The data synchronization cluster 103 is configured to subscribe to data synchronization messages from the distributed message queue 102, and after subscribing to each data synchronization message, implement double write synchronization of data in the old system 10 and the new system 20 according to information in each data synchronization message.

In the application scenario 100, the two systems, the old system 10 and the new system 20, can provide services to the outside at the same time, for example, a non-grayscale user enters access from the old system 10 and a grayscale user enters access from the new system 20.

In the normal online access mode, the request of the non-grayscale user may enter from the old application server cluster 11, and after performing the relevant operation, the data in the old application database 12 is directly operated according to the application program logic operation, and meanwhile, after intercepting the user request, the interceptor 101A provided in the old application server cluster 11 assembles and pushes the information such as the identification information (e.g., the data main key) of the data to be operated to the distributed message queue 102.

Accordingly, a request of a grayscale user may enter from the new application server cluster 21, and after performing related operations, data in the new application database 22 is directly operated according to application program logic operations, and meanwhile, after intercepting the user request, the interceptor 101B provided in the new application server cluster 21 assembles information such as identification information (e.g., a data main key) of the data to be operated and then pushes the assembled information to the distributed message queue 102.

The distributed message queue 102 receives push messages from the clusters on both sides and notifies the data synchronization cluster 103 to consume.

The data synchronization cluster 103 may also be a set of server clusters deployed with application programs, and after subscribing data synchronization messages pushed by the new and old systems from the distributed message queue 102, acquires each message, acquires complete data from the source database according to the source of the message, and directly writes the complete data into the target database, thereby implementing data double-write synchronization between the new and old systems.

According to the embodiment of the disclosure, the distributed message queue 102 is adopted to decouple the business operation and data synchronization of the old system 10 and the new system 20, so that the respective external service availability is not dependent on each other, and the final consistency of the data of the databases of the old system and the new system can be ensured in a short time to a greater extent. Moreover, when the new and old systems (including the application layer and the database) are switched integrally by a gray release mechanism, the gray user range can be flexibly switched according to the embodiment of the disclosure, and the timeliness of data interaction between users respectively using the new and old systems can also be effectively ensured.

According to the application scenario 100, when the system needs to be subjected to technology transformation and the whole system needs to be reconstructed or the database needs to be replaced for model selection, the method and the system of the embodiment of the disclosure can effectively ensure that the data of the cluster databases on the two sides of the distributed message queue 102 can achieve final consistency in a short time to a certain extent, and enable the new system and the old system to run in parallel, so as to fully verify the reliability of the new technology in actual production, gradually switch user traffic to the new system, and avoid generating a scenario with great influence on stable operation. And when the problem of influencing the use of the user is met, the user flow can be flexibly switched back without influencing the continuity of the service.

It should be noted that fig. 1 is only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

For example, the system and method of the embodiments of the present disclosure may also be applied in a database backup scenario, that is, a source database unidirectionally synchronizes data into a backup database (i.e., a target database) through the data synchronization system of the embodiments of the present disclosure.

For another example, the system and method of the embodiments of the present disclosure may also be applied to a scenario in which data synchronization between two or more systems is implemented, or data backup of one or more of the systems is implemented. For example, each of three or more systems may be distributed to push a data synchronization message to the distributed message queue 102, where the data synchronization message may include direction information of data synchronization, so that after the data synchronization cluster 103 subscribes to each data synchronization message from the distributed message queue 102, the source database and the target database are determined according to the direction information therein for data synchronization. It can be seen that in the data synchronization system and method of the embodiments of the present disclosure, data synchronization is decoupled from the traffic of the database itself by using the distributed message queue 102. In this way, after a set of data synchronization system is constructed, any plurality of systems which need to be synchronized can be accessed according to actual needs.

The method of the embodiments of the present disclosure is exemplarily described below with reference to fig. 2 to 5 in conjunction with the application scenario 100 of fig. 1.

Fig. 2 schematically shows a flow chart of a data synchronization method according to an embodiment of the present disclosure.

As shown in fig. 2, the data synchronization method may include operations S210 to S250.

In operation S210, in response to a user request for data operation received by any one of M databases, a user request is obtained, where M is an integer greater than or equal to 1. For example, the

interceptors

101A, 101B intercept respective user requests to manipulate data in their respective databases when the old system 10 and the new system 20 receive the respective user requests in.

In some embodiments, the

interceptors

101A, 101B may grab any user request to perform a data operation. That is, once the old system 10 and/or the new system 20 receives the user request, a corresponding interception may be performed.

In other embodiments, the

interceptors

101A, 101B may only grab a portion of the user-specified request, rather than all of the request, when grabbing the user request. For example, in operation S210, in response to receiving the user request, a transaction name in the user request may be first identified, where the transaction name is information in the user request indicating a service function performed by the user request, and then the user request is obtained when the transaction name belongs to a preset transaction name set, where the transaction name set is a preset set of transaction names that need data synchronization.

For example, in a scenario where the application scenario 100 relates to migration of new and old systems, transactions that generally involve user information are transactions that require data synchronization (e.g., transfer payment transactions, payroll batch transactions, user information modification transactions, etc.), and some transactions such as parameter setting, log viewing, log recording, etc. may have little influence on business operations, may be regarded as non-important transactions, and may not be intercepted and synchronized.

In operation S220, a data synchronization message is obtained based on the user request, wherein the data synchronization message includes identification information indicating data operated by the user request and direction information indicating a synchronization direction of the data between the databases. The identification information may be, for example, a data primary key, or may be, for example, other information that may be located to a data record operated by a user. The direction information is used to clarify which database the data synchronization is from to, and for example, to characterize the direction of the data synchronization by a predetermined character. The capacity of the data synchronization messages is generally not too large and therefore only some critical information may be encapsulated therein.

After the identification information and the direction information are encapsulated to obtain the data synchronization message, when the data synchronization cluster 103 consumes one data synchronization message, the source database and the target database can be determined through the direction information, and the complete data in the source database can be located through the identification information.

According to some embodiments of the present disclosure, when the

interceptors

101A, 101B are grabbing user requests of transactions belonging to a preset set of transaction names, the data synchronization message further includes the transaction name. The process of encapsulating the data synchronization message in operation S230 may be, on one hand, extracting the transaction name and the identification information from the user request, on the other hand, obtaining the direction information according to the data synchronization configuration of the database operated by the user request, and then encapsulating the transaction name, the identification information, and the direction information to obtain the data synchronization message.

In operation S230, the data synchronization message is pushed to the distributed message queue 102.

In operation S240, a data synchronization message is subscribed from the distributed message queue 102. For example, data synchronization cluster 103 subscribes.

In operation S250, after subscribing to the data synchronization message, the data synchronization cluster 103 obtains the complete data indicated by the identification information from the source database indicated by the direction information, and writes the complete data into the target database indicated by the direction information.

According to an embodiment of the present disclosure, the decoupling and asynchronous operation of data synchronization and system traffic is implemented using the distributed message queue 102, so that the source database and the target database are independent of each other. And the characteristic of peak clipping of the distributed message queue 102 can be utilized, the efficiency and timeliness of data synchronization are improved, and the influence on the respective performances of the source database and the target database is reduced.

Fig. 3 schematically shows a flow chart for obtaining a data synchronization message according to an embodiment of the present disclosure.

As shown in fig. 3, the process of obtaining the data synchronization message according to the embodiment may include operations S301 to S306.

The user request is grabbed in operation S301. Mechanisms for triggering data synchronization may be added to the application servers (i.e., the old application server cluster 11 and the new application server cluster 21) before this, including setting up the

interceptors

101A, 101B, and timing of intercepting the user request, intercepting conditions (e.g., intercepting only the user-specified request), and so on.

Then, in operation S302, the request operation of the user is continuously executed, and the reading and writing of the clustered database are completed. After triggering the filter, firstly, continuing to call the corresponding program according to the original functional logic to perform corresponding operation on the database in the cluster.

Next, in operation S203, it is determined whether the transaction requested by the user requires double write synchronization. And judging whether the transaction needs to trigger double-write synchronization to another cluster according to the parameters configured in advance.

In operation S304, a normal return is made if a double write is not required.

In operation S305, if double writing is required, an asynchronous processing request is called, parameters such as a transaction name and a data key in the intercepted request are obtained, and a direction of data synchronization (i.e., direction information) is determined

Then, in operation S306, the transaction name, the data primary key, and the direction information in the intercepted request are encapsulated as a data synchronization message, and pushed to the distributed message queue 102.

Fig. 4 schematically shows a flowchart of writing to a target database after subscribing to a data synchronization message in operation S250 according to an embodiment of the present disclosure.

As shown in fig. 4, operation S250 may include operations S401 to S407 according to an embodiment of the present disclosure. In this embodiment, the data synchronization message includes a transaction name, identification information, and direction information.

In operation S401, information of N data tables related to a transaction indicated by a transaction name is acquired. The transactions that each system can perform are determined when the system is put into operation. Assuming that the legacy system 10 is a system that provides credit card services to users, the transactions involved therein may include, for example, credit card transactions, credit card payments, credit card cash withdrawals, and the like. Credit card transactions assume that the user is required to provide basic information (name, phone number, identification number, etc.), revenue information, asset information, etc., and accordingly the N data tables involved in credit card transactions may be, for example, basic information data tables, payroll revenue data tables, personal asset information tables, etc.

In operation S402, data is queried from N data tables in the source database to obtain complete data.

For example, a query statement (select statement, denoted as SQL-a) for obtaining a data manipulation result from each data table in the source database may be used to obtain data indicated by the identification information from each data table in the source database, and obtain local data corresponding to each data table, where N local data are obtained by N data tables. And then merging the obtained N local data to obtain complete data.

For example, if it is determined that transaction data of the transaction card of zhang san applition is to be synchronized according to the transaction name and the identification information in the data synchronization message, the information of zhang san can be obtained by querying the basic information data table, the payroll data table, and the personal asset information table from the source database, and then combining the queried information.

In operation S403, data is queried from N data tables in the target database, and check data is obtained.

Correspondingly, for example, a query statement (also referred to as a select statement, SQL-a) for obtaining a data manipulation result from each data table in the target database may be used to obtain data indicated by the identification information from each data table in the target database, so as to obtain local check data corresponding to each data table, where N data tables correspond to N local check data. And then merging the N local check data to obtain check data.

Similarly, for example, when transaction data of a credit card is to be transacted by the application of zhang san, information registered by zhang san at the time of transacting the credit card (i.e., check data) recorded in the target database can be obtained by querying the basic information data table, the wage income data table, and the personal asset information table from the target database, respectively, for zhang san information (if any), and then combining the queried information.

In operation S404, the complete data and the check data are compared to obtain a comparison result.

In operation S405, it is determined whether the complete data violates a database constraint (e.g., a primary key conflict) of the target database according to the comparison result. If not, performing operation S406; if so, operation S407 is performed. For example, in one case, assuming that the information for Zhang three is not found from the target database, it is clear that the database constraints are not violated. For another example, in a search scenario, Zhang III information is found from the target database, but the timestamp of this information is earlier than the information from the source database, indicating that the complete data does not violate the target database constraints. As another example, the timestamp of the Zhang three information found from the target database may be later than the full information in the source database in a case where the full data violates the database constraints. It should be noted that several common situations of database constraint-based determination are described here by way of example only. The specific decision logic of the database constraints depends on the settings of the respective databases.

In operation S406, the complete data is written to the target database when the complete data does not violate the database constraint of the target database.

The description is continued by taking the data of the synchronous three-transaction credit card as an example. For example, a replay execution statement (denoted as SQL-B) for Data synchronization in the target database may be assembled according to all field Data in the complete Data to form a Data Manipulation statement (DML), and then the Data Manipulation statement is submitted to the target database for execution.

In operation S407, when the complete data violates the database constraint of the target database, it indicates that the complete data has existed in the target database before, so the complete data is discarded to avoid adverse effects on the target database. .

According to the embodiment of the disclosure, the synchronous data is verified through the database constraint, so that the data security and accuracy can be improved, and the adverse effect on the database can be avoided.

According to the disclosed embodiment, the application layer can use an asynchronous trigger mechanism through the interceptor, does not use a database trigger, simultaneously uses a simple select statement to acquire the original data from the source database query, uses a simple DML statement to perform the operation of adding, deleting and modifying on the target database, and has lower influence on the performance of the source database and the target database on the original database and the target database.

FIG. 5 schematically illustrates a flow diagram for writing to a target database after subscribing to a data synchronization message, according to another embodiment of the disclosure.

As shown in fig. 5, in this embodiment, the data synchronization cluster 103 may subscribe to the data synchronization message from the distributed message queue 102 through operations S501 to S511, and write to the target database.

First, in operation S501, a message is subscribed from the distributed message queue 102. Before subscription, it is also possible to preset, in the data synchronization cluster 103, information of N data tables related to a transaction indicated by a transaction name, a select query statement SQL-a for obtaining a data manipulation result from each data table in the source database and the target database, respectively, and a replay execution statement SQL-B of the target database, where the replay execution statement SQL-B is used to manipulate data in the target database (e.g., insert, update, delete, etc.).

Then in operation S502, M messages may be acquired at a time, and M may be configured by parameters. The transaction name, parameters and direction of double-write are obtained from each message.

In operation S503, it is determined whether there are any unconsumed messages in the distributed message queue 102. If yes, returning to the distributed message queue 102 to continue to acquire the message after the current message consumption is completed. If not, then a message notification is awaited for the distributed message queue 102.

Next, in operation S504, a transaction name, a data primary key, and a direction of double writing are acquired from the data synchronization message.

Then, in operation S505, it is determined whether the transaction requires double writing according to the information in operation S504. If necessary, operation S506 is continued, and if not, operation S503 is returned to.

In operation S506, for a transaction requiring double writing, the related tables are obtained according to the transaction name, and each table may be processed in parallel, but the concurrent line is controlled to be S, which may be configured by a parameter. S is an integer, and S is greater than or equal to 2 and less than or equal to N if the transaction involves N data tables.

In operation S507, it is determined whether there is any table that needs to be processed. If not, return to operation S506. If so, processing continues with each table.

In operation S508, data is queried from the source database and the target database, respectively, using SQL-A.

In operation S509, the data is obtained and compared to ensure that the database correlation constraint is not violated during subsequent SQL-B replay of the executed DML.

In operation S510, it is determined whether a target database related constraint is violated. If so, the process of the obtained data is abandoned and returns to operation S507. If not, operation S511 continues.

In operation S511, all field data in SQL-B is assembled to form a simple DML statement, which is executed and submitted in the target database. After submission, the processed message is submitted to the distributed message queue 102 and returned for the next round of message acquisition.

In this way, according to the embodiment of the present disclosure, an asynchronous trigger mechanism is used in the application layer, a database trigger is not used, the original data is obtained by querying the original database using a simple select statement, and the performance of the original database and the target database is less affected by performing add/delete modification operations on the target database using a simple DML statement. And moreover, the characteristics of message queue decoupling, asynchronization and peak clipping are utilized to ensure that the availability of external services of clusters on two sides for data synchronization does not depend on each other.

Fig. 6 schematically shows a block diagram of a data synchronization apparatus 600 according to an embodiment of the present disclosure.

As shown in fig. 6, the database synchronization apparatus 600 may include a request acquisition module 610, a synchronization message acquisition module 620, a push module 630, a subscription module 640, and a data synchronization module 650. According to other embodiments of the present disclosure, the data synchronization apparatus 600 may further include a setting module 660. The data synchronization apparatus 600 may be used to implement the methods described with reference to fig. 2-5.

The request obtaining module 610 is configured to obtain a user request in response to a user request for data operation received by any one of M databases, where M is an integer greater than or equal to 1.

The synchronization message obtaining module 620 is configured to obtain a data synchronization message based on the user request, where the data synchronization message includes identification information indicating data operated by the user request and direction information indicating a synchronization direction of the data between the databases.

The pushing module 630 is configured to push the data synchronization message to the distributed message queue.

The subscription module 640 is used for subscribing data synchronization messages from the distributed message queue.

The data synchronization module 650 is configured to, after subscribing to the data synchronization message, obtain the complete data indicated by the identification information from the source database indicated by the direction information, and write the complete data into the target database indicated by the direction information.

The setup module 660 is configured to: presetting information of N data tables related to the transaction indicated by the transaction name; presetting query statements for acquiring data manipulation results from each data table in a source database and a target database; and presetting a replay execution statement of the target database, wherein the replay execution statement is used for manipulating data in the target database.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any plurality of the request acquisition module 610, the synchronization message acquisition module 620, the push module 630, the subscription module 640, the data synchronization module 650, and the setting module 660 may be combined and implemented in one module, or any one of them may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the request obtaining module 610, the synchronization message obtaining module 620, the pushing module 630, the subscribing module 640, the data synchronizing module 650, and the setting module 660 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware, and firmware, or by a suitable combination of any several of them. Alternatively, at least one of the request obtaining module 610, the synchronization message obtaining module 620, the pushing module 630, the subscribing module 640, the data synchronizing module 650, and the setting module 660 may be at least partially implemented as a computer program module that, when executed, may perform corresponding functions.

Fig. 7 schematically shows a block diagram of an electronic device 700 adapted to implement a data synchronization method according to an embodiment of the present disclosure. The electronic device 700 shown in fig. 7 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 7, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. The processor 701 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 701 may also include on-board memory for caching purposes. The processor 701 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. The processor 701 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 702 and/or the RAM 703. It is noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 700 may also include input/output (I/O) interface 705, which input/output (I/O) interface 705 is also connected to bus 704, according to an embodiment of the present disclosure. The electronic device 700 may also include one or more of the following components connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program, when executed by the processor 701, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 702 and/or the RAM 703 and/or one or more memories other than the ROM 702 and the RAM 703 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being adapted to cause the electronic device to carry out the image recognition method provided by the embodiments of the present disclosure.

The computer program, when executed by the processor 701, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via the communication section 709, and/or installed from the removable medium 711. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A database synchronization method, comprising:

responding to a user request of data operation received by any one of M databases, and acquiring the user request, wherein M is an integer greater than or equal to 1;

obtaining a data synchronization message based on the user request, wherein the data synchronization message comprises identification information used for indicating data operated by the user request and direction information used for indicating a synchronization direction of the data between databases;

pushing the data synchronization message to a distributed message queue;

subscribing to the data synchronization message from the distributed message queue; and

and after subscribing the data synchronization message, acquiring the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information.

2. The method of claim 1, wherein said obtaining a user request for a data operation in response to any one of the M databases receiving the user request comprises:

in response to receiving the user request, identifying a transaction name in the user request, the transaction name indicating a business function performed by the user request; and

and when the transaction name belongs to a preset transaction name set, acquiring the user request, wherein the transaction name set is a preset transaction name set needing data synchronization.

3. The method of claim 2, wherein the data synchronization message further comprises the transaction name, wherein the deriving a data synchronization message based on the user request comprises:

extracting the transaction name and the identification information from the user request;

obtaining the direction information according to the data synchronization configuration of the database operated by the user request; and

and packaging the transaction name, the identification information and the direction information to obtain the data synchronization message.

4. The method of claim 3, wherein the obtaining the complete data indicated by the identification information from the source database indicated by the direction information comprises:

acquiring information of N data tables related to the transaction indicated by the transaction name;

querying data from the N data tables in the source database to obtain the complete data comprises:

acquiring data indicated by the identification information from each data table in the source database by using a query statement for acquiring a data manipulation result from each data table in the source database to obtain local data corresponding to each data table, wherein N data tables correspond to N local data; and

and combining the N local data to obtain the complete data.

5. The method of claim 4, wherein the writing the complete data to the target database indicated by the direction information comprises:

inquiring data from the N data tables in the target database to obtain check data, wherein the check data comprises the following steps:

acquiring data indicated by the identification information from each data table in the target database by using a query statement for acquiring a data manipulation result from each data table in the target database to obtain local check data corresponding to each data table, wherein N data tables correspond to N local check data; and

merging the N local check data to obtain the check data;

comparing the complete data with the check data to obtain a comparison result; and

writing the complete data to the target database when the comparison result indicates that the complete data does not violate the database constraint of the target database.

6. The method according to any one of claims 1 to 5, wherein the writing the complete data into the target database indicated by the direction information comprises:

assembling the replay execution statement for data synchronization in the target database according to all field data in the complete data to form a data manipulation statement; and

and submitting the data manipulation statement to the target database for execution.

7. The method of claim 5, wherein when N is greater than 1, concurrently querying data from S of the data tables, wherein S is an integer and 2 ≦ S ≦ N.

8. The method of claim 5, wherein the method further comprises:

presetting the information of N data tables related to the transaction indicated by the transaction name;

presetting query statements for respectively acquiring data manipulation results from each data table in the source database and the target database; and

presetting a replay execution statement of the target database, the replay execution statement being used to manipulate data in the target database.

9. The method of claim 1, wherein M is greater than 1, and the source database and the target database both belong to M of the databases.

10. A database synchronization system comprising:

the device comprises an interceptor and a control device,

a distributed message queue, and

a data synchronization cluster;

wherein the content of the first and second substances,

the interceptor is configured to:

obtaining a data synchronization message based on the user request, wherein the data synchronization message comprises identification information used for indicating data operated by the user request and direction information used for indicating a synchronization direction of the data between databases; and

pushing the data synchronization message to the distributed message queue;

the distributed message queue is to:

receiving the data synchronization message; and

notifying the data synchronization cluster to consume the data synchronization message;

the data synchronization cluster is to:

11. A database synchronization apparatus, comprising:

the request acquisition module is used for responding to a user request of data operation received by any one of M databases and acquiring the user request, wherein M is an integer greater than or equal to 1;

a synchronous message obtaining module, configured to obtain a data synchronous message based on the user request, where the data synchronous message includes identification information used for indicating data operated by the user request, and direction information used for indicating a synchronous direction of the data between databases;

the pushing module is used for pushing the data synchronization message to a distributed message queue;

a subscription module, configured to subscribe to the data synchronization message from the distributed message queue; and

and the data synchronization module is used for acquiring the complete data indicated by the identification information from the source database indicated by the direction information after subscribing the data synchronization message, and writing the complete data into the target database indicated by the direction information.

12. An electronic device, comprising:

one or more memories storing executable instructions; and

one or more processors executing the executable instructions to implement the method of any one of claims 1-9.

13. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 9.