WO2022134876A1 - Data synchronization method and apparatus, and electronic device and storage medium - Google Patents

Abstract

A data synchronization method and apparatus, and an electronic device and a storage medium. The data synchronization method provided in the present application comprises: acquiring, according to a transaction log of a slice, a transaction sub-log to be replayed (301); detecting, according to a global transaction snapshot table updated according to a preset cycle, whether a global transaction, to which said transaction sub-log belongs, belongs to a submitted global transaction (302), wherein the global transaction snapshot table is configured to record a submission state of the global transaction; and after it is determined that the global transaction, to which said transaction sub-log belongs, belongs to the submitted global transaction, replaying said transaction sub-log to a downstream database (303).

Description

Data synchronization method, device, electronic device, storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202011549237.7 and the filing date of December 24, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is incorporated herein by reference.

technical field

The embodiments of the present application relate to the field of databases, and in particular, to a data synchronization method, apparatus, electronic device, and storage medium.

Background technique

As a transaction-oriented processing system (On-Line Transaction Processing, referred to as "OLTP"), distributed databases need to regularly unload data to analysis systems, such as data warehouses, for subsequent analysis and processing by other systems. There are multiple data shards in a distributed database, and concurrent data synchronization between shards has high performance. When the distributed database unloads the data to the analysis system, it will replay the sub-transaction log of the global transaction generated by each shard to the downstream database. .

However, there is a global transaction across shards in a distributed database, and the global transaction across shards includes multiple sub-transactions. When multiple sub-transactions are executed in multiple data shards, if the global transaction across shards on a shard is Corresponding sub-transactions have generated logs, and some sub-transactions in the cross-shard global transaction have not generated logs, that is, the global transaction may not be committed or to be rolled back. The transaction log will be played back to the downstream database, so that the uncommitted data will also be synchronized to the downstream database, that is, the problem of uncommitted read occurs.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a data synchronization method, apparatus, electronic device, and storage medium.

In view of this, an embodiment of the present application provides a data synchronization method, including: acquiring a sub-transaction log to be played back according to a fragmented transaction log; detecting the sub-transaction log to be played back according to a global transaction snapshot table updated with a preset period Whether the global transaction to which the transaction log belongs is a submitted global transaction; wherein, the global transaction snapshot table is configured to record the submission status of the global transaction; after determining that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction After the global transaction, the sub-transaction log to be played back is played back to the downstream database.

In view of this, an embodiment of the present application also provides a data synchronization device, comprising: a sub-transaction log acquisition module, configured to acquire sub-transaction logs to be played back according to fragmented transaction logs; a submitted global transaction judgment module, according to Detecting whether the global transaction to which the sub-transaction log to be played belongs belongs to the global transaction snapshot table updated at a preset period is a submitted global transaction; wherein, the global transaction snapshot table is configured to record the submission status of the global transaction; The playback module, after determining that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction, plays back the sub-transaction log to be played back to the downstream database.

In view of this, an embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores data that can be used by the at least one processor. Instructions executed by a processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described data synchronization method.

In view of this, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned data synchronization method is implemented.

Description of drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

1 is a schematic diagram of a data synchronization system according to a first embodiment of the present application;

2 is a flowchart of updating a global transaction snapshot table according to the first embodiment of the present application;

3 is a flowchart of a data synchronization method according to the first embodiment of the present application;

4 is a schematic diagram of transaction log analysis according to the first embodiment of the present application;

5 is a flowchart of a data synchronization method according to a second embodiment of the present application;

6 is a flowchart of a data synchronization apparatus according to a third embodiment of the present application;

FIG. 7 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that, in each embodiment of the present application, many technical details are provided for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be realized. The following divisions of the various embodiments are for the convenience of description, and should not constitute any limitation on the specific implementation of the present application, and the various embodiments may be combined with each other and referred to each other on the premise of not contradicting each other.

The first embodiment of the present application relates to a data synchronization method. Referring to FIG. 1 , a data synchronization system that can be used to implement the data synchronization method in the first embodiment of the present application includes: an upstream database, that is, a distributed database 101 , a data synchronization tool 102 , and a downstream database 103 .

The distributed database 101 includes: a global transaction manager, computing nodes, and data shards. Data sharding, which can be referred to as sharding for short. The Global Transaction Manager (GTM, Global Transaction Manager) is configured to create and release Global Transaction Numbers (GTIDs), maintain a list of active global transactions, also known as global transaction snapshots, and active global transactions are distributed A global transaction in the database that is started but not committed. The global transaction manager records the current active global transaction list, and the active global transaction list includes the global transaction numbers (GTIDs) of all active global transactions in the current distributed database. When a global transaction starts, GTM records its GTID into the list; when a global transaction commits, GTM removes the GTID corresponding to the transaction. Compute nodes are configured for global transaction resolution and execution in a distributed database. Data sharding is responsible for local storage and local execution of data, and generates a local transaction log, which records the data rows before and after modification of the table in the shard.

The distributed database 101 supports a replicated table schema. Regarding the replication table mode, a distributed database can distribute the data of a large table to multiple data shards according to certain rules, and each shard stores a part of the data of the table, which is different from each other. In addition, according to the data scale Smaller tables can use the replicated table mode, that is, the complete data of the table is stored on each data shard, and the data on each shard is the same. The replication table update is also a global transaction. When the data in the replication table is updated, each shard is updated successfully together, otherwise all shards are rolled back to the state before the update. The global transaction snapshot table in this embodiment adopts the replication table mode, that is, each fragment in the distributed database has a global transaction snapshot table, and the global transaction snapshot table is configured to represent the commit state of the global transaction. The update of the global transaction snapshot table is equivalent to updating the data of the global transaction snapshot table of each data fragment in the distributed database 101. During the update process, the computing node updates the data on the global transaction manager (GTM) to each data segment. In the global transaction snapshot table of each data shard, for example, the computing node queries the global transaction manager for the current active global transaction list, and updates the currently queried active global transaction list to each data shard.

It is worth mentioning that updating the global transaction snapshot table is also a global transaction of the distributed database. When the global transaction snapshot table in the shard is updated, each shard will generate a sub-subsection corresponding to the global transaction that updates the global transaction snapshot table. The transaction log, the subtransaction log records the value of the record row before modification and the value of the record row after modification. The distributed database 101 updates the global snapshot table at a preset period, and a global snapshot point is generated in the log stream of each shard every preset period. That is to say, the global snapshot point is when the global transaction snapshot table is regularly updated. Log records generated in the log stream of each data shard. When the global snapshot table is updated, each data shard generates a sub-transaction log, which records the data rows before and after the change of the global snapshot table. After the data synchronization tool parses the log record, it can obtain the current global transaction snapshot table. Since the table is a replicated table, the same log records are generated in the log stream of each data shard. Through the information in the log records, synchronization control can be performed between data synchronization processes. The process of generating global snapshot points every preset time period is shown in FIG. 2 .

Step 201, the computing node obtains the current active global transaction list from the global transaction manager according to the configured preset period.

In some instances, the computing node obtains the current active global transaction list from the global transaction manager according to a configured timer, such as a configuration of 5 seconds, that is, a preset period of 5 seconds.

Step 202, the computing node writes the active global transaction list to the global transaction snapshot table of each shard.

Step 203: When the transaction commits, each data shard writes the update log of the global transaction snapshot table to the log replication stream. The update log records the row record value before and the modified row record value of the global transaction snapshot table.

The data synchronization tool 102 includes multiple log collection and playback processes and distributed lock resources. Each log collection and playback process corresponds to each data fragment in the distributed database 101 one-to-one, receives the transaction log of the data fragment, processes it, and transmits the processed SQL statement to the downstream database 103 for playback. The distributed lock resource in the data synchronization tool 102 can be provided by open source software such as ZoomKeeper. Multiple log collection and playback processes compete with the distributed lock resource for the master control node, and the process that obtains the master control authority coordinates other nodes to perform log playback.

It should be noted that the data synchronization tool 102 shown in FIG. 1 may be an independent device, or may be integrated into the distributed database 101 or the downstream database 103 .

The following description will be given by taking the data synchronization method applied to the data synchronization tool 102 as an example. The data synchronization tool may be applied to a server or other electronic devices. The data synchronization method of this embodiment includes: acquiring a sub-transaction log to be played back according to a fragmented transaction log; detecting whether the global transaction to which the sub-transaction log belongs is a submitted global transaction according to a global transaction snapshot table; The global transaction snapshot table is configured to record the commit state of the global transaction; after it is determined that the global transaction to which the sub-transaction log belongs belongs to the committed global transaction, the sub-transaction log is played back to the downstream database. In this embodiment, sub-transactions will be played back to the downstream database only when the global transaction is committed, avoiding the problem of uncommitted reads. In addition, in this embodiment, shards can be granular, and each shard can be replayed according to the global transaction snapshot table. The sub-transaction log facilitates the realization of concurrency between shards and improves the efficiency of data synchronization.

The implementation details of the data synchronization method in this embodiment will be described in detail below with reference to the flowchart of the data synchronization method in the first embodiment of the present application. must.

Step 301: Obtain the sub-transaction log to be played back according to the transaction log of the fragment.

In one example, a process that corresponds one-to-one with each shard in the distributed database is created, and the process is configured to obtain the sub-transaction log to be played back according to the transaction log of the shard. In this implementation, one process corresponds to one shard, and parses the transaction log of the corresponding shard. Multiple processes can be created for multiple shards in the database to parse the transaction log at the same time, which facilitates the realization of concurrency between shards.

In some instances, the data synchronization tool creates log collection and playback processes that correspond one-to-one with each shard in the distributed database, and each log collection and playback process obtains the transaction log of the shard from the corresponding shard node of the distributed database 101 And parse the transaction log of the shard, the sub-transaction log parsed from the transaction log of the shard but not played back is called the sub-transaction log to be played back, and the sub-transaction log to be played back is temporarily stored in the parsing order, and marked The GTID list consisting of the global transaction number GTID to which each sub-transaction log to be played back belongs, and the global transaction number to which all the sub-transaction logs to be played back belong, is called the to-be-played-back list. For example, as shown in Table 1, list A to be played back contains GTID lists 7, 8, 9, 10, 11, 12; list B to be played back contains GTID lists of 4, 5, 6, 7, 8, 9; List C to be played back, including GTID lists 1, 2, 3, 4, and 5.

Table 1

To be played back list	The global transaction number to which the subtransaction log to be played back belongs
C: 7,8,9,10,11,12	log 7, log 8, log 9, log 10, log 11, log 12
B: 4,5,6,7,8,9	log 4, log 5, log 6, log 7, log 8, log 9
A: 1,2,3,4,5	log 1, log 2, log 3, log 4, log 5

Step 302: Detect whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction according to the global transaction snapshot table updated at a preset period.

In one example, the process is configured to detect, according to the global transaction snapshot table, whether the global transaction to which the sub-transaction log to be played belongs belongs to a committed global transaction.

In one example, the global transaction snapshot table includes: an active global transaction list; the active global transaction list records the number of the global transaction in an active state; if the global transaction number of the global transaction to which the sub-transaction log to be played back belongs is in the In the active global transaction list, the global transaction to which the sub-transaction log to be played back belongs does not belong to the submitted global transaction. In this implementation, the number of uncommitted global transactions is determined by comparing with the list of active global transactions.

In one example, the global transaction snapshot table further includes: a global transaction number corresponding to the global transaction that updates the table. Specifically, updating the global transaction snapshot table is a distributed global transaction, and the number of the global transaction is the global transaction number corresponding to the global transaction that updates this table. It is worth mentioning that the distributed database itself has system hidden Column, record the global transaction number that modifies the row of this record, that is to say, the global transaction number corresponding to the global transaction that updates this table is the system column.

In some instances, when the log collection and playback process parses the transaction log of the shard, it also parses the sub-transaction log updated to the global transaction snapshot table, and the GTID value of the sub-transaction log is the global transaction that updates the table. The corresponding global transaction number is used as an identifier of the global transaction snapshot table, and the identifier is configured to uniquely identify the global transaction snapshot table, that is, different identifiers indicate different global transaction snapshot tables. Also identifies the aforementioned global snapshot point. Since the global transaction snapshot table is a replicated table, each log collection and playback process can resolve to the same global transaction snapshot table. The log collection and playback process compares the current list to be played back with the active global transaction list snapshot in the global transaction snapshot table. If a global transaction number appears in the current list to be played back, it also appears in the global transaction snapshot, that is, the active global transaction list. , it indicates that the global transaction is not a committed transaction. If a global transaction number appears in the current to-be-playback list, but does not appear in the global transaction snapshot, that is, the active global transaction list, it indicates that the global transaction is a committed transaction. The sub-transaction log corresponding to the transaction can be played back to the downstream database.

The following is briefly described with reference to specific examples. As shown in FIG. 4 , the circle in FIG. 4 represents the active global transaction list snapshot corresponding to the global snapshot point, and the log represents the parsed list to be played back. The global transaction numbers 1, 2, and 3 appear in the list A to be played back but not in snapshot2, indicating that the three transactions 1, 2, and 3 belong to the global commit, while the global transaction numbers 4 and 5 appear in the list A to be played back. , also appears in snapshot2, indicating that 4 and 5 do not belong to the committed global transaction. By analogy, global transaction numbers 4, 5, and 6 appear in the to-be-playback list B, but not in snapshot3, indicating that the global transaction has been committed before snapshot3 is generated, while global transaction numbers 7, 8, and 9 appear in the to-be-playback list. B also appears in snapshot3, indicating that 7, 8, and 9 do not belong to the committed global transaction. Global transaction numbers 7, 8, 9, and 10 appear in the to-be-playback list C, but not in sanpshot4, indicating that the global transaction has been committed before snapshot4, and global transaction numbers 11 and 12 appear in the to-be-playback list C and also appear in In snapshot4, it indicates that 11 and 12 do not belong to the committed global transaction.

In another example, the global transaction snapshot table further includes: the current maximum committed global transaction number; if the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is not in the active global transaction list, it is determined that Whether the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is greater than the current maximum submitted global transaction number; if the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is greater than the current The maximum committed global transaction number, the global transaction to which the sub-transaction log to be played back belongs does not belong to the submitted global transaction; otherwise, the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction.

That is to say, after determining the uncommitted global transaction through the active global transaction list in the global transaction snapshot table, the global transaction number to which the undetermined global transaction belongs needs to be compared with the current maximum committed global transaction number to determine the The number of the committed global transaction, rather than all global transactions other than uncommitted global transactions identified by the list of active global transactions as committed global transactions. In this implementation, it is avoided that the global transaction number of the global transaction to which the sub-transaction log in the list to be played back belongs is generated after the current global transaction snapshot table and is not recorded in the global snapshot table, so that the judgment of the submitted global transaction is avoided. more precise. If the global transaction number is numbered in the chronological order of the global transaction, the global transaction label of the global transaction generated after the global transaction snapshot table is larger than any number that appears in the global transaction snapshot table, that is, it is larger than the current maximum committed number. The global transaction number is large.

In some instances, the current maximum committed global transaction number in the global transaction snapshot table is 20, and after updating the global transaction snapshot table, a global transaction with the global transaction number 21 is generated, and the global transaction is also in an active state, That is, the uncommitted state, but because it is generated after the global transaction snapshot table, it will not be in the active global transaction list of the global transaction snapshot table. In order to avoid determining the global transaction of global transaction number 21 as a committed global transaction, Comparing the global transaction number with the largest committed global transaction number, 21 is greater than the largest committed global transaction number 20, then the global transaction corresponding to the global transaction number 21 is an uncommitted global transaction. Table 2 below is a schematic diagram of the table structure of the global snapshot table including the current maximum committed global transaction number.

Table 2

field	type	illustrate
GTID	INT8	System column, the global transaction number corresponding to the global transaction that updates this table
GTIDLIST	BLOB	List of active global transactions
MAXGTID	INT8	The current maximum committed global transaction number

The data in the above table is small in scale. The table can have only one record, and the replication table mode will not occupy too much hardware and software resources. The table structure of the global transaction snapshot table is shown in Table 2. Combined with Table 2, determine one The rules for whether a global transaction is a global commit is as follows. In addition, in order to distinguish it from the GTID that appears in the table, the global transaction number to be judged is recorded as GTID_1:

If GTID_1 is found in GTIDLIST, the global transaction identified by GTID_1 is active, not globally committed, and does not belong to a globally committed transaction; otherwise, if GTID_1 is greater than MAXGTID, that is, the current maximum committed transaction number, then the global transaction identified by GTID_1 It is active and not submitted globally; otherwise, if GTID_1 is less than or equal to MAXGTID, the GTID_1 has been submitted globally.

In addition, the global transaction snapshot table is updated at a preset cycle. For example, in the above example, it is updated every 5 seconds. Every 5 seconds, the computing node will query the global transaction manager for the current GTIDLIST and MAXGTID, and use the queried GTIDLIST and MAXGTID pair. The global transaction snapshot table is updated, and the global transaction number of the global transaction of the global snapshot table is updated, that is, the GTID in the global update snapshot table is also updated, and the updated global transaction snapshot table is updated. The row record value of the shard is recorded in the transaction log of the shard. Therefore, an updated global transaction snapshot table is parsed from the transaction log of the shard at regular intervals. For example, according to the global transaction snapshot updated at a preset period When the table detects whether the global transaction to which the sub-transaction log to be replayed belongs is a committed global transaction, the field GTID used is the 30 global transaction snapshot table. The global transaction snapshot table, at this point, the step of detecting whether the global transaction to which the sub-transaction log to be played belongs belongs to the global transaction to which the sub-transaction log belongs is a submitted global transaction has been completed according to the global transaction snapshot table whose field GTID is 30. Next, according to the field whose GTID is 40 The global transaction snapshot table detects whether the global transaction to which the sub-transaction log to be replayed belongs is a committed global transaction.

Step 303: After it is determined that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction, the sub-transaction log to be played back is played back to the downstream database.

In one example, it is confirmed that the processes corresponding to each shard in the distributed database have completed detection based on the same global transaction snapshot table; The target sub-transaction log is used as the target sub-transaction log; the target sub-transaction log is played back to the downstream database. In this implementation, each shard will synchronously replay the sub-transactions corresponding to the submitted global transactions detected according to the same global transaction snapshot table, so as to avoid the problem of inconsistent playback data among shards, and make the data synchronization progress between shards the same. For example: synchronizing data to a downstream database.

It should be noted that as long as the global transaction snapshot table identifier can be configured to indicate a global transaction snapshot table, it can be the update time of the global transaction snapshot table or the value of the field GTID in the global transaction snapshot table. There are no restrictions. In this embodiment, the global transaction snapshot table is identified as the value of the field GTID in the global transaction snapshot table as an example.

In some instances, the log collection and playback process performs detection based on a global transaction snapshot table. After the detection is completed, a detection completion notification is sent. The detection completion notification includes the global transaction snapshot table identifier corresponding to the global transaction snapshot table. The tool creates a module that is configured to obtain the detection completion notification sent by each log collection and playback process, and confirms that the processes corresponding to each shard in the distributed database have completed detection based on the same global transaction snapshot table according to the detection completion notification. , and then issue a playback command to each log collection and playback process. The playback command includes the above-mentioned global transaction snapshot table identifier of the same global transaction snapshot table. Each log collection and playback process will use the same global transaction snapshot table identifier to indicate The detected subtransaction logs belonging to committed global transactions are played back to the downstream database. For example, according to the global transaction snapshot table whose GTID is 30 in the global transaction snapshot table, it is detected that the submitted global transaction has been completed. The sub-transaction log corresponding to the committed global transaction determined by the table is played back to the downstream database, and the committed global transaction determined according to the global transaction snapshot table with GTID of 30 is a cross-shard transaction. At the same time, when the data is changed, data shard 1 generates sub-transaction A, and data shard 2 generates sub-transaction B. Process 1 synchronizes data fragment 1, parses and plays back the sub-transaction A log, and process 2 synchronizes data fragment 2, parses and plays back the sub-transaction B log. In this embodiment, each log collection and playback process will respond to the playback command, and data fragment 1 will Sub-transaction A log, data shard 2 will synchronously replay the sub-transaction B log to the downstream database, so that the data generated by each shard can be played back concurrently, avoiding that when each shard is played back, it belongs to the same submitted global transaction. The problem of inconsistent progress of sub-transaction log playback to the downstream database, for example, the sub-transaction A log is played back to the downstream database, but the sub-transaction B log is not played back to the downstream database, that is to say, the problem of inconsistent reading is solved and data synchronization between shards The progress is similar, so as to avoid the dirty read of the downstream database caused by the difference of the sub-transaction data synchronization progress.

In some cases, in order to solve the problem of inconsistent reading, a method is also provided, which sorts the global transactions in order, and replays multiple sub-transactions belonging to the same global transaction to the downstream together. Sorting, and monitoring the sub-transactions corresponding to the global transaction of each shard, to determine whether the global transaction has been committed, the performance overhead is high, and the synchronization efficiency is low. In this embodiment, the sub-transaction log to be played back is obtained according to the transaction log of the shard, and each shard detects the global transaction to which the sub-transaction log belongs according to the global transaction snapshot table updated at a preset period, whether it belongs to the submitted global transaction, and then determines whether the sub-transaction log belongs to the global transaction. After the global transaction to which the sub-transaction log belongs belongs to the submitted global transaction, the sub-transaction log is played back to the downstream database, and the transaction can be operated in parallel with the transaction log generated by the fragmentation as the granularity, and the error between the fragments is controlled within seconds level, there is no need to sort global transactions, and the performance overhead is small, enabling each shard to synchronize data in parallel between shards, improving synchronization efficiency, and ensuring that only committed global transactions will be synchronized to the downstream database. , the sub-transaction log that belongs to the submitted global transaction detected according to the same global transaction snapshot table is used as the target sub-transaction log, and the target sub-transaction is played back, so that the data read by the upstream database and the downstream database are kept consistent in real time, that is, each Each data shard synchronizes data to the downstream database in parallel, and the performance is close to concurrent synchronization, which can ensure the consistency of data synchronization between shards.

The second embodiment of the present application relates to a data synchronization method. The second embodiment is roughly the same as the first embodiment, the main difference is that: in the first embodiment, the data synchronization tool creates a command sending module to send the playback command to the log collection and playback process. In the second embodiment of the present application, in addition to performing the above steps 301 to 303, each log collection and playback process also needs to preempt the master control authority from the distributed lock resource, and the process that preempts the master control authority obtains the detection of each process. Notification, the global transaction snapshot table identifier in the playback command is determined according to the detection completion notification.

A flowchart of the data synchronization method in the second embodiment of the present application is shown in FIG. 5 .

Step 501: At startup time, preempt the master control authority from the distributed lock resource. Specifically, when the log collection and playback process starts, it grabs the master control permission from the distributed lock, and when the master control permission is obtained, it becomes the master control process. If the master control permission is not occupied, it can seize the master control permission from the distributed lock resource. .

In one example, if the preemption of the master control authority succeeds, the process with the master control authority executes and receives the detection completion notices respectively sent by the processes corresponding to other shards; The process that successfully preempts the control authority sends a detection completion notification, where the detection completion notification carries the identifier of the global transaction snapshot table for which detection has been completed.

In an example, in the case of successfully preempting the master control authority, the process with the master control authority notifies the processes corresponding to other shards in the distributed database to detect the transactions belonging to the submitted transactions according to the same global transaction snapshot table. The sub-transaction log of the global transaction is used as the target sub-transaction log to be played back to the downstream database.

In an example, in the case of successfully preempting the master control authority, receiving detection completion notices respectively sent by processes corresponding to other shards in the distributed database, and the detection completion notice carries the data of the same global transaction snapshot table. logo.

In the following, the log collection and playback process 1 preempts the master control authority, and the log collection and playback process m does not preempt the master control authority as an example.

Step 502, when the master control authority is not preempted, query the current master control process. Specifically, the log collection and playback process m that has not preempted the master control authority queries the process that has preempted the master control authority, so as to exchange information with the master control authority, that is, to send a completion notification to the master control authority and receive playback commands. The playback command is configured to notify the processes corresponding to other shards in the distributed database to use the sub-transaction logs belonging to the submitted global transactions detected according to the same global transaction snapshot table as the target sub-transaction logs, and playback to the downstream database.

Step 503, parse the transaction log of the fragment. This step is to obtain the submitted global transaction currently to be played back according to steps 301 to 302 in the first embodiment of the present application.

Step 504, sending a detection completion notification.

Following the above example, if the log collection and playback process m detects that the submitted global transaction has been completed according to the global transaction snapshot table whose GTI D is 30 in the global transaction snapshot table, it sends a completion notification to the log collection and playback process 1. The completion notification includes: GTI D is 30.

Step 505, sending a playback command.

Following the above example, the log collection and playback process determines the global transaction snapshot table identifier of the playback command according to the global transaction snapshot table identifier in the sending completion notification of each process. For example, the master control process receives all processes according to the global transaction snapshot table. The global transaction snapshot table of 30 detects that the submitted global transaction has been completed, and the master process, that is, the log collection and playback process 1, sends a playback command, and the playback command includes a GTID of 30.

Step 506: Play back the target sub-transaction log to the downstream database.

Following the above example, when other processes receive the command, they will play back the target sub-transaction log corresponding to the committed global transaction detected according to the global transaction snapshot table whose GTID is 30 in the global transaction snapshot table to the downstream database, and the master process will also According to the generated playback command, the target sub-transaction log is played back to the downstream database. After all processes play back the target sub-transaction log to the downstream database, the detection based on the global transaction snapshot table whose GTID is 30 in the global transaction snapshot table is realized. The committed global transaction is played back to the downstream database.

In one example, if the process that has preempted the master control permission fails, the master control permission is released. Specifically, when there is a problem with the running of the process that has preempted the master control authority, the master control authority is released, and other processes can preempt the master control authority from the distributed lock resource again. It avoids abnormal data synchronization in the upstream database and the downstream database caused by process failure, and improves the robustness.

That is to say, the master process is the coordinator of multiple log collection and playback processes, and is responsible for receiving playback progress notifications from other processes; when all processes have finished parsing a log snapshot point, it commands all processes to playback to the downstream database. Through the above notification-command mechanism, the master control process can coordinate multiple log collection and playback processes to play back the global transaction at the same time, which solves the problem that some process delays lead to the increasing deviation of playback progress between processes. The data is in a globally consistent state. The master control process itself also replicates the log collection and playback work. If the master control node fails, other processes can regain the master and continue to work, which also improves the robustness.

The step division of the above various methods is only for the purpose of describing clearly, and can be combined into one step or split some steps during implementation, and decomposed into multiple steps, as long as the same logical relationship is included, all within the protection scope of the present application ; Adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of the algorithm and process are within the protection scope of this application.

The third embodiment of the present application relates to a data synchronization apparatus. As shown in FIG. 6 , it includes: a sub-transaction log acquisition module 601, which is configured to acquire sub-transaction logs to be played back according to fragmented transaction logs; submitted global transaction detection Module 602 is configured to detect whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction according to the global transaction snapshot table updated with a preset period; wherein, the global transaction snapshot table is configured as Record the commit status of the global transaction; the playback module 603 is configured to play back the sub-transaction log to be played back to the downstream database after determining that the global transaction to which the sub-transaction log to be played belongs belongs to the committed global transaction.

The sub-transaction log obtaining module 601 is further configured to create a process that corresponds to the fragmentation one-to-one; the process is configured to execute the obtaining of the sub-transaction log to be played back according to the transaction log of the fragment, the The snapshot table detects whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction, and after determining that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction, The sub-transaction log to be played back is played back to the downstream database.

The playback module 603 is further configured to confirm that the processes corresponding to each fragment in the distributed database have completed detection based on the same global transaction snapshot table; the playback of the sub-transaction log to be played back to the downstream database includes: : Use the sub-transaction log to be played back belonging to the submitted global transaction detected according to the same global transaction snapshot table as the target sub-transaction log; playback the target sub-transaction log to the downstream database.

The playback module 603 is further configured to notify the processes corresponding to other shards in the distributed database to use the sub-transaction log to be played back belonging to the submitted global transaction detected according to the same global transaction snapshot table as the target sub-transaction log, Playback to the downstream database.

The playback module 603 is further configured to receive detection completion notifications respectively sent by processes corresponding to other shards in the distributed database, where the detection completion notifications carry the identifier of the same global transaction snapshot table.

The playback module 603 is further configured to preempt the master control authority; if the master control authority is preempted successfully, execute the detection completion notification respectively sent by the processes corresponding to the other shards; if the master control authority is preempted If it fails, a detection completion notification is sent to the process that has successfully preempted the master control authority, and the detection completion notification carries the identifier of the global transaction snapshot table for which detection has been completed.

The submitted global transaction detection module 602 is further configured to: if the global transaction number of the global transaction to which the sub-transaction log to be played back belongs is in the active global transaction list, then the global transaction to which the sub-transaction log to be played back belongs Does not belong to a committed global transaction; wherein, the global transaction snapshot table includes: an active global transaction list; the active global transaction list records the number of the active global transaction.

The submitted global transaction detection module 602 is further configured to determine the global transaction to which the sub-transaction log to be played belongs if the global transaction number of the global transaction to which the sub-transaction log to be played belongs is not in the active global transaction list. Whether the global transaction number of the transaction is greater than the current maximum committed global transaction number; if the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is greater than the current maximum committed global transaction number, then the The global transaction to which the sub-transaction log to be replayed belongs does not belong to the submitted global transaction; otherwise, the global transaction to which the sub-transaction log to be replayed belongs belongs to the submitted global transaction; wherein, the global transaction snapshot table further includes: The current largest committed global transaction number.

It is not difficult to find that this embodiment is a system example corresponding to the first embodiment, and this embodiment can be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in this embodiment, and are not repeated here in order to reduce repetition. Correspondingly, the related technical details mentioned in this embodiment can also be applied to the first embodiment.

It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit may be a physical unit, a part of a physical unit, or multiple physical units. A composite implementation of the unit. In addition, in order to highlight the innovative part of the present application, this embodiment does not introduce units that are not closely related to solving the technical problem raised by the present application, but this does not mean that there are no other units in this embodiment.

The fourth embodiment of the present application relates to an electronic device, as shown in FIG. 7 , comprising at least one processor 701 ; and a memory 702 communicatively connected to the at least one processor; wherein the memory stores data that can be instructions executed by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can execute the above-mentioned data synchronization method.

The memory and the processor are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors and various circuits of the memory. The bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor.

The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Instead, memory may be used to store data used by the processor in performing operations.

The fifth embodiment of the present application relates to a computer-readable storage medium storing a computer program. The above method embodiments are implemented when the computer program is executed by the processor.

That is, those skilled in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device ( It may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

In the data synchronization method proposed in the present application, the sub-transaction log to be played back is obtained according to the fragmented transaction log; and whether the global transaction to which the sub-transaction log to be played belongs is detected according to the global transaction snapshot table updated at a preset period, whether it belongs to the The submitted global transaction; wherein, the global transaction snapshot table is configured to record the submission status of the global transaction; after determining that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction, the to-be-played The sub-transaction log is played back to the downstream database, avoiding the problem of synchronizing uncommitted data to the downstream database. In addition, this embodiment uses sharding as the granularity, and the shards in the distributed database determine the sub-transactions that can be played back according to the global transaction snapshot table. The transaction log facilitates the realization of concurrency between shards and improves the efficiency of data synchronization.

Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific examples for realizing the present application, and in practical applications, various changes can be made in form and details without departing from the spirit and the spirit of the present application. scope.

Claims (10)

1. A data synchronization method, comprising:

   Obtain the sub-transaction log to be played back according to the transaction log of the shard;

   Detecting whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction according to the global transaction snapshot table updated with a preset period; wherein, the global transaction snapshot table is configured to record the submission status of the global transaction;

   After it is determined that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction, the sub-transaction log to be played back is played back to the downstream database.
2. The data synchronization method according to claim 1, wherein before acquiring the sub-transaction log to be played back according to the transaction log of the fragmentation, the method comprises:

   Create a process that corresponds to each shard in the distributed database one-to-one; the process is configured to execute the acquisition of the sub-transaction log to be played back according to the transaction log of the shard, and the detection of the to-be-played log according to the global transaction snapshot table. Whether the global transaction to which the replayed sub-transaction log belongs is a submitted global transaction, and after determining that the global transaction to which the sub-transaction log to be replayed belongs belongs to the submitted global transaction, The transaction log is played back to the downstream database;

   After it is determined that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction, and before the sub-transaction log to be played back is played back to the downstream database, the method further includes:

   Confirm that the processes corresponding to each fragment in the distributed database have completed detection based on the same global transaction snapshot table;

   The replaying the sub-transaction log to be replayed to the downstream database includes:

   Taking the sub-transaction log that belongs to the submitted global transaction detected according to the same global transaction snapshot table as the target sub-transaction log;

   The target sub-transaction log is played back to the downstream database.
3. The data synchronization method according to claim 2, wherein after confirming that the processes corresponding to each fragment in the distributed database have completed the detection based on the same global transaction snapshot table, the method further comprises:

   Notifying the processes corresponding to other shards in the distributed database to play back the sub-transaction logs belonging to the committed global transactions detected according to the same global transaction snapshot table to the downstream database as target sub-transaction logs.
4. The data synchronization method according to claim 2, wherein the confirming that the processes corresponding to each fragment in the distributed database have completed detection based on the same global transaction snapshot table, comprising:

   Receive detection completion notifications respectively sent by processes corresponding to other shards in the distributed database, where the detection completion notification carries the identifier of the same global transaction snapshot table.
5. The data synchronization method according to claim 4, wherein before the receiving the detection completion notification respectively sent by the processes corresponding to the other shards, the method further comprises:

   Seize the master control authority;

   If the preemption of the master control authority succeeds, executing the detection completion notification respectively sent by the processes corresponding to the other shards;

   If the master control authority preemption fails, a detection completion notification is sent to the process for which the master control authorization preemption succeeds, and the detection completion notification carries the identifier of the global transaction snapshot table for which detection has been completed.
6. The data synchronization method according to any one of claims 1 to 5, wherein the global transaction snapshot table comprises: an active global transaction list; the active global transaction list records the number of the global transaction in an active state;

   Detecting whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction according to the global transaction snapshot table includes:

   If the global transaction number of the global transaction to which the sub-transaction log to be played back belongs is in the active global transaction list, the global transaction to which the sub-transaction log to be played back belongs does not belong to the committed global transaction.
7. [Corrected 18.11.2021 according to Rule 26]
   

   The data synchronization method according to claim 6, wherein the global transaction snapshot table further comprises: the current maximum submitted global transaction number;

   Detecting, according to the global transaction snapshot table, whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction, further comprising:

   If the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is not in the active global transaction list, determine whether the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is greater than the current maximum the committed global transaction number;

   If the global transaction number of the global transaction to which the sub-transaction log to be replayed belongs is greater than the current maximum submitted global transaction number, then the global transaction to which the sub-transaction log to be replayed belongs does not belong to the submitted global transaction;

   Otherwise, the global transaction to which the sub-transaction log to be played back belongs belongs to the submitted global transaction.
8. [Corrected 18.11.2021 according to Rule 26]
   

   8. A data synchronization device, comprising:

   The sub-transaction log obtaining module is configured to obtain the sub-transaction log to be played back according to the fragmented transaction log;

   The submitted global transaction detection module is configured to detect whether the global transaction to which the sub-transaction log to be played belongs belongs to a submitted global transaction according to the global transaction snapshot table updated with a preset period; wherein, the global transaction snapshot The table is configured to record the commit status of global transactions;

   The playback module is configured to play back the sub-transaction log to be played back to the downstream database after determining that the global transaction to which the sub-transaction log to be played belongs belongs to the submitted global transaction.
9. An electronic device comprising:

   at least one processor; and,

   a memory communicatively coupled to the at least one processor; wherein,

   The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any one of claims 1 to 7 data synchronization method.
10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the data synchronization method according to any one of claims 1 to 7 is implemented.

Images