WO2020025049A1 - Procédé et appareil de synchronisation de données, hôte de base de données et support de stockage - Google Patents

Procédé et appareil de synchronisation de données, hôte de base de données et support de stockage Download PDF

Info

Publication number
WO2020025049A1
WO2020025049A1 PCT/CN2019/099048 CN2019099048W WO2020025049A1 WO 2020025049 A1 WO2020025049 A1 WO 2020025049A1 CN 2019099048 W CN2019099048 W CN 2019099048W WO 2020025049 A1 WO2020025049 A1 WO 2020025049A1
Authority
WO
WIPO (PCT)
Prior art keywords
transaction
thread
database
host
log
Prior art date
Application number
PCT/CN2019/099048
Other languages
English (en)
Chinese (zh)
Inventor
周亚运
林品
刘志文
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2020025049A1 publication Critical patent/WO2020025049A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/80—Database-specific techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/82—Solving problems relating to consistency

Definitions

  • the present disclosure relates to database processing technologies, for example, to a method, device, database host, and storage medium for data synchronization.
  • the client access layer is composed of multiple applications (Applications, APPs), and supports universal Open Database Connectivity (ODBC) and Java Database Connectivity (JDBC) interfaces. Users use the distributed through the client access layer database;
  • APPs Applications
  • ODBC Open Database Connectivity
  • JDBC Java Database Connectivity
  • the computing node cluster is composed of multiple database middleware (Database Proxy, DBProxy), and Structured Query Language (SQL) statements complete the basic processing and distribution in the computing nodes;
  • the management node consists of multiple components, including a distributed database operation and maintenance management server (Operation, Management, Server, OMM), metadata server (MetaData, Server, MDS), DBProxy management center (ProxyManager, PM), and cluster management center (Cluster Manager, CM), etc., set to manage and guarantee the distributed database system;
  • a distributed database operation and maintenance management server Operation, Management, Server, OMM
  • metadata server MetaData, Server, MDS
  • DBProxy management center ProxyManager, PM
  • cluster management center Cluster Manager, CM
  • GTM Global transaction management
  • IDs global transaction identifiers
  • the data node cluster is composed of multiple database groups (DB-GROUP), and each DB-GROUP is composed of 1 master and 1 standby database (Database, DB). Each data node is set to access data, and its performance directly affects the level of performance. The overall performance of the distributed database;
  • Post-installed middleware monitors, backs up, and restores data nodes
  • Each data node in the data node cluster includes 1 master and 1 standby DB (that is, the database master (DB-M) and database standby machine (DB-S)), which are set to protect the data of the database when the main database is down Consistency and high availability characteristics of external services.
  • DB-M database master
  • DB-S database standby machine
  • the embodiments of the present disclosure provide a data synchronization method, a device, a database host, and a storage medium to improve the synchronization performance of the primary and secondary databases.
  • an embodiment of the present disclosure provides a data synchronization method, including:
  • the database host receives a transaction submission request
  • a host transaction thread in the database host writes a log to a disk file according to the transaction submission request, instructs a log forwarding thread to send the log to a database standby machine, and the host transaction thread exits;
  • the database host receives the acknowledgement ACK response sent by the database standby machine, and selects a thread from the thread pool to perform a transaction commit operation.
  • an embodiment of the present disclosure further provides an apparatus for data synchronization, including:
  • a first receiving module configured to receive a transaction submission request
  • a log module configured to control a host transaction thread in a database host to write a log to a disk file according to the transaction submission request, instruct a log forwarding thread to send the log to a database standby machine, and control the host transaction thread to exit;
  • a second receiving module configured to receive an acknowledgement ACK response sent by the database standby machine
  • the transaction commit module is set to select a thread from the thread pool to perform a transaction commit operation.
  • the embodiment of the present disclosure further provides a database host, including a memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor implements the program when the processor executes the program. Method of data synchronization.
  • an embodiment of the present disclosure further provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are used to perform the data synchronization method.
  • Figure 1 is a schematic diagram of the overall architecture of a distributed database
  • Figure 2 is a flowchart of thread synchronization waiting
  • FIG. 3 is a flowchart of a data synchronization method according to an embodiment of the present disclosure
  • step 4 is a flowchart of step 220 in a data synchronization method according to an embodiment of the present disclosure
  • step 230 is a flowchart of step 230 in a data synchronization method according to an embodiment of the present disclosure
  • FIG. 6 is a flowchart of step 401 in a data synchronization method according to an embodiment of the present disclosure
  • step 402 in the data synchronization method according to an embodiment of the present disclosure
  • FIG. 8 is a flowchart of another data synchronization method according to an embodiment of the present disclosure.
  • FIG. 9 is a flowchart of a data synchronization method according to application example one of the present disclosure.
  • FIG. 10 is a timing diagram of a data synchronization method according to application example 1 of the present disclosure.
  • FIG. 11 is a flowchart of a data synchronization method according to application example two of the present disclosure.
  • FIG. 12 is a flowchart of a data synchronization method according to application example three of the present disclosure.
  • FIG. 13 is a flowchart of a data synchronization method according to application example 4 of the present disclosure.
  • FIG. 14 is a schematic diagram of a composition of a data synchronization device according to an embodiment of the present disclosure.
  • FIG. 15 is a schematic diagram of a database host according to an embodiment of the present disclosure.
  • the synchronization technology of thread synchronization waiting is a mainstream technology of database master and backup data node synchronization.
  • the flowchart is shown in FIG. 2 and includes steps 101 to 111.
  • Step 101 The user initiates a transaction submission request.
  • step 102 the database host receives a transaction submission request and starts a thread for processing.
  • Step 103 The host transaction thread writes a log to a disk file.
  • Step 104 The host transaction thread wakes up a log forwarding (DUMP) thread.
  • DUMP log forwarding
  • Step 105 The log DUMP thread starts sending logs to the database standby machine.
  • step 106 the host transaction thread is in a state of waiting for a database standby (Acknowledgement, ACK) response.
  • ACK Acknowledgement
  • step 107 the database standby machine receives a corresponding log.
  • Step 108 After receiving the corresponding logs, the database standby machine returns an ACK response to the database host, indicating that all log files have been received.
  • Step 109 After receiving the database standby ACK response, the database host ACK thread wakes up the host transaction thread.
  • step 110 the database host transaction thread is woken up and continues to complete the final transaction commit operation.
  • Step 111 Return the transaction submission success information to the client.
  • the synchronous waiting standby ACK response is modified to the asynchronous waiting standby ACK response, and the transaction thread is released for processing the remaining tasks to improve the overall performance.
  • the method for data synchronization in the embodiment of the present disclosure includes steps 210 to 230.
  • Step 210 The database host receives a transaction submission request.
  • the user initiates a transaction submission request
  • the database host receives the transaction submission request sent by the user.
  • the database host selects a thread from the thread pool as the host transaction thread.
  • Step 220 The host transaction thread in the database host writes a log to a disk file according to the transaction submission request, instructs the log forwarding thread to send the log to a database standby machine, and the host transaction thread exits.
  • the host transaction thread exits and returns to the thread pool.
  • the host transaction thread is released without synchronously waiting for the ACK response from the standby machine, and the released host transaction thread is set to process the remaining tasks.
  • Step 230 The database host receives a confirmation ACK response sent by the database standby machine, and selects a thread from the thread pool to perform a transaction commit operation.
  • step 220 includes steps 301 to 304.
  • Step 301 The host transaction thread in the database host writes a log to a disk file according to the transaction submission request.
  • Step 302 The host transaction thread instructs the log forwarding thread to send the log to a database standby machine.
  • the host transaction thread wakes up the log forwarding thread, and the log forwarding thread sends the log to the database standby machine.
  • Step 303 The host transaction thread saves the transaction information corresponding to the transaction submission request in a transaction cache list.
  • the transaction information in the transaction cache list is associated with a log location.
  • the transaction information may include connection information of the transaction, execution statement information of the transaction, log position information of the transaction, and group submission information of the transaction.
  • the transaction information is associated with the log position of the submitted transaction and stored in the transaction cache list.
  • the log position of the transaction information cache list may be sequentially increased. Later transaction information is placed at the end of the list, and the search starts from the beginning of the list.
  • Step 304 The host transaction thread exits.
  • the host transaction thread After the host transaction thread completes the DUMP log notification and transaction information cache, it immediately exits and returns the thread pool. It adopts a thread asynchronous waiting mechanism for the standby ACK response mechanism to release the host transaction thread. The released host transaction thread is set to handle the remaining tasks.
  • step 230 includes steps 401 to 402.
  • Step 401 The database host receives an ACK response sent by the database standby machine, and adds a transaction corresponding to the ACK response to a transaction submission task queue.
  • the ACK monitoring thread or the log forwarding thread may be responsible for monitoring the ACK response.
  • Step 402 The database host selects a thread from the thread pool to preferentially perform a transaction commit operation on a transaction corresponding to the ACK response.
  • the timeliness of transaction submission is ensured by preferentially performing a transaction submission operation on the transaction.
  • step 401 includes steps 501 to 503.
  • Step 501 An ACK monitoring thread or a log forwarding thread in the database host monitors an ACK response.
  • different types of threads can be used to listen to the ACK response.
  • it can be a dedicated ACK listening thread or a log forwarding thread.
  • Step 502 The ACK monitoring thread or the log forwarding thread searches for transaction information corresponding to the log position in the transaction cache list according to the log position carried in the ACK response.
  • the cached transaction information to be processed can be found according to the log position of the ACK response.
  • the transaction cache list may be in the form of a linked list.
  • Step 503 The ACK monitoring thread or the log forwarding thread adds the transaction corresponding to the ACK response to the transaction submission task queue according to the transaction information.
  • the transaction submission task queue includes transactions to be submitted in order.
  • step 402 includes steps 601 to 602.
  • Step 601 When a super (SUPER) thread group in the database host detects that there are pending tasks in a transaction submission task queue, preferentially select a SUPER thread in a SUPER thread group in a thread pool corresponding to the ACK response. A transaction performs a transaction commit operation.
  • SUPER super
  • the SUPER thread group listens to a task in the transaction submission task queue
  • the SUPER thread is allocated to process the final transaction submission operation.
  • the threads in the SUPER thread group are high-priority threads to ensure the timeliness of transaction submission.
  • Step 602 In response to no SUPER threads in the SUPER thread group, allocating ordinary threads in the thread pool to preferentially perform a transaction commit operation on a transaction corresponding to the ACK response.
  • the common thread is another thread in the thread pool except the SUPER thread group.
  • the threads in the thread pool will also preferentially process the final transaction submission operation in the transaction submission queue, and perform the second-level priority scheduling for the final transaction submission operation. Ensure that the final transaction commit operation completes first.
  • the embodiment of the present disclosure adopts an asynchronous waiting standby ACK response mechanism and a transaction submission priority mechanism to ensure the consistency of data replicated by the active and standby data nodes of the database and improve the synchronization performance of the active and standby data nodes of the database.
  • a flowchart of another data synchronization method according to an embodiment of the present disclosure includes:
  • Step 701 The user initiates a transaction submission request.
  • step 702 the database host receives a transaction submission request, selects a thread from the thread pool as the host transaction thread, and starts executing a transaction submission operation.
  • Step 703 The database host transaction thread writes a log to a disk file according to the transaction submission request.
  • step 704 the database host transaction thread wakes up the log DUMP thread, and the log DUMP thread starts executing step 707.
  • Step 705 The database host transaction thread saves the transaction information in the transaction cache list.
  • Step 706 The database host transaction thread exits and returns the thread pool.
  • Step 707 The log DUMP thread sends a log to the database standby machine.
  • Step 708 The database standby machine receives the corresponding log.
  • step 709 the database standby machine finishes receiving the corresponding logs and returns an ACK response to the database host, indicating that all log files have been received.
  • Step 710 After the database host receives the database standby ACK response, it adds the transaction corresponding to the ACK response to the transaction submission task queue.
  • step 711 the host SUPER thread group detects that there is a task that needs to be executed, and determines whether there are idle threads in the SUPER thread group. If so, step 713 is performed, and if not, step 714 is performed.
  • Step 713 The SUPER thread in the SUPER thread group is preferentially used to complete the final transaction commit operation, and step 715 is performed.
  • Step 714 In the case that there is no idle SUPER thread in the SUPER thread group, use a thread in a thread pool other than the SUPER thread to complete the final transaction submission operation.
  • step 715 after the host transaction submission is completed, the transaction submission success information is returned to the client.
  • step 705 and step 707 are both executed after step 704.
  • a SUPER thread group in a thread pool is introduced, a transaction priority is defined, and a transaction submission processing mechanism is optimized; a synchronous waiting standby ACK response is modified to an asynchronous waiting standby ACK response, and a transaction thread is released for processing the rest Tasks to improve overall performance.
  • Database performance testing tools such as benchmark-sql, sysbench, etc. are used to compare and optimize the optimized version and the unoptimized version of the embodiment of the present disclosure. At the highest performance, the optimized version of the embodiment of the present disclosure can be improved relative to the unoptimized version About 10%.
  • the database host is referred to as the host
  • the database standby is referred to as the standby
  • the host transaction thread is referred to as the transaction thread.
  • FIG. 9 A flowchart of a data synchronization method according to application example 1 of the present disclosure is shown in FIG. 9 and includes:
  • step 801 the user initiates a transaction submission request on the MySQL client.
  • step 802 the MySQL host receives a transaction submission request, takes a thread from the thread pool, and processes the received transaction submission request.
  • step 803 the transaction thread processes the transaction submission request, and flush && sync the BINLOG log to disk to ensure that the host data can be recovered in various scenarios.
  • flush indicates that the disk file is flushed from the cache to the memory once and is not actually updated to the disk file.
  • sync indicates that the memory of the disk file is synchronized to the file, which is a real disk operation.
  • the BINLOG log is a MySQL log in binary format.
  • step 804 the transaction thread wakes up the independent DUMP thread to send the BINLOG log of the transaction, and the DUMP thread sends the corresponding BINLOG event to the standby machine according to the BINLOG position of the transaction.
  • Step 805 The transaction thread saves the transaction information in the TranxNode linked list.
  • the TranxNode linked list is composed of a group of TranxNode nodes. Each TranxNode node includes the transaction information of the transaction and the BINLOG log location corresponding to the transaction.
  • the TranxNode linked list is ordered from beginning to end, and its BINLOG log position is gradually increasing. Therefore, the latest transaction information is cached at the tail of the TranxNode linked list.
  • Step 806 The transaction thread completes the task, returns the thread pool, and continues processing the remaining tasks.
  • the asynchronous response mechanism of waiting for the standby ACK is embodied.
  • step 807 after the DUMP thread is awakened, the corresponding BINLOG event is sent, and after the sending is completed, it continues to be in a waiting state, and continues to wait for subsequent transactions to send BINLOG event requests.
  • Step 808 the input / output (Input / Output, I / O) thread of the MySQL standby machine receives the BINLOG event sent by the host, reads and writes the RELAY_LOG file.
  • step 809 when the standby IO thread receives all the BINLOG events, it returns an ACK response to the host.
  • the host has an ACK listening thread dedicated to receiving the standby ACK response. After the ACK listening thread receives the standby ACK response, it finds the corresponding transaction information from the TranxNode list according to the BINLOG position carried in the ACK response. Join the pending transaction queue of the SUPER thread group.
  • step 811 there is always a thread for monitoring in the SUPER thread group, and it monitors in a loop whether there are tasks to be processed in the queue of pending transactions.
  • step 812 when it is detected that there is a task to be processed in the transaction queue to be submitted, it is determined whether there is an idle SUPER thread in the SUPER thread group; if it exists, step 813 is performed; if it does not exist, step 814 is performed.
  • Step 813 There is an idle SUPER thread in the SUPER thread group, and the SUPER thread is allocated to process the final transaction submission operation.
  • step 814 there is no idle SUPER thread in the SUPER thread group, and the ordinary thread allocated to the thread pool executes first.
  • Step 815 After processing the final transaction submission operation, a successful response is returned to the user.
  • the real-time process of the synchronization technology of the active and standby data nodes is shown in the dimension of time sequence advancement. Described as follows:
  • step 901 the user initiates a transaction submission request SQL on the APP side.
  • Step 902 The MySQL host receives the SQL request, takes a thread from the thread pool, and starts the transaction submission process.
  • Step 903 The transaction thread processes the transaction submission request, and flush && sync the BINLOG log to disk to ensure that the host data can be recovered in various scenarios.
  • step 904 the transaction thread wakes up the independent DUMP thread.
  • step 905 the transaction thread saves the transaction information and exits the thread, and the idle thread returns the thread pool.
  • step 906 after the DUMP thread is awakened, the corresponding BINLOG event is sent. After the completion of the sending, the DUMP thread continues to be in a waiting state and continues to wait for subsequent transactions to send BINLOG event requests.
  • Step 907 The IO thread of the MySQL standby machine receives the BINLOG event sent by the host, reads and writes the RELAY_LOG file, and returns an ACK response to the host after completion.
  • the host has an ACK listening thread that specifically receives the ACK response from the standby machine.
  • the ACK monitoring thread After the ACK monitoring thread receives the ACK response from the standby machine, it finds the corresponding transaction information from the TranxNode list according to the BINLOG position carried in the ACK response. Join the pending transaction queue of the SUPER thread group.
  • step 909 the SUPER thread group completes the final transaction commit operation.
  • Step 910 A successful response is returned to the user.
  • a thread synchronization mechanism is used to wait for the ACK response from the standby machine. There are scenarios that need to improve the synchronization performance of the active and standby data nodes.
  • FIG. 11 The flowchart of the data synchronization method in the second application example of the present disclosure is shown in FIG. 11 and includes:
  • step 1001 the user initiates a transaction submission request on the MariaDB client.
  • step 1002 the MariaDB host receives a transaction submission request, takes a thread from the thread pool, and processes the received transaction submission request.
  • Step 1003 The transaction thread processes the transaction submission request, and flush && sync BINLOG logs to disk to ensure that the host data can be recovered in various scenarios.
  • Step 1004 the transaction thread wakes up the independent BINLOGDUMP thread to send the BINLOG log of the transaction, and the BINLOG DUMP thread sends the corresponding BINLOG event to the standby machine according to the BINLOG position of the transaction;
  • the transaction thread saves the transaction information in a transaction information linked list.
  • the transaction information linked list is composed of a set of nodes, and each node includes transaction information of the transaction and a BINLOG log position corresponding to the transaction.
  • the transaction information linked list is ordered from beginning to end, and its BINLOG log position is gradually increasing. Therefore, the latest transaction information is cached at the end of the transaction information linked list.
  • Step 1006 The transaction thread completes the task, returns the thread pool, and continues processing the remaining tasks.
  • the asynchronous response mechanism of waiting for the standby ACK is embodied.
  • step 1007 after the DUMP thread is awakened, the corresponding BINLOG event is sent, and after the sending is completed, it waits for the standby machine to return a transaction response.
  • step 1008 the IO thread of the MariaDB standby machine receives the BINLOG event sent by the host, reads and writes the RELAY_LOG file.
  • step 1009 when the standby IO thread receives all the BINLOG events, it returns a transaction response to the host.
  • step 1010 after the BINLOG and DUMP threads of the host judge that the transaction response of the standby machine is received, the transaction information is taken out from the transaction information list according to the BINLOG position of the transaction response, and added to the pending transaction queue of the SUPER thread group.
  • step 1011 there is always a thread for monitoring in the SUPER thread group, and it is monitored in a cyclic manner whether there are tasks to be processed in the transaction queue to be submitted.
  • step 1012 when it is detected that there is a task to be processed in the transaction queue to be submitted, it is determined whether there is an idle SUPER thread in the SUPER thread group; if it exists, step 1013 is performed; if it does not exist, step 1014 is performed.
  • Step 1013 there is an idle SUPER thread in the SUPER thread group, and the SUPER thread is allocated to handle the final transaction submission operation.
  • step 1014 there is no idle SUPER thread in the SUPER thread group, and the ordinary thread allocated to the thread pool executes first.
  • Step 1015 After processing the final transaction submission operation, a successful response is returned to the user.
  • FIG. 12 A flowchart of a data synchronization method according to application example 3 of the present disclosure is shown in FIG. 12 and includes:
  • step 1101 the user initiates a transaction submission request on the Percona client.
  • step 1102 the Percona host receives a request for transaction submission, takes a thread from the thread pool, and processes the received transaction submission request.
  • step 1103 the transaction thread processes the transaction submission request, and flush && sync binary logs are sent to the disk to ensure that the host data can be recovered in various scenarios.
  • Step 1104 the transaction thread wakes up the independent DUMP thread to send the binary log of the transaction, and the DUMP thread sends the corresponding binary log to the standby machine according to the binary log position of the transaction.
  • the transaction thread saves the transaction information in a transaction information linked list.
  • the transaction information linked list is composed of a group of nodes. Each node includes transaction information of the transaction and a binary log position corresponding to the transaction.
  • the transaction information linked list is ordered from beginning to end, and its binary log position is gradually increased. Therefore, the latest transaction information is cached at the end of the transaction information linked list.
  • Step 1106 the transaction thread completes the task, returns the thread pool, and continues processing the remaining tasks.
  • the principle of asynchronously waiting for the standby ACK response mechanism is mainly reflected here.
  • step 1107 after the DUMP thread is awakened, the corresponding binary log is sent. After sending, the DUMP thread continues to be in the waiting state, and continues to wait for subsequent transactions to send binary log requests.
  • step 1108 the IO thread of the Percona standby machine receives the binary log sent by the host, reads and writes the RELAY_LOG file.
  • step 1109 when the standby IO thread receives all the binary logs, it returns a transaction ACK response to the host.
  • Step 1110 After the host's ACK thread judges that it has received the transaction ACK response from the standby machine, it takes out the transaction information from the transaction information list according to the binary log position of the transaction ACK response and adds it to the pending transaction queue of the SUPER thread group.
  • step 1111 there is always a thread for monitoring in the SUPER thread group, which monitors in a loop whether there are tasks to be processed in the transaction queue to be submitted.
  • step 1112 if it is detected that there is a task to be processed in the transaction queue to be submitted, it is determined whether there is an idle SUPER thread in the SUPER thread group; if it exists, step 1113 is performed; if it does not exist, step 1114 is performed.
  • Step 1113 there is an idle SUPER thread in the SUPER thread group, and the SUPER thread is allocated to handle the final transaction commit operation.
  • step 1114 there is no idle SUPER thread in the SUPER thread group, and the ordinary thread allocated to the thread pool executes first.
  • Step 1115 After processing the final transaction submission operation, a successful response is returned to the user.
  • a thread synchronization mechanism is used to wait for the ACK response of the standby machine.
  • the synchronization performance of the active and standby data nodes needs to be improved.
  • FIG. 13 A flowchart of a data synchronization method according to application example 4 of the present disclosure is shown in FIG. 13 and includes:
  • step 1201 the user initiates a transaction submission request on the PostgreSQL client.
  • Step 1202 The PostgreSQL database host receives a user transaction submission request, and takes a thread from the thread pool to process the transaction submission request.
  • Step 1203 The transaction thread writes a write-ahead logging (WAL) recording log related to the transaction to the disk.
  • WAL write-ahead logging
  • step 1204 the transaction thread wakes up the write-ahead log sending (WAL_SENDER) thread to send a WAL record log to the standby machine.
  • WAL_SENDER write-ahead log sending
  • Step 1205 The transaction thread saves the transaction information to the transaction cache linked list.
  • Step 1206 The transaction thread exits and returns to the thread pool, so that the thread asynchronously waits for a standby ACK response.
  • step 1207 the WAL_SENDER thread of the host sends a WAL log to the standby.
  • step 1208 the write-ahead log receiving (WAL_RECEIVER) thread of the standby machine receives the WAL log sent by the host, and performs related operations such as writing to a disk.
  • WAL_RECEIVER write-ahead log receiving
  • step 1209 the WAL_RECEIVER thread of the standby machine sends the latest received WAL log position information response to the WAL_SENDER thread of the host.
  • Step 1210 After receiving the WAL log location information response from the host, the host determines that the transaction can perform the final commit operation based on the location information. If it can, the transaction information is taken from the transaction cache linked list and added to the pending transaction processing queue of the SUPER thread group.
  • step 1211 there is always a thread for monitoring in the SUPER thread group, and it is monitored cyclically whether there are tasks to be processed in the pending transaction queue.
  • step 1212 when it is detected that there is a task to be processed in the transaction queue to be submitted, it is determined whether there is an idle SUPER thread in the SUPER thread group; if it exists, step 1213 is performed; if it does not exist, step 1214 is performed.
  • Step 1213 There is an idle SUPER thread in the SUPER thread group, and the SUPER thread is allocated to handle the final transaction submission operation.
  • step 1214 there is no idle SUPER thread in the SUPER thread group, and the ordinary thread allocated to the thread pool executes first.
  • Step 1215 After processing the final transaction submission operation, a successful response is returned to the user.
  • an embodiment of the present disclosure further provides an apparatus for data synchronization, including a first receiving module 131, a log module 132, a second receiving module 133, and a transaction submission module 134.
  • a first receiving module 131 configured to receive a transaction submission request
  • the log module 132 is configured to control a host transaction thread in a database host to write a log to a disk file according to the transaction submission request, instruct a log forwarding thread to send the log to a database standby machine, and control the host transaction thread to exit;
  • a second receiving module 133 configured to receive an acknowledgement ACK response sent by the database standby machine
  • the transaction commit module 134 is configured to select a thread from the thread pool to perform a transaction commit operation.
  • the log module 132 is further configured to control the host transaction thread to save transaction information corresponding to the transaction submission request to a transaction cache list.
  • the transaction information in the transaction cache list is associated with a log location.
  • the transaction submission module 134 is configured to add a transaction corresponding to the ACK response to a transaction submission task queue, and select a thread from the thread pool to preferentially perform a transaction submission operation on the transaction corresponding to the ACK response.
  • the transaction submission module 134 is configured to listen to an ACK response, and according to the log position carried in the ACK response, find transaction information corresponding to the log position in the transaction cache list, The transaction information adds a transaction corresponding to the ACK response to a transaction submission task queue.
  • the transaction submission module 134 is configured to preferentially select a SUPER thread in a SUPER thread group in the thread pool corresponding to the ACK response when a pending task is detected in the transaction submission task queue. Transaction performs a transaction commit operation.
  • the transaction submission module 134 is further configured to allocate ordinary threads in the thread pool to preferentially perform a transaction commit operation on a transaction corresponding to the ACK response in response to no idle SUPER threads in the SUPER thread group. ; Wherein the ordinary thread is a thread in the thread pool other than the SUPER thread group.
  • an embodiment of the present disclosure further provides a database host, including a memory 142, a processor 141, and a computer program stored on the memory 142 and executable on the processor 141.
  • the processor executes the program When implementing the method described in the above embodiment.
  • An embodiment of the present disclosure further provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, where the computer-executable instructions are used to execute the method described in the foregoing embodiments.
  • the storage medium may include: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.
  • ROM read-only memory
  • RAM random access memory
  • mobile hard disk a magnetic disk
  • optical disk a magnetic disk
  • All or some steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof.
  • the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of multiple The physical components work together.
  • Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit.
  • Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media).
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in a method or technology for storing information such as computer-readable instructions, data structures, program modules or other data .
  • Computer storage media include RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other storage technologies, and Compact Disc Read-Only Memory (CD-ROM) , Digital Video Disk (DVD) or other optical disk storage, magnetic case, magnetic tape, disk storage or other magnetic storage device, or other media that can be used to store desired information and can be accessed by a computer.
  • a communication medium typically contains computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include an information delivery medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Cette invention concerne un procédé et un appareil de synchronisation de données, un hôte de base de données et un support de stockage, le procédé comprenant les étapes consistant à : recevoir, par un hôte de base de données, une demande de dépôt de transaction ; écrire, par un fil de transaction d'hôte dans l'hôte de base de données, un journal dans un fichier de disque en fonction de la demande de dépôt de transaction pour ordonner à un fil d'acheminement de journal d'envoyer le journal à une machine de veille de base de données, pendant que le fil de transaction d'hôte est fourni en sortie ; recevoir, par l'hôte de base de données, une réponse d'accusé de réception (ACK) envoyée par la machine de veille de base de données, et sélectionner une opération de dépôt de transaction d'exécution de fil à partir d'un groupe de fils.
PCT/CN2019/099048 2018-08-02 2019-08-02 Procédé et appareil de synchronisation de données, hôte de base de données et support de stockage WO2020025049A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810873311.7A CN110807062B (zh) 2018-08-02 2018-08-02 一种数据同步的方法、装置和数据库主机
CN201810873311.7 2018-08-02

Publications (1)

Publication Number Publication Date
WO2020025049A1 true WO2020025049A1 (fr) 2020-02-06

Family

ID=69231509

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/099048 WO2020025049A1 (fr) 2018-08-02 2019-08-02 Procédé et appareil de synchronisation de données, hôte de base de données et support de stockage

Country Status (2)

Country Link
CN (1) CN110807062B (fr)
WO (1) WO2020025049A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111414344B (zh) * 2020-03-25 2023-03-14 电子科技大学 一种应用于遥爆系统的数据保存方法
CN113868014A (zh) * 2020-06-30 2021-12-31 华为技术有限公司 一种数据同步方法及装置
CN112597251B (zh) * 2020-12-29 2023-01-24 天津南大通用数据技术股份有限公司 数据库集群日志同步方法、装置、服务器及存储介质
CN112948136A (zh) * 2021-02-02 2021-06-11 华东计算技术研究所（中国电子科技集团公司第三十二研究所） 一种嵌入式操作系统异步日志记录的实现方法
CN113326318A (zh) * 2021-05-28 2021-08-31 广州市炬元互联网科技有限公司 一种双系统间数据同步的方法与系统
CN113377582B (zh) * 2021-08-16 2021-11-05 北京易鲸捷信息技术有限公司 基于日志推送的分布式数据库容灾方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080288511A1 (en) * 2006-10-02 2008-11-20 Salesforce.Com, Inc. Asynchronous method and system for performing an operation on metadata
CN103064761A (zh) * 2012-12-24 2013-04-24 华为技术有限公司 一种数据同步方法、设备和系统
CN105468727A (zh) * 2015-11-20 2016-04-06 国家电网公司 基于zookeeper实现MySQL强一致性复制方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008134767A (ja) * 2006-11-28 2008-06-12 Hitachi Software Eng Co Ltd コネクション割当管理方法
CN102081611B (zh) * 2009-11-26 2012-12-19 中兴通讯股份有限公司 一种主、备网管系统数据库同步的实现方法及装置
CN202058147U (zh) * 2011-05-23 2011-11-30 北京六所和瑞科技发展有限公司 分布式实时数据库管理系统
CN103605498B (zh) * 2013-12-05 2016-07-06 用友网络科技股份有限公司 单线程任务的多线程执行方法及系统
US9619343B2 (en) * 2015-02-19 2017-04-11 International Business Machines Corporation Accelerated recovery after a data disaster
CN104899274B (zh) * 2015-05-27 2018-04-03 北方信息控制集团有限公司 一种内存数据库高效远程访问方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080288511A1 (en) * 2006-10-02 2008-11-20 Salesforce.Com, Inc. Asynchronous method and system for performing an operation on metadata
CN103064761A (zh) * 2012-12-24 2013-04-24 华为技术有限公司 一种数据同步方法、设备和系统
CN105468727A (zh) * 2015-11-20 2016-04-06 国家电网公司 基于zookeeper实现MySQL强一致性复制方法

Also Published As

Publication number Publication date
CN110807062B (zh) 2021-06-22
CN110807062A (zh) 2020-02-18

Similar Documents

Publication Publication Date Title
WO2020025049A1 (fr) Procédé et appareil de synchronisation de données, hôte de base de données et support de stockage
US10990610B2 (en) Synchronization on reactivation of asynchronous table replication
CN107888657B (zh) 低延迟分布式存储系统
US11010262B2 (en) Database system recovery using preliminary and final slave node replay positions
US10255343B2 (en) Initialization protocol for a peer-to-peer replication environment
US11218418B2 (en) Scalable leadership election in a multi-processing computing environment
EP3195117B1 (fr) Configuration automatisée de groupes de stockage coordonnés avec des journaux
US20180101558A1 (en) Log-shipping data replication with early log record fetching
US7330860B2 (en) Fault tolerant mechanism to handle initial load of replicated object in live system
US11928089B2 (en) Data processing method and device for distributed database, storage medium, and electronic device
CN105493474B (zh) 用于支持用于同步分布式数据网格中的数据的分区级别日志的系统及方法
CN109388481B (zh) 一种事务信息的传输方法、系统、装置、计算设备和介质
CN103870570A (zh) 一种基于远程日志备份的HBase数据可用性及持久性的方法
KR20140106588A (ko) 공유 볼륨의 어플리케이션 일관된 스냅샷 기법
US11748215B2 (en) Log management method, server, and database system
WO2018010501A1 (fr) Procédé, appareil et système de synchronisation d'identificateur de transaction globale (gtid), et support de stockage
JP2022531867A (ja) データ読み取り方法、装置、コンピュータ装置及びコンピュータプログラム
CN115994053A (zh) 数据库备机的并行回放方法、装置、电子设备及介质
CN112084206A (zh) 数据库的事务请求处理方法、相关设备及存储介质
WO2023116419A1 (fr) Procédé de synchronisation de données, dispositif et support de stockage lisible par ordinateur
WO2017096942A1 (fr) Système de mémorisation de fichiers, procédé de programmation de données et nœud de données
WO2020119709A1 (fr) Procédé de mise en œuvre de fusion de données, dispositif, système et support de stockage
US10579604B2 (en) Database system, information processing device, method and medium
CN108460047B (zh) 数据同步方法及设备
US10572503B2 (en) Database system, information processing device, method and medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19843226

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 16/06/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19843226

Country of ref document: EP

Kind code of ref document: A1