CN111930693A - Transaction merging execution method and device based on log analysis synchronization - Google Patents

Transaction merging execution method and device based on log analysis synchronization Download PDF

Info

Publication number
CN111930693A
CN111930693A CN202010470229.7A CN202010470229A CN111930693A CN 111930693 A CN111930693 A CN 111930693A CN 202010470229 A CN202010470229 A CN 202010470229A CN 111930693 A CN111930693 A CN 111930693A
Authority
CN
China
Prior art keywords
transaction
linked list
merged
lsn
merging
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010470229.7A
Other languages
Chinese (zh)
Other versions
CN111930693B (en
Inventor
孙峰
付铨
黄海明
余院兰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Dameng Database Co Ltd
Original Assignee
Wuhan Dameng Database Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Dameng Database Co Ltd filed Critical Wuhan Dameng Database Co Ltd
Priority to CN202010470229.7A priority Critical patent/CN111930693B/en
Publication of CN111930693A publication Critical patent/CN111930693A/en
Application granted granted Critical
Publication of CN111930693B publication Critical patent/CN111930693B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/466—Transaction processing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to the technical field of computers, in particular to a transaction merging execution method and device based on log analysis synchronization, wherein the method comprises the following steps: sequentially taking out transaction combinations from the transaction linked list to be combined, and when the ith transaction is taken out, marking out i-1 operation intervals in the operation container by using i-1 submitted LSNs in the combined transaction linked list as a partition boundary; taking out the operation from the ith transaction, positioning the operation to a corresponding interval in the operation container through the LSN of the current operation, finding the position of the same operation type in the corresponding interval, and inserting the current operation; and after the transaction combination is completed, extracting the operation from the operation container in sequence for execution, and performing batch execution after the same operation is combined. The invention takes the commit LSN of the first committed transaction as the partition boundary during merging, moves up the partial operation of the later committed transaction to find the same operation, can execute in batch during execution, improves the merging rate of the operation among the transactions, and can effectively improve the synchronization performance.

Description

Transaction merging execution method and device based on log analysis synchronization
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of computers, in particular to a transaction merging execution method and device based on log analysis synchronization.
[ background of the invention ]
The traditional main and standby mechanism based on the database realizes real-time copying of database data, and is an important solution for carrying out disaster recovery backup and ensuring data safety. Currently, DataGuard of ORACLE corporation, DM7 master and slave products of dreams database, etc. are all mature commercial products for implementing this solution. Under the master-slave mechanism of the database, a general backup machine is used as a backup node and only provides read-only access, and some applications of read-only access such as report analysis, data mining and the like can be performed on the backup machine, but read-write access cannot be provided like a host. In addition, the database master-slave mechanism requires the database system of the slave machine to be consistent with the host machine, and for the heterogeneous database system environment, the effective real-time data replication cannot be realized by using the master-slave mechanism of the database.
Aiming at the defect of realizing data copying of a main and standby mechanism of a database, the existing heterogeneous database copying technology based on software is widely applied. The technology captures the incremental data of the database at the source end, then sends the incremental data to the target end, and applies the incremental data to the target database at the target end through a general database access interface to realize data replication. The technology uses a universal database interface, so that heterogeneous database system replication is supported, heterogeneous operating system environments are supported, and the database system of the target terminal standby machine can read and write, so that the system is a 'double-active' system.
At present, there are various technical ways to achieve obtaining incremental data of a source database, wherein a real-time data synchronization technology based on database log capture analysis is greatly developed and researched due to its small intrusiveness on the source database and high performance of capture analysis. The technology captures INSERT, UPDATE and DELETE operation logs of a database by analyzing source database filing or online logs, and then sends the INSERT, UPDATE and DELETE operation logs to a target end; and the target end reversely generates the log information, restores the log information into an SQL statement mode, and then uses a database general interface to apply to the target database to realize data replication. Therefore, in the real-time database replication process, the execution efficiency of the target end is an important factor influencing the data synchronization performance.
Generally, a large number of small transactions may exist in each transaction concurrently executed on a source-end database, and a database system executes according to a concurrency control mechanism, performs mutually exclusive execution on conflicting transaction operations, and sequentially records an operation log of each transaction in a log file; during data synchronization, a transaction merging execution strategy is adopted for small transactions, and operations executed by a source end for multiple times are merged and then executed in batch, so that the synchronization performance is improved. However, existing data synchronization systems often perform synchronization in transaction units, and are allocated to execution threads to execute after a transaction is determined to be committed; under the mechanism, the transaction merging method usually adopts a transaction adding mode to merge, although the merging strategy can solve the problem of synchronization efficiency under partial application scenes, the operation merging rate between the transactions after the transactions are merged under some special application scenes is not high, the execution efficiency of a target end is still low, and the synchronization performance cannot be effectively improved. Therefore, it is an important technical problem to be solved in the industry to merge transactions and operations between transactions on the premise of ensuring consistency of data replication and improve execution efficiency of data replication at a target end.
In view of the above, it is an urgent problem in the art to overcome the above-mentioned drawbacks of the prior art.
[ summary of the invention ]
The technical problems to be solved by the invention are as follows:
the existing data synchronization system usually performs synchronization by taking transactions as a unit and performs transaction combination by adopting simple superposition, but the operation combination rate of the transactions after the transaction combination is not high, the execution efficiency of a target end is still low, and the synchronization performance cannot be effectively improved; therefore, how to increase the merging rate of operations between the transactions after the transaction merging and increase the execution efficiency of the target-side data copying on the premise of ensuring the consistency of the data copying becomes an important technical problem to be solved urgently in the industry.
The invention solves the technical problems through the following technical scheme:
in a first aspect, the present invention provides a transaction merge execution method based on log parsing synchronization, including:
scattering the affairs with a plurality of table operations according to the table IDs, and selecting a plurality of affairs with the same table IDs to be added to the affair linked list to be merged;
sequentially taking out all the transactions from the transaction linked list to be merged for merging, adding the submission LSN of the transaction to the merged transaction linked list when merging one transaction, and storing the operation generated after merging to an operation container; when the ith transaction is taken out from the transaction linked list to be merged, i-1 submitted LSNs in the merged transaction linked list are used as partition boundaries, and i-1 operation intervals are marked out from an operation container; sequentially taking out operations from the ith transaction, positioning to a corresponding operation interval in an operation container through the LSN of the current operation, finding an operation position with the same operation type as the current operation in the corresponding operation interval, and inserting the current operation to the position;
after the combination of the transactions is completed, extracting the operation execution from the operation container in sequence; the method comprises the following steps that a submitting operation clamped between two operations is omitted for execution, and the same operations are combined and then executed in batch;
and the transaction linked list to be merged and the objects in the merged transaction linked list are stored according to the size sequence of the submitted LSNs of the transactions.
Preferably, before the transaction in which there are a plurality of table operations is scattered by the table ID, the method further includes:
after reading the logs from the source database, the source database synchronization system analyzes the logs according to the order of capturing the logs and sends the analyzed operations to the destination database synchronization system in order;
after receiving the operation of the source end, the target end database synchronization system carries out classification management according to the transaction ID and the table ID of the operation, and adds the transaction with the same transaction ID to a transaction linked list to be executed after receiving the transaction submission operation;
and storing the transactions in the to-be-executed transaction linked list according to the size sequence of transaction submission LSN.
Preferably, the selecting a plurality of transactions with the same table ID to be added to the to-be-merged transaction linked list specifically includes:
and the execution thread takes out a transaction from the to-be-executed linked list and adds the transaction to the to-be-merged linked list, selects the transaction with the same table ID from the to-be-executed linked list according to the table ID of the transaction, and adds the selected transaction with the same table ID to the to-be-merged linked list.
Preferably, for any transaction a in the transaction linked list to be merged, the merging process specifically includes:
taking out the transaction A from the transaction linked list to be merged, taking a submitted LSN from the merged transaction linked list, and taking the submitted LSN as the current segmented LSN;
taking out an operation from the transaction A, and judging whether the operation type of the current operation is a commit operation or not; if the operation is a commit operation, the commit operation is added to the operation container, the corresponding commit LSN is added to the merged transaction linked list, and the next transaction is continuously taken out from the to-be-merged transaction linked list in sequence; if not, determining whether the LSN of the current operation is less than the split LSN.
Preferably, an operation linked list is adopted to store the operations needing batch combination in the combination process, and if the LSN of the current operation is smaller than the segmented LSN, whether the operation type of the current operation is consistent with the operation type of the operation in the operation linked list is judged; if so, adding the current operation to the operation linked list, and sequentially continuing to extract the next operation from the transaction A.
Preferably, if the operation type of the current operation is not consistent with the operation type of the operation in the operation linked list, the operation container is positioned by using the split LSN, and whether the operation with the same operation type as the operation in the operation linked list exists or not is searched from the position forward until the position of the last merging operation or the commit operation of the last transaction is searched;
if the operation with the same operation type as the operation in the operation linked list is searched, the operation in the operation linked list is merged to the searched position; if the search is not completed, combining the operation in the operation linked list to the position where the search is stopped;
and emptying the operation linked list after the combination is finished, adding the current operation into the operation linked list, and continuously extracting the next operation from the transaction A in sequence.
Preferably, an operation linked list is adopted to store the operations needing batch combination in the combination process, and if the LSN of the current operation is greater than the segmented LSN, whether the operation exists in the current operation linked list is judged;
if so, using the partition LSN to locate in the operation container, and searching forward from the location whether the operation of the same operation type as the operation in the operation linked list exists, and searching until the position of the last merge operation or the commit operation of the last transaction.
Preferably, if the operation with the same operation type as the operation in the operation linked list is searched, the operation in the operation linked list is merged to the searched position; if the search is not completed, combining the operation in the operation linked list to the position where the search is stopped;
clearing the operation linked list after the merging is finished, continuously taking a submitted LSN from the merged transaction linked list in sequence, taking the submitted LSN as a new segmentation LSN, and continuously judging whether the LSN currently operated is smaller than the new segmentation LSN;
and if the tail end of the merged transaction linked list is reached when the submitted LSN is taken, adding the rest operations in the transaction A into the operation container, and continuously taking out the next transaction from the transaction linked list to be merged in sequence.
Preferably, when fetching the commit LSN from the merged transaction linked list, if the merged transaction linked list is empty, or the transaction a has traversed the commit LSN in the merged transaction linked list, the remaining operations in the transaction a are sequentially added to the operation container; and after the addition is finished, continuously taking out the next transaction from the transaction linked list to be merged in sequence.
In a second aspect, the present invention provides a transaction merge execution apparatus based on log resolution synchronization, including at least one processor and a memory, where the at least one processor and the memory are connected through a data bus, and the memory stores instructions executable by the at least one processor, where the instructions are used to complete the transaction merge execution method based on log resolution synchronization according to the first aspect after being executed by the processor.
Compared with the prior art, the invention has the beneficial effects that:
in the transaction merging execution method based on log analysis synchronization, the transactions with a plurality of table operations are scattered according to the table IDs, then the transactions with the same table IDs are selected according to the table IDs to be merged, the submitted LSN of the first submitted transaction is used as a partition boundary during merging, the partial operations executed in parallel by the second submitted transaction in a source database are moved upwards to find the same operation, so that the same operations in the second submitted transaction and the first submitted transaction are merged together, batch execution can be realized during execution, the merging rate of the operations among the transactions after transaction merging is improved, the execution efficiency of target end data copying can be effectively improved, and the synchronization performance is improved.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a flowchart of a transaction merge execution method based on log resolution synchronization according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a transaction merging method according to an embodiment of the present invention;
fig. 3 is an architecture diagram of a transaction merge execution apparatus based on log resolution synchronization according to an embodiment of the present invention.
[ detailed description ] embodiments
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other. The invention will be described in detail below with reference to the figures and examples.
Example 1:
data synchronization in the target-side database synchronization transaction, operations should be executed by minimizing the number of interactions with the database, because each interaction with the database causes additional overhead, and the time taken to execute an insert operation for one thousand rows in batch is far less than the time taken to execute a single insert row for one thousand rows each time, so operation merging is a very important technology in data synchronization. The synchronization is based on the log analysis, and is based on the transaction unit, so that the synchronization performance can be effectively improved by combining a plurality of small transactions into a large transaction and then executing the combined operation in the large transaction.
The database logs are written in a serial mode when operations in the database are written into the logs, that is, logs generated by transaction operations executed in parallel inside the database are alternately written into a log file. For the above reasons, if only one active transaction performs a DML operation on a table, the DML operation log of the table is continuously recorded in the database log; if two active transactions respectively perform DML operations on the same table, the DML logs of the two transactions on the table are alternately recorded in the database log. Data synchronization adopts a transaction combination mode to synchronize the two transactions when the target end database synchronizes the two transactions, and if a simple superposition mode is adopted, the operation aiming at the two transactions in the optimal state can be combined into one batch execution under the condition that the two transaction operations are the same operation; however, if the operations of the two transactions are performed alternately by various DML operations, the merging rate between the transactions merged by the simple superposition method is low even if the transactions are merged.
In some application scenarios, the destination database is only used as a redundant backup of the source database, and the application does not directly access the data on the destination database, and in this case, the application does not care about the logic between the data on the destination database; that is, the synchronization service does not need to consider the data association within the transaction or between the transactions when synchronizing the transactions, as long as it is ensured that the data on the source database and the destination database are consistent as a whole after the synchronization is completed. In this scenario, the order of operation execution within a single transaction or among multiple transactions is irrelevant, and the destination may have a more flexible method in terms of a policy for improving the warehousing performance. When the data synchronization service synchronizes data, the classification of operations can be further refined on the basis of transaction classification, the operations on a plurality of tables in a single transaction are scattered into a plurality of smaller transactions according to the table IDs, and then the small transactions are combined into larger transactions according to the table IDs to be combined, so that the combination rate of the operations can be more effectively improved, and the transactions combined by each table can be executed without the sequence of transaction submission in the environment of parallel execution.
According to the above thought, an embodiment of the present invention provides a transaction merge execution method based on log parsing synchronization, as shown in fig. 1, which mainly includes the following steps:
step 201, scattering the transactions with a plurality of table operations according to the table IDs, and selecting a plurality of transactions with the same table ID to add to the transaction linked list to be merged.
Each transaction of the source end includes one or more table operations, each operation correspondingly has a Log Sequence Number (LSN) to indicate an execution sequence of the operation, and the operations analyzed by the source end are sent to the destination end in sequence (i.e., in a sequence from small to large according to the LSN). For each transaction of the source end, if only one table operation is contained, no processing is needed; if multiple table operations are involved, the corresponding transactions need to be broken up into multiple small transactions according to the table IDs. After the break, a plurality of transactions having the same table ID are selected based on the table ID involved by a certain transaction, and the plurality of transactions are combined into a large transaction to be submitted.
For convenience of operation, a transaction linked list to be merged may be initialized in advance, and the selected transaction to be merged is added to the transaction linked list to be merged after the selection is completed. The transactions in the to-be-merged transaction linked list are stored according to the size sequence of the commit LSN of the transactions (i.e., the LSN of the commit operation included in the transactions), that is, according to the commit sequence of the transactions.
Step 202, taking out the transactions from the transaction linked list to be merged in sequence for merging, adding the submission LSN of the transaction to the merged transaction linked list when merging one transaction, and storing the operation generated after merging to an operation container.
In the process of merging the transactions, the transactions to be merged can be sequentially taken out from the transaction linked list to be merged for merging according to the sequence of the submitted LSNs of the transactions from small to large. For convenience of operation, a merged transaction linked list and an operation container may be initialized in advance, and each time a transaction is merged, a commit LSN of the transaction is added to the merged transaction linked list for standby, and an operation generated after merging is stored in the operation container. Therefore, when the ith transaction is taken from the to-be-merged transaction linked list in sequence, the merging of the first i-1 transactions in the to-be-merged transaction linked list is already completed, at this time, the transaction operation generated after the merging of the first i-1 transactions is already stored in the operation container, and the commit LSN of the first i-1 transactions is already stored in the merged transaction linked list.
Furthermore, when the ith transaction is taken out from the transaction linked list to be merged, i-1 operation intervals are marked out in the operation container by using i-1 submitted LSNs in the merged transaction linked list as a partition boundary; and then sequentially taking out each operation from the ith transaction, locating the operation in a corresponding operation interval in the operation container through the LSN of the current operation for each operation, finding an operation position with the same operation type as the current operation in the corresponding operation interval, and inserting the current operation into the position, so that the same operations are grouped together, and subsequent combination and batch execution are facilitated. The specific merging process will be described in detail in embodiment 2, and will not be described herein.
Step 203, after the combination of the transactions is completed, extracting the operation execution from the operation container in sequence; the commit operation sandwiched between the two operations is omitted from execution, and the same operations are executed in batch after being merged.
When the execution thread finishes traversing the to-be-merged transaction linked list, namely all the transactions in the to-be-merged transaction linked list are taken out and the transaction merging is finished, a plurality of transactions in the to-be-merged transaction linked list are merged into one large transaction, at the moment, the operations related to all the to-be-merged transactions are inserted into the operation container according to a set rule, and the same operations are grouped together, so that the operations can be extracted from the operation container in sequence for execution. It should be noted that the same operation needs to be executed in batch after being merged, and the commit operation sandwiched between the two operations needs to be omitted, that is, only the commit operation at the tail of the operation container is executed (i.e., the commit operation with the largest commit LSN is submitted); and submitting the merged large transaction until all operations in the operation container are executed, and finishing warehousing.
In the transaction merging execution method provided by the embodiment of the invention, the transactions with a plurality of table operations are scattered according to the table IDs, then the transactions with the same table IDs are selected according to the table IDs to be merged, the submitted LSN of the first submitted transaction is used as a partition boundary during merging, partial operations which are executed in parallel by the later submitted transaction in the source database are moved upwards to find the operations with the same operation type, and the same operations are grouped together as much as possible, so that batch execution can be performed during execution, the merging rate of the operations among the transactions after transaction merging is improved, the execution efficiency of target end data copying can be effectively improved, and the synchronization performance is improved. The method mainly adjusts the operation sequence based on the principle that the operation generated by the source database has no data conflict between two submitted logs, and classifies the operations in the range according to the table and the operation types by taking the submitted logs as boundaries, so that the merging rate of the operations after transaction merging can be effectively improved, and batch execution is realized to improve the warehousing performance.
Example 2:
on the basis that the main steps of the transaction merging execution method based on log parsing synchronization are introduced in the above embodiment 1, the embodiment of the present invention further introduces the complete steps of the transaction merging execution method in the embodiment 1 in detail.
With reference to fig. 2, an embodiment of the present invention provides a transaction merge execution method based on log parsing synchronization, which specifically includes the following steps:
step 101, deploying a synchronization system in a source database and a destination database. The source database synchronization system is used for reading logs from a source database, and the destination database synchronization system is responsible for applying synchronization operation sent by a source end to a destination database.
For convenience of subsequent operation, after the target-end database synchronization system is started, a to-be-executed transaction linked list, a to-be-merged transaction linked list and a merged transaction linked list can be initialized at the target end. The to-be-executed transaction linked list is used for storing the to-be-executed and warehousing transactions, the to-be-merged transaction linked list is used for storing the transactions (to-be-merged transactions) needing merging operation, the merged transaction linked list is used for storing the submitted LSNs of the merged transactions, and the transaction sequences in the three linked lists are stored according to the size sequence of the submitted LSNs of the transactions, namely according to the submitting sequence of the transactions.
Step 102, after the source-end database synchronization system reads the logs from the source-end database, the log analysis is performed according to the order of capturing the logs, and the operation after the analysis is sequentially sent to the target-end database synchronization system.
And 103, after receiving the operation of the source end, the target end database synchronization system carries out classification management according to the transaction ID and the table ID of the operation, and after receiving the transaction submitting operation, adds the transaction with the same transaction ID to the transaction linked list to be executed.
Specifically, after receiving an operation of a source end, a destination-end database synchronization system first determines whether the operation is a commit operation; if the operation is a commit operation, extracting the transaction ID of the operation, finding out all transactions with consistent transaction ID in the classified managed transactions, adding the transactions into a to-be-executed linked list and waiting for execution; if the operation is not a commit operation, the transaction ID and the involved table ID of the operation are extracted, and then sort management is performed by a combination key of the transaction ID and the table ID.
Wherein, because a combination key of a transaction ID and a table ID is adopted to manage the transaction, the transaction related to a plurality of table operations in the source database is scattered into a plurality of small transactions; when a commit operation of the transaction is received, all scattered small transactions in the transaction need to be found, and all the small transactions are added to the to-be-executed linked list to wait for execution.
And 104, taking out a transaction from the to-be-executed linked list and adding the transaction to the to-be-merged linked list by the execution thread in charge of transaction warehousing, then selecting the transaction with the same table ID from the to-be-executed linked list according to the table ID of the transaction, and adding the selected transaction with the same table ID to the to-be-merged linked list.
This step corresponds to step 201 in embodiment 1, the purpose of which is to select the merged transaction. The selected transaction is moved out of the to-be-executed linked list, so that the next time the merged transaction is selected, the transaction which is not merged last time can continue the action. The number of merged transactions to be selected may be determined based on a size value of the merged transactions set in advance. If the number of the operations after the merging is set to be N, counting the operands of the merged transaction when the transaction is selected; when the operand reaches or exceeds N, the selection is stopped. Here, N affects the effect of the operation merging if set too small, and affects the parallelism between the execution threads if set too large, so the value of N needs to be determined according to the synchronization environment, and is not particularly limited herein.
In step 105, an operation container is initialized. After the execution thread selects the merged transaction, the execution thread needs to continue the merging operation between the transactions, and for the convenience of operation, an operation container is initialized to be ready for storing the transaction operation generated by merging.
And 106, taking out a transaction A from the transaction linked list to be merged, and preparing to merge the operation in the transaction into the operation container.
The transaction a may refer to any transaction in a transaction linked list to be merged, and is preferably extracted sequentially according to the sequence of the submitted LSNs of the transactions from small to large during extraction. Since each transaction may involve multiple operations, an operation chain table may be initialized for storing operations requiring batch consolidation for the convenience of subsequent batch consolidation operations. If all the transactions are merged, namely the transactions in the to-be-merged transaction linked list are all completed, directly jumping to the last step 113, and sequentially extracting the operation execution from the operation container.
Step 107, a commit LSN is taken from the merged transaction linked list and used as the current segment LSN.
For any transaction A, traversing and taking out each submitted LSN in the merged transaction linked list when merging is carried out, namely sequentially taking each submitted LSN as a segmentation LSN, and preferentially and sequentially extracting according to the sequence from small to large of the submitted LSNs when the submitted LSNs are taken out. During extraction, if the merged transaction linked list is empty or the transaction a has traversed the commit LSN in the merged transaction linked list, adding the remaining operations in the transaction a to the operation container in sequence, and after the addition is completed, continuously taking out the next transaction from the transaction linked list to be merged in sequence, namely jumping to step 106 to continue execution; otherwise, the step 108 is executed to proceed the operation merging.
When the operation of the first transaction is merged into the operation container (namely, the transaction A is the first transaction in the to-be-merged transaction linked list), the merged transaction linked list is an empty linked list, so the operation is directly plugged into the operation container in sequence in an operation adding mode, and then the submission LSN of the first transaction is added into the merged transaction linked list; when the transaction A finishes traversing the merged transaction linked list, the LSN representing the subsequent operation of the transaction A is larger than the LSN submitted last in the merged transaction linked list, and then the operations can not be merged with the operation of the previous transaction, so the operations are directly plugged into the operation container in sequence in the form of operation addition.
Step 108, taking out an operation from the transaction A, and judging whether the operation type of the current operation is a submitting operation; if the operation is a commit operation, the commit operation is added to the operation container, the corresponding commit LSN is added to the merged transaction linked list, and the next transaction is continuously taken out from the to-be-merged transaction linked list in sequence, namely, the step 106 is skipped; if not, execution continues with step 109.
Step 109, if the operation is not a commit operation, determine whether the LSN of the current operation is less than the split LSN. If the currently operated LSN is less than the split LSN, proceed to step 110; if the LSN of the current operation is greater than the split LSN, then execution continues with step 112.
Step 110, if the LSN of the current operation is less than the segmented LSN, judging whether the operation type of the current operation is consistent with the operation type of the operation in the operation linked list; if the operation is consistent with the operation list, adding the current operation into the operation list, and continuously extracting the next operation from the transaction A in sequence, namely jumping to step 108 for continuous execution; if not, execution continues down to step 111.
When the transactions are executed in parallel, the source database may separate operations executed in batches in some transactions from operations of other transactions in the log stream, and use the operation linked list to cache consecutive operations in the same transaction, so that batch operations that are not consecutive in the log stream but are consecutive in the transaction may be merged.
And step 111, if the operation type of the current operation is not consistent with the operation type of the operation in the operation linked list, positioning the operation container by using the partition LSN, and searching forward from the positioning position whether the operation with the same operation type as the operation in the operation linked list (namely the current operation to be merged) exists or not until the position of the last merging operation or the submitting operation of the last transaction is searched.
If the operation with the same operation type as the operation in the operation linked list is searched, the operation in the operation linked list is merged to the searched position; if the search is not completed, combining the operation in the operation linked list to the position where the search is stopped; here, the merging position is also recorded, and the position should be stopped when the forward search is merged again. And emptying the operation linked list after the merging is finished, adding the current operation into the operation linked list, and continuously extracting the next operation from the transaction A in sequence, namely jumping to the step 108 to continue executing.
It should be noted that, when the first operation of the transaction a searches for a specified position in the operation container, the operations in the interval and the first operation of the transaction a are logically conflict-free from each other and can be merged to any position in the interval from the beginning of positioning to the beginning of the commit operation of the previous transaction; once the operations are merged, the location searched forward when the subsequent operations are merged again cannot cross the location merged last time, or the operation sequence of the current operation in the transaction A is destroyed.
Step 112, if the LSN of the current operation is larger than the segmented LSN, judging whether the operation exists in the current operation linked list; if so, using the partition LSN to locate in the operation container, and searching forward from the location whether the operation of the same operation type as the operation in the operation linked list exists, and searching until the position of the last merge operation or the commit operation of the last transaction.
If the operation with the same operation type as the operation in the operation linked list is searched, the operation in the operation linked list is merged to the searched position; if the search is not completed, combining the operation in the operation linked list to the position where the search is stopped; here, the merging position is also recorded, and the merging forward search should be stopped at the next time; and clearing the operation linked list after the merging is finished, continuously taking a submitted LSN from the merged transaction linked list in sequence, taking the submitted LSN as a new segmentation LSN, and continuously judging whether the LSN of the current operation is smaller than the new segmentation LSN. When taking and submitting the LSN, if the end of the merged transaction linked list is reached, adding the rest operations in the transaction A into the operation container, and continuously taking out the next transaction from the to-be-merged transaction linked list in sequence, namely jumping to the step 106 to be continuously executed.
Step 113, after the execution thread completes the transaction combination, extracting the operation from the operation container in sequence for execution; this step corresponds to step 203 in example 1.
When the operation reaches step 112, all the transactions in the to-be-merged transaction linked list have already been merged (i.e. all the transactions in the to-be-merged transaction linked list are merged into one large transaction), the operations involved in each transaction have also been stored in the operation container, and the same operations are grouped together. Then, when the operations are extracted from the operation container in sequence, the operations with the same operations can be cached in the linked list by adopting the operation linked list and then executed in batch, and the submission operation between the two operations is ignored; and submitting the merged large transaction until all operations in the operation container are taken out and executed, and finishing warehousing.
The principle of the transaction merging execution method provided by the embodiment of the invention is that a series of operation intervals are marked in an operation container by taking the submitted LSN of each transaction to be merged as a partition boundary; during the merging operation, the LSN of the operation is positioned to the corresponding operation interval, and then the operation finds the operation position of the same table and the same operation type in the interval to which the operation belongs, and the same operations are merged together. Therefore, when the transaction combination is completed, the combination rate is higher than that of a transaction superposition combination mode, the execution times are fewer, and the synchronization performance is improved.
During merging operation, when the current operation is merged, the interval of upward movement of the operation after the positioning by the split LSN is limited by the previous merging position, if the transactions of the source end are not scattered according to the table ID, the transactions are merged, and the interval of upward movement of the current operation is influenced by the merging positions of other table operations in the transaction; after the table ID is used to scatter the source transaction, the merged transactions are all operations for the same table, and the operations of the same type are arranged more closely when the operations are merged.
The basic steps of the transaction merge execution method are explained as follows:
firstly, the operation sequence recorded in the log stream of the database can directly reflect the execution sequence of the operation of each transaction in the database, and the commit operation in the log stream is used as a boundary to reflect the parallelism of the operation of each transaction in the database, the operation of two transactions is bounded by the first commit operation in the log stream, the previously overlapped part of the operation can represent that the operation is executed in parallel in the source database, and the operation of the subsequent single transaction can possibly access the data of the previous transaction when the source database runs. Then the two transactions are merged and, if a simple overlap-add is used, execution can only be merged if the last operation of the first committed transaction and the first operation of the later committed transaction are the same. The invention uses the commit operation of the first committed transaction in the log stream as the boundary, the later committed transaction searches for a proper position in the prior committed transaction operation of the operation before the boundary, and the operation before the later committed transaction boundary is inserted according to a certain rule, so that the operations and the prior committed transaction operation are merged and executed, thereby improving the merging rate of the operation after the transaction is merged.
Secondly, on the premise that the atomicity of the transactions is not guaranteed, the transactions with a plurality of table operations are scattered by using the table IDs, then the transactions with the same table IDs are selected for combination by using the table IDs, the combination position influence caused by the operations of other tables can be effectively prevented when the transactions are combined, and the operation combination rate can be further improved.
Example 3:
on the basis of the foregoing embodiment 1 and embodiment 2, the embodiment of the present invention provides a specific example of a transaction merge execution method based on log resolution synchronization, so as to better understand the whole transaction merge execution process.
In this particular embodiment, assume that both the source and destination databases now have tables T1(ID VARCHAR) and T2(ID VARCHAR).
The source application has three transactions, and the following operations are performed on tables T1 and T2:
TRX1:INSERT INTO T1(ID)VALUES('TRX1_T1_1')；
TRX2:INSERT INTO T1(ID)VALUES('TRX2_T1_10')；
TRX1:INSERT INTO T1(ID)VALUES('TRX1_T1_2')；
TRX1:UPDATE T1 SET ID＝'TRX1_T1_2'WHERE ID＝'TRX1_T1_2'；
TRX3:INSERT INTO T2(ID)VALUES('TRX3_T2_300')；
TRX3:INSERT INTO T1(ID)VALUES('TRX3_T1_30')；
TRX1:COMMIT；
TRX2:INSERT INTO T1(ID)VALUES('TRX2_T1_20')；
TRX3:UPDATE T1 SET ID＝'TRX3_T1_30'WHERE ID＝'TRX3_T1_30'；
TRX2:UPDATE T1 SET ID＝'TRX2_T1_20'WHERE ID＝'TRX2_T1_20'；
TRX2:COMMIT；
TRX3:UPDATE T1 SET ID＝'TRX3_T1_10'WHERE ID＝'TRX3_T1_10'；
TRX3:COMMIT；
the sequence of the above operations is as shown in table 1 after the log receiving thread receives it.
Table 1:
Figure BDA0002514059800000161
Figure BDA0002514059800000171
with reference to the execution steps shown in fig. 2, the transaction merging process in the embodiment of the present invention is specifically as follows:
step 301, scattering the transactions with multiple table operations according to the table IDs, and selecting multiple transactions with the same table ID to add to the to-be-merged transaction linked list. According to the selection rule, the details of the transactions in the chain table to be merged are { TRX1(T1), TRX2(T1), TRX3(T1) }, that is, three transactions TRX1, TRX2, and TRX3 in the chain table to be merged are directed to the operation (or small transaction) on table T1.
In step 302, an operating vessel is constructed. For convenience of presentation, the process of operation consolidation is explained here with a table as a container, which is shown in table 2.
Table 2:
LSN operation of Remarks for note
In step 303, the first transaction TRX1 is fetched from the to-be-merged linked list, and its operation information is shown in table 3.
Table 3:
transaction ID Operation of LSN
TRX1 INSERT INTO T1(ID)VALUES('TRX1_T1_1') 1
TRX1 INSERT INTO T1(ID)VALUES('TRX1_T1_2') 3
TRX1 UPDATE T1 SET ID＝'TRX1_T1_2'WHERE ID＝'TRX1_T1_2' 4
TRX1 COMMIT； 8
At this time, since the merged transaction list is empty, the operation order of the first transaction TRX1 is appended to the operation container, forming the operation surface as shown in table 4.
Table 4:
LSN operation of Remarks for note
1 INSERT INTO T1(ID)VALUES('TRX1_T1_1') TRX1
3 INSERT INTO T1(ID)VALUES('TRX1_T1_2') TRX1
4 UPDATE T1 SET ID＝'TRX1_T1_2'WHERE ID＝'TRX1_T1_2' TRX1
8 COMMIT； TRX1
The commit LSN is then added to the merged transaction linked list, forming the TRX1(LSN 8).
At step 304, the second transaction TRX2 is fetched from the to-be-merged linked list, and its operation information is shown in Table 5.
Table 5:
Figure BDA0002514059800000181
Figure BDA0002514059800000191
step 305, the commit LSN (LSN 8) of the first transaction TRX1 is taken from the merged transaction linked list and used as the first split LSN to split the operations in transaction TRX 2.
At step 306, the first INSERT operation is extracted from transaction TRX2, whose LSN (LSN 2) is smaller than the first split LSN, so that the operation is added to the operation chain table.
At step 307, a second INSERT operation is extracted from transaction TRX2, whose LSN (LSN 7) is smaller than the first split LSN, so that the operation is added to the operation chain table.
Step 308, extracting a third UPDATE operation from the transaction TRX2, where its LSN (LSN is 10) is larger than the first segment LSN, so that the operation chain table needs to be cleared; the operations in the operation chain table (i.e. the first two INSERT operations of the transaction TRX2, LSN ═ 2 and LSN ═ 7) are inserted into the operation container using the first split LSN (LSN ═ 8) located in the operation container and the location of the same operation type searched forward from the location, forming the operation plane layout as shown in table 6.
Table 6:
Figure BDA0002514059800000192
Figure BDA0002514059800000201
from Table 6, it can be seen that the first two INSERT operations are classified together into INSERT operations in the operation container that precede the first partition LSN. After the operation container is inserted, taking the next submitted LSN in the merged linked list; since the end of the merged linked list has been reached, the operations subsequent to transaction TRX2 (i.e., the last two operations, LSN 10 and LSN 11) are appended directly to the operation container, forming the operation face layout as shown in table 7.
Table 7:
LSN operation of Remarks for note
1 INSERT INTO T1(ID)VALUES('TRX1_T1_1') TRX1
3 INSERT INTO T1(ID)VALUES('TRX1_T1_2') TRX1
2 INSERT INTO T1(ID)VALUES('TRX2_T1_10') TRX2
7 INSERT INTO T1(ID)VALUES('TRX2_T1_20') TRX2
4 UPDATE T1 SET ID＝'TRX1_T1_2'WHERE ID＝'TRX1_T1_2' TRX1
8 COMMIT； TRX1
10 UPDATE T1 SET ID＝'TRX2_T1_20'WHERE ID＝'TRX2_T1_20' TRX2
11 COMMIT； TRX2
The commit LSN of transaction TRX2 is then added to the merged transaction linked list to form: { TRX1(LSN ═ 8), TRX2(LSN ═ 11) }. When used subsequently, LSN-8 is still used as the first segment LSN, and LSN-11 is used as the second segment LSN.
In step 309, the third transaction TRX3 is fetched from the to-be-merged linked list, and its operation information is shown in table 8.
Table 8:
transaction ID Operation of LSN
TRX3 INSERT INTO T1(ID)VALUES('TRX3_T1_30') 6
TRX3 UPDATE T1 SET ID＝'TRX3_T1_30'WHERE ID＝'TRX3_T1_30' 9
TRX3 UPDATE T1 SET ID＝'TRX3_T1_10'WHERE ID＝'TRX3_T1_10'； 12
TRX3 COMMIT； 13
At step 310, the commit LSN (LSN 8) of the first transaction TRX1 is taken from the merged transaction linked list and used as the first split LSN to split the operations in the transaction TRX 3.
In step 311, the first INSERT operation is extracted from transaction TRX3, whose LSN (LSN 6) is smaller than the first segment LSN, so that the operation is added to the operation chain table.
Step 312, extracting a second UPDATE operation from the transaction TRX3, whose LSN (LSN ═ 9) is greater than the first segment LSN, so that the operation list needs to be cleared; the operation in the operation list (i.e. the first INSERT operation of the transaction TRX3, LSN ═ 6) is inserted into the operation container using the first partition LSN (LSN ═ 8) located in the operation container and searching forward from the location to the location of the same operation type, resulting in the operation face layout as shown in table 9.
Table 9:
Figure BDA0002514059800000211
Figure BDA0002514059800000221
the current operation (i.e., the second UPDATE operation of transaction TRX3, LSN 9) is then added to the operation chain table.
In step 313, the subsequent operation is extracted from the transaction TRX3, and at this time, the LSN of the current operation needs to be compared with the second segment LSN (LSN is 11), which may specifically refer to the merge rule of the transaction TRX2, and is not described herein again. After merging, the operation layout shown in table 10 is formed.
Table 10:
Figure BDA0002514059800000222
Figure BDA0002514059800000231
in step 314, after the three transactions in the to-be-merged transaction linked list are merged, the same operation may be merged when the whole large transaction is executed, so as to form the execution mode shown in table 11.
Table 11:
Figure BDA0002514059800000232
Figure BDA0002514059800000241
as can be seen from the operation layout after merging shown in table 11, the originally independent INSERT operations in the transaction TRX2 and the transaction TRX3 are executed in batch, and a significant increase in the merging rate can be seen by comparing with the operation layout after merging in an overlay manner. The operation layout after merging in the superposition manner is shown in table 12.
Table 12:
Figure BDA0002514059800000242
as can be seen by comparing the two operation layouts in table 11 and table 12, the number of times of execution after merging is 2 times by using the scheme of the embodiment of the present invention, and the number of times of execution is 6 times by using the scheme of superposition and merging.
In step 315, after the merging of the first group of transactions (i.e., the transactions for table T1) is completed, the transactions { TRX3(T2) } remain, and the merging execution is continued according to the above principle, and the specific process is not described again.
It should be noted that, the embodiment of the present invention is described by taking three transactions TRX1, TRX2, TRX3 and two tables T1 and T2 as examples, but the present invention is not limited thereto. When the number of related transactions and the number of tables change, the specific transaction merging execution process still refers to the steps in the above embodiments, which are not described herein again.
Example 4:
on the basis of the transaction merge execution method based on log parsing synchronization provided in embodiments 1 to 3, the present invention further provides a transaction merge execution device based on log parsing synchronization, which can be used for implementing the method, as shown in fig. 3, is a schematic diagram of a device architecture in an embodiment of the present invention. The transaction merge execution device based on log resolution synchronization of the present embodiment includes one or more processors 21 and a memory 22. In fig. 3, one processor 21 is taken as an example.
The processor 21 and the memory 22 may be connected by a bus or other means, and fig. 3 illustrates the connection by a bus as an example.
The memory 22, as a non-volatile computer-readable storage medium for a transaction merge execution method based on log resolution synchronization, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the transaction merge execution method based on log resolution synchronization in embodiments 1-3. The processor 21 executes various functional applications and data processing of the transaction merge execution device based on log resolution synchronization by executing the nonvolatile software program, instructions and modules stored in the memory 22, that is, implements the transaction merge execution method based on log resolution synchronization of embodiments 1 to 3.
The memory 22 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 22 may optionally include memory located remotely from the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The program instructions/modules are stored in the memory 22, and when executed by the one or more processors 21, perform the transaction merge execution method based on log resolution synchronization in embodiments 1-3, for example, perform the steps illustrated in fig. 1 and fig. 2 described above.
Those of ordinary skill in the art will appreciate that all or part of the steps of the various methods of the embodiments may be implemented by associated hardware as instructed by a program, which may be stored on a computer-readable storage medium, which may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (10)

1. A transaction merging execution method based on log analysis synchronization is characterized by comprising the following steps:
scattering the affairs with a plurality of table operations according to the table IDs, and selecting a plurality of affairs with the same table IDs to be added to the affair linked list to be merged;
sequentially taking out all the transactions from the transaction linked list to be merged for merging, adding the submission LSN of the transaction to the merged transaction linked list when merging one transaction, and storing the operation generated after merging to an operation container; when the ith transaction is taken out from the transaction linked list to be merged, i-1 submitted LSNs in the merged transaction linked list are used as partition boundaries, and i-1 operation intervals are marked out from an operation container; sequentially taking out operations from the ith transaction, positioning to a corresponding operation interval in an operation container through the LSN of the current operation, finding an operation position with the same operation type as the current operation in the corresponding operation interval, and inserting the current operation to the position;
after the combination of the transactions is completed, extracting the operation execution from the operation container in sequence; the method comprises the following steps that a submitting operation clamped between two operations is omitted for execution, and the same operations are combined and then executed in batch;
and the transaction linked list to be merged and the objects in the merged transaction linked list are stored according to the size sequence of the submitted LSNs of the transactions.
2. The log resolution synchronization-based transaction merge execution method of claim 1, wherein before the selecting the plurality of transactions of the same table ID to add to a to-be-merged transaction linked list, the method further comprises:
after reading the logs from the source database, the source database synchronization system analyzes the logs according to the order of capturing the logs and sends the analyzed operations to the destination database synchronization system in order;
after receiving the operation of the source end, the target end database synchronization system carries out classification management according to the transaction ID and the table ID of the operation, and adds the transaction with the same transaction ID to a transaction linked list to be executed after receiving the transaction submission operation;
and storing the transactions in the to-be-executed transaction linked list according to the size sequence of transaction submission LSN.
3. The transaction merging execution method based on log parsing synchronization of claim 2, wherein the selecting of the multiple transactions with the same table ID to be added to the transaction linked list to be merged specifically includes:
and the execution thread takes out a transaction from the to-be-executed linked list and adds the transaction to the to-be-merged linked list, selects the transaction with the same table ID from the to-be-executed linked list according to the table ID of the transaction, and adds the selected transaction with the same table ID to the to-be-merged linked list.
4. The transaction merge execution method based on log resolution synchronization according to claim 1, wherein for any transaction a in the transaction linked list to be merged, the merging process specifically includes:
taking out the transaction A from the transaction linked list to be merged, taking a submitted LSN from the merged transaction linked list, and taking the submitted LSN as the current segmented LSN;
taking out an operation from the transaction A, and judging whether the operation type of the current operation is a commit operation or not; if the operation is a commit operation, the commit operation is added to the operation container, the corresponding commit LSN is added to the merged transaction linked list, and the next transaction is continuously taken out from the to-be-merged transaction linked list in sequence; if not, determining whether the LSN of the current operation is less than the split LSN.
5. The transaction merge execution method based on log parsing synchronization of claim 4, wherein an operation chain table is used to store the operations that need to be merged in batch during the merging process, and if the LSN of the current operation is less than the split LSN, it is determined whether the operation type of the current operation is consistent with the operation type of the operation in the operation chain table; if so, adding the current operation to the operation linked list, and sequentially continuing to extract the next operation from the transaction A.
6. The transaction merge execution method based on log resolution synchronization of claim 5, wherein if the operation type of the current operation is not consistent with the operation type of the operation in the operation linked list, the split LSN is used to locate in the operation container, and whether the operation of the same operation type as the operation in the operation linked list exists is searched from the location forward until the position of the last merge operation or the commit operation of the last transaction;
if the operation with the same operation type as the operation in the operation linked list is searched, the operation in the operation linked list is merged to the searched position; if the search is not completed, combining the operation in the operation linked list to the position where the search is stopped;
and emptying the operation linked list after the combination is finished, adding the current operation into the operation linked list, and continuously extracting the next operation from the transaction A in sequence.
7. The transaction merge execution method based on log parsing synchronization of claim 4, wherein an operation linked list is used to store operations that need to be merged in batch during merging, and if the LSN of the current operation is greater than the split LSN, it is determined whether there is an operation in the operation linked list;
if so, using the partition LSN to locate in the operation container, and searching forward from the location whether the operation of the same operation type as the operation in the operation linked list exists, and searching until the position of the last merge operation or the commit operation of the last transaction.
8. The transaction merge execution method based on log resolution synchronization of claim 7, wherein if an operation of the same operation type as the operation in the operation linked list is searched, the operation in the operation linked list is merged to the searched position; if the search is not completed, combining the operation in the operation linked list to the position where the search is stopped;
clearing the operation linked list after the merging is finished, continuously taking a submitted LSN from the merged transaction linked list in sequence, taking the submitted LSN as a new segmentation LSN, and continuously judging whether the LSN currently operated is smaller than the new segmentation LSN;
and if the tail end of the merged transaction linked list is reached when the submitted LSN is taken, adding the rest operations in the transaction A into the operation container, and continuously taking out the next transaction from the transaction linked list to be merged in sequence.
9. The log resolution synchronization-based transaction merge execution method of any one of claims 4 to 8, wherein when fetching commit LSNs from the merged transaction linked list, if the merged transaction linked list is empty, or if transaction a has traversed over commit LSNs in the merged transaction linked list, then the remaining operations in transaction a are sequentially appended to the operation container; and after the addition is finished, continuously taking out the next transaction from the transaction linked list to be merged in sequence.
10. A transaction merge execution apparatus based on log resolution synchronization, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected through a data bus, and the memory stores instructions executable by the at least one processor, and the instructions are configured to complete the transaction merge execution method based on log resolution synchronization according to any one of claims 1 to 9 after being executed by the processor.
CN202010470229.7A 2020-05-28 2020-05-28 Transaction merging execution method and device based on log analysis synchronization Active CN111930693B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010470229.7A CN111930693B (en) 2020-05-28 2020-05-28 Transaction merging execution method and device based on log analysis synchronization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010470229.7A CN111930693B (en) 2020-05-28 2020-05-28 Transaction merging execution method and device based on log analysis synchronization

Publications (2)

Publication Number Publication Date
CN111930693A true CN111930693A (en) 2020-11-13
CN111930693B CN111930693B (en) 2024-02-06

Family

ID=73316482

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010470229.7A Active CN111930693B (en) 2020-05-28 2020-05-28 Transaction merging execution method and device based on log analysis synchronization

Country Status (1)

Country Link
CN (1) CN111930693B (en)

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1387269A1 (en) * 2002-08-02 2004-02-04 Hewlett Packard Company, a Delaware Corporation Backup system and method of generating a checkpoint for a database
US20040030703A1 (en) * 2002-08-12 2004-02-12 International Business Machines Corporation Method, system, and program for merging log entries from multiple recovery log files
CN101551822A (en) * 2009-05-27 2009-10-07 北京星网锐捷网络技术有限公司 Method and apparatus for copying database
CN102156720A (en) * 2011-03-28 2011-08-17 中国人民解放军国防科学技术大学 Method, device and system for restoring data
US20150379062A1 (en) * 2014-06-26 2015-12-31 Amazon Technologies, Inc. Multi-database log with multi-item transaction support
US20160171002A1 (en) * 2014-12-12 2016-06-16 International Business Machines Corporation Merging database operations for serializable transaction execution
US9552242B1 (en) * 2013-09-25 2017-01-24 Amazon Technologies, Inc. Log-structured distributed storage using a single log sequence number space
CN106777270A (en) * 2016-12-28 2017-05-31 中国民航信息网络股份有限公司 A kind of Heterogeneous Database Replication parallel execution system and method based on submission point time line locking
CN106844507A (en) * 2016-12-27 2017-06-13 星环信息科技（上海）有限公司 A kind of method and apparatus of data batch processing
CN107026880A (en) * 2016-02-01 2017-08-08 北京京东尚科信息技术有限公司 Method of data synchronization and device
CN107077492A (en) * 2014-09-10 2017-08-18 亚马逊科技公司 The expansible transaction management based on daily record
US20180144015A1 (en) * 2016-11-18 2018-05-24 Microsoft Technology Licensing, Llc Redoing transaction log records in parallel
CN109101627A (en) * 2018-08-14 2018-12-28 交通银行股份有限公司 heterogeneous database synchronization method and device
CN109657007A (en) * 2018-12-17 2019-04-19 武汉达梦数据库有限公司 Database in phase based on asynchronous submission executes method and apparatus parallel
CN110262929A (en) * 2018-08-13 2019-09-20 武汉达梦数据库有限公司 A kind of method guaranteeing duplication transaction consistency and corresponding reproducing unit
CN110597663A (en) * 2019-09-16 2019-12-20 阿里巴巴集团控股有限公司 Transaction processing method and device
CN111143389A (en) * 2019-12-27 2020-05-12 腾讯科技（深圳）有限公司 Transaction execution method and device, computer equipment and storage medium

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1387269A1 (en) * 2002-08-02 2004-02-04 Hewlett Packard Company, a Delaware Corporation Backup system and method of generating a checkpoint for a database
US20040030703A1 (en) * 2002-08-12 2004-02-12 International Business Machines Corporation Method, system, and program for merging log entries from multiple recovery log files
CN101551822A (en) * 2009-05-27 2009-10-07 北京星网锐捷网络技术有限公司 Method and apparatus for copying database
CN102156720A (en) * 2011-03-28 2011-08-17 中国人民解放军国防科学技术大学 Method, device and system for restoring data
US9552242B1 (en) * 2013-09-25 2017-01-24 Amazon Technologies, Inc. Log-structured distributed storage using a single log sequence number space
US20150379062A1 (en) * 2014-06-26 2015-12-31 Amazon Technologies, Inc. Multi-database log with multi-item transaction support
CN107077492A (en) * 2014-09-10 2017-08-18 亚马逊科技公司 The expansible transaction management based on daily record
US20160171002A1 (en) * 2014-12-12 2016-06-16 International Business Machines Corporation Merging database operations for serializable transaction execution
CN107026880A (en) * 2016-02-01 2017-08-08 北京京东尚科信息技术有限公司 Method of data synchronization and device
US20180144015A1 (en) * 2016-11-18 2018-05-24 Microsoft Technology Licensing, Llc Redoing transaction log records in parallel
CN106844507A (en) * 2016-12-27 2017-06-13 星环信息科技（上海）有限公司 A kind of method and apparatus of data batch processing
CN106777270A (en) * 2016-12-28 2017-05-31 中国民航信息网络股份有限公司 A kind of Heterogeneous Database Replication parallel execution system and method based on submission point time line locking
CN110262929A (en) * 2018-08-13 2019-09-20 武汉达梦数据库有限公司 A kind of method guaranteeing duplication transaction consistency and corresponding reproducing unit
CN109101627A (en) * 2018-08-14 2018-12-28 交通银行股份有限公司 heterogeneous database synchronization method and device
CN109657007A (en) * 2018-12-17 2019-04-19 武汉达梦数据库有限公司 Database in phase based on asynchronous submission executes method and apparatus parallel
CN110597663A (en) * 2019-09-16 2019-12-20 阿里巴巴集团控股有限公司 Transaction processing method and device
CN111143389A (en) * 2019-12-27 2020-05-12 腾讯科技（深圳）有限公司 Transaction execution method and device, computer equipment and storage medium

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
张小强 等: "一种基于目录的软件事务性内存实现算法", 《计算机研究与发展》, no. 09, pages 1517 - 1523 *
张金玲: "基于事务合并优化的数据同步算法研究与应用", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 01, pages 138 - 743 *
胡爽 等: "内存数据库事务提交的关键技术与挑战", 《华东师范大学学报(自然科学版)》, no. 05, pages 18 - 26 *
钟容: "基于事务日志的数据同步模型研究", 《科技创新与应用》, no. 10, pages 17 - 18 *

Also Published As

Publication number Publication date
CN111930693B (en) 2024-02-06

Similar Documents

Publication Publication Date Title
US11003689B2 (en) Distributed database transaction protocol
EP3401804B1 (en) Adaptive query routing in a replicated database environment
EP3185143B1 (en) Decentralized transaction commit protocol
EP3185142B1 (en) Distributed database transaction protocol
US11010262B2 (en) Database system recovery using preliminary and final slave node replay positions
US20180067962A1 (en) Consistent execution of partial queries in hybrid dbms
US9881041B2 (en) Multiple RID spaces in a delta-store-based database to support long running transactions
JP5577350B2 (en) Method and system for efficient data synchronization
EP3413215A1 (en) Dynamic snapshot isolation protocol selection
CN110309233B (en) Data storage method, device, server and storage medium
US11416350B2 (en) Recovery of in-memory databases from log records
CN107665219B (en) Log management method and device
CN111930692B (en) Transaction merging execution method and device based on log analysis synchronization
WO2022002103A1 (en) Method for playing back log on data node, data node, and system
CN106897123B (en) Database operation method and device
JP2018526746A (en) Method and apparatus for optimizing database transactions
CN112035222B (en) Transaction operation merging execution method and device based on log analysis synchronization
CN114063922A (en) Method, device, equipment and medium for accelerating replication of master and slave library streams
CN112559626A (en) Synchronous method and synchronous system of DDL operation based on log analysis
WO2022206398A1 (en) Method and apparatus for reading data maintained in tree data structures
CN111858504B (en) Operation merging execution method based on log analysis synchronization and data synchronization system
CN105045563B (en) A kind of method for collision management for speculating nested software transaction storage
CN111858503B (en) Parallel execution method and data synchronization system based on log analysis synchronization
US10437812B2 (en) Information processing method, information processing device, and medium
CN111930693A (en) Transaction merging execution method and device based on log analysis synchronization

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province

Applicant after: Wuhan dream database Co.,Ltd.

Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province

Applicant before: WUHAN DAMENG DATABASE Co.,Ltd.

CB02 Change of applicant information
CB03 Change of inventor or designer information

Inventor after: Sun Feng

Inventor after: Huang Haiming

Inventor after: Yu Yuanlan

Inventor before: Sun Feng

Inventor before: Fu Quan

Inventor before: Huang Haiming

Inventor before: Yu Yuanlan

CB03 Change of inventor or designer information
GR01 Patent grant
GR01 Patent grant