CN117349371A

CN117349371A - Method and device for statically modifying data synchronization packet

Info

Publication number: CN117349371A
Application number: CN202311204116.2A
Authority: CN
Inventors: 孙峰; 彭青松; 刘启春; 陈江辉
Original assignee: Wuhan Dream Database Co ltd
Current assignee: Wuhan Dream Database Co ltd
Priority date: 2023-09-15
Filing date: 2023-09-15
Publication date: 2024-01-05

Abstract

The invention relates to the technical field of database synchronization, and provides a method and a device for statically modifying data synchronization packets. According to table information and new grouping configuration related to all current sub-transactions to be cut, the method and the device acquire the target grouping corresponding to the sub-transactions to be cut, determine a table to be modified of the current grouping configuration according to the table corresponding to the table information of the target grouping and the original grouping, and determine the active transactions to be cut of the corresponding source-side main transactions to be cut. According to the original group of the table to be modified and the target group of the table to be modified, constructing a virtual transaction of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding active transaction to be cut, and warehousing all sub-transactions of the active transaction to be cut. By setting the active transaction and the virtual transaction, the corresponding operations are put in storage according to the sequence of the operations in the main transaction at the target end, and the modification of the grouping configuration of the table to be modified among the groups is completed under the condition that the data synchronization is not affected.

Description

Method and device for statically modifying data synchronization packet

Technical Field

The present invention relates to the field of database synchronization technologies, and in particular, to a method and an apparatus for statically modifying a data synchronization packet.

Background

At present, in real-time synchronization of database data based on log analysis, operations in different groups are often synchronized after grouping the operations according to a table on which the operations depend in order to ensure the synchronization performance of the database and improve the parallelism. Wherein the table on which the operation is updated is a table on which the operation depends. In an actual production environment, packet configuration and data synchronization policies are often formulated individually according to specific services, so as to meet service requirements or improve data synchronization performance of individual packets. The service requirements often change over time, and the packet configuration of the table to be synchronized also needs to change with the service requirements, i.e. the packets divided by the operations depending on the relevant table to be synchronized are modified in the data synchronization process.

In the running process of the data synchronization service, the delay of each packet for data synchronization may be different, some packets are continuously subjected to data synchronization, and some packets may have larger delay due to the difference of synchronization efficiency, so that the packet configuration of the table to be modified can relate to the transaction dependence relationship of data synchronization between the packets; the table to be modified is a table to be synchronized which is obtained by comparing the grouping configuration before modification and the grouping configuration after modification and needs to modify the grouping configuration. If a table to be modified is set up in groups multiple times in a short time and there is a round-trip adjustment between groups of the same group number, this may result in a situation where a single master transaction involving the table to be modified is split into multiple parts and the split sub-transactions are spread among multiple groups or there are multiple split sub-transactions in a group. The cut sub-transactions belong to the same main transaction, the prior art cannot be executed according to the preset sequence of all operations in the main transaction, the updating sequence of the table to be synchronized cannot be ensured, the data of the table to be modified in the target-end database is very easy to be inconsistent with the data of the table to be modified in the source-end database, the consistency of the data in the database system is destroyed, and the operation error of the data synchronization service is caused.

In view of this, overcoming the drawbacks of the prior art is a problem to be solved in the art.

Disclosure of Invention

In view of the above-mentioned drawbacks or improvements of the prior art, the present invention provides a method and apparatus for statically modifying data synchronization packets, which aims to implement that, during data synchronization, the execution sequence of operations in a source-side master transaction is strictly observed, and the packet configuration of a table to be modified is modified without affecting data synchronization.

The invention adopts the following technical scheme:

in a first aspect, the present invention provides a method of statically modifying a data synchronization packet, comprising:

acquiring a target group corresponding to all sub-transactions to be cut according to table information and new group configuration related to all the current sub-transactions to be cut;

taking a table corresponding to table information of the target packet and the original packet which are different as a table to be modified;

constructing a virtual transaction of the table to be modified according to the original group of the table to be modified and the target group of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding active transaction to be cut, and warehousing all sub-transactions of the active transaction to be cut; wherein all sub-transactions of the active transaction to be cut include the corresponding sub-transaction to be cut and the corresponding virtual transaction.

Further, before the obtaining, according to the table information and the new packet configuration related to all the current sub-transactions to be cut, the target packet corresponding to the sub-transactions to be cut includes:

stopping the synchronous service, storing the original group of each original sub-transaction and the table information related to the original sub-transaction in all current active transactions to a check point file, and exiting the synchronous service after storing;

acquiring the new packet configuration, modifying and storing the current packet configuration of a system according to the new packet configuration so as to divide the packets according to the current packet configuration after restarting the synchronous service later;

restarting the synchronous service, and obtaining the original grouping and table information of each original sub-transaction in all active transactions according to the latest check point file;

and taking the original sub-transaction without the end mark in all the current active transactions as a sub-transaction to be cut, and obtaining the table information related to the sub-transaction to be cut.

Further, when the synchronization service is stopped, saving the original group of each original sub-transaction in all current active transactions and the table information related to the original sub-transaction to a check point file, and after saving, before exiting the synchronization service, the method comprises the following steps:

Receiving a log, and analyzing the log to obtain at least one operation, corresponding table information, a main transaction ID and an operation number;

judging whether a corresponding active transaction exists according to the main transaction ID, and creating the active transaction corresponding to the main transaction ID when the corresponding active transaction does not exist;

judging whether an original sub-transaction corresponding to the original group exists under the active transaction according to the table information, and creating the original sub-transaction when the original sub-transaction does not exist; partitioning the operation to the original sub-transaction;

the operation number of the last operation in each original sub-transaction is used as the commit number of the corresponding original sub-transaction;

and the active transaction receiving the commit message adds all the original sub-transactions into a committed linked list of a corresponding original group according to the sequence from small to large of the combination of the commit numbers of all the original sub-transactions in the active transaction and the commit LSN of all the original sub-transactions, and executes and stores all the original sub-transactions according to the committed linked list.

Further, the constructing a virtual transaction of the table to be modified according to the original group of the table to be modified and the target group of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding active transaction to be cut, and warehousing all sub-transactions of the active transaction to be cut includes:

Taking an active transaction corresponding to a sub-transaction to be cut related to the table to be modified as an active transaction to be cut, and adding an end mark for the sub-transaction to be cut;

constructing a sub-transaction belonging to the target group of the active transaction to be cut as a virtual transaction, and adding an end mark for the virtual transaction;

constructing a modification operation for dividing the table to be modified into the target group according to the original group and the target group, and adding the modification operation to the virtual transaction;

and continuing to receive the log, receiving a commit message, and warehousing all sub-transactions of the active transaction to be cut according to the commit message.

Further, the continuing to receive the log, receiving a commit message, and warehousing all sub-transactions of the active transaction to be cut according to the commit message includes:

receiving a commit message, taking a sub-transaction of an active transaction corresponding to the commit message as a sub-transaction to be put in storage, and judging whether the active transaction is an active transaction to be cut;

when the transaction is an active transaction to be cut, traversing all sub transactions to be put in the warehouse of the active transaction to be cut, and finding corresponding virtual transactions from the sub transactions to be put in the warehouse;

taking the commit LSN of the commit message as a commit LSN and a wait LSN of the virtual transaction; the submitting number of the virtual transaction is the operation number of the last operation in the active transaction to be cut;

Adding all sub-transactions to be put into a submitted linked list of a corresponding target group according to the sequence from small to large of the combination of the submitted numbers of all sub-transactions to be put and the submitted LSNs of all sub-transactions to be put;

and warehousing all the sub-transactions to be warehoused according to the submitted linked list and the waiting LSN.

Further, the warehousing all the sub-transactions to be warehoused according to the submitted linked list and the waiting LSN comprises:

sequentially judging whether the commit LSN of the sub-transaction to be put in storage in the committed chain table is smaller than or equal to the commit LSN of the corresponding target packet, and judging whether the commit number of the sub-transaction to be put in storage is smaller than or equal to the commit number of the target packet;

if yes, discarding the sub-transaction to be put in storage; if not, judging whether the sub-transaction to be put in storage is a virtual transaction or not;

when the virtual transaction is a virtual transaction, according to the commit LSN and the commit number corresponding to the original packet relied by the virtual transaction, all sub-transactions to be put in storage or not put in storage in the target packet to which the virtual transaction belongs are selectively put in storage;

after executing the sub-transaction to be put into the warehouse each time, taking the commit LSN of the sub-transaction to be put into the warehouse as the commit LSN of the corresponding target group, and taking the commit number of the sub-transaction to be put into the warehouse as the commit number of the target group.

Further, when the virtual transaction is a virtual transaction, selectively warehousing or not warehousing all sub-transactions to be warehoused in the target packet to which the virtual transaction belongs according to the commit LSN and the commit number corresponding to the original packet on which the virtual transaction depends includes:

sequentially judging whether the commit LSN corresponding to the original packet relied by the virtual transaction is larger than or equal to the wait LSN of the virtual transaction, and judging whether the commit number corresponding to the original packet is larger than or equal to the commit number of the virtual transaction;

if not, not warehousing all sub-transactions to be warehoused in the target group to which the virtual transaction belongs in the submitted linked list;

if yes, all sub-transactions to be put in the target group to which the virtual transaction belongs are put in storage according to the submitted linked list.

restarting the synchronous service, continuously receiving a log, and judging the type of operation corresponding to the log;

when the operation is a DML operation, dividing the DML operation into corresponding sub-transactions according to a main transaction ID, table information and grouping configuration of the DML operation; when the sub-transaction has added an end mark, the sub-transaction is a sub-transaction of an active transaction to be cut, a corresponding target sub-transaction is created according to the grouping configuration, and the operation number of the DML operation is used as the commit number of the target sub-transaction;

When the operation is a commit operation, finding a corresponding active transaction to be cut according to a log corresponding to the commit operation, adding a commit message for all sub-transactions to be put in storage of the active transaction to be cut, taking a commit LSN of the commit operation as a commit LSN of all sub-transactions to be put in storage, adding all sub-transactions to be put in storage to a committed linked list of a target group, and putting all sub-transactions to be put in storage according to the committed linked list.

Further, upon restart recovery of the synchronization service failure, wherein:

restoring the synchronized commit LSN, commit number and virtual transaction in each target group according to the latest checkpoint file;

when the sub-transaction is executed and put in storage after recovery, judging whether the commit LSN and the commit number corresponding to the executed sub-transaction are smaller than or equal to the commit LSN and the corresponding commit number of the last synchronized sub-transaction;

if yes, not executing the sub-transaction and warehousing; if not, executing the sub-transaction, and modifying the current grouping configuration and storing according to the modification operation in the virtual transaction.

In a second aspect, the present invention further provides an apparatus for implementing the method for statically modifying a data synchronization packet according to the first aspect, where the apparatus for statically modifying a data synchronization packet includes:

At least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method of statically modifying data synchronization packets according to the first aspect.

In a third aspect, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions for execution by one or more processors to perform the method of statically modifying data synchronization packets according to the first aspect.

Unlike the prior art, the invention has at least the following beneficial effects:

according to the table information and the new grouping configuration related to all the current sub-transactions to be cut, the invention acquires the target grouping corresponding to the sub-transactions to be cut, takes the table corresponding to the table information of the target grouping different from the original grouping as the table to be modified, realizes the determination of the table to be modified of the current grouping configuration, and determines the active transactions to be cut of the corresponding source-side main transactions to be cut. According to the original group of the table to be modified and the target group of the table to be modified, constructing a virtual transaction of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding active transaction to be cut, and warehousing all sub-transactions of the active transaction to be cut. By setting the active transaction and the virtual transaction, the corresponding operations are put in storage according to the sequence of the operations in the main transaction at the target end, and the modification of the grouping configuration of the table to be modified among the groups is completed under the condition that the data synchronization is not affected. The method solves the problems that when the grouping configuration is modified, the data of the table to be modified in the target end database is inconsistent with the data of the table to be modified in the source end database because the table to be modified cannot be updated according to the sequence in the main transaction, the consistency of the data in the database system is destroyed, and the operation error of the synchronous service is caused.

Drawings

In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings that are required to be used in the embodiments of the present invention will be briefly described below. It is evident that the drawings described below are only some embodiments of the present invention and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is a flow chart of a method for statically modifying a data sync packet according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of step 30 according to an embodiment of the present invention;

FIG. 3 is a schematic illustration of a specific flow chart of step 304 according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart of step 3045a according to the embodiment of the present invention;

FIG. 5 is a schematic diagram showing another embodiment of step 304 of the present invention;

fig. 6 is a schematic diagram of an architecture of an apparatus for statically modifying a data synchronization packet according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

In the description of the present invention, the terms "inner", "outer", "longitudinal", "transverse", "upper", "lower", "top", "bottom", etc. refer to an orientation or positional relationship based on that shown in the drawings, merely for convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.

The terms "first," "second," and the like herein are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first", "a second", etc. may explicitly or implicitly include one or more such feature. In the description of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.

In addition, the technical features of the embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.

Example 1:

in a packet synchronization mode based on log analysis, a target database receives a log sent by a source database, analyzes the log to obtain at least one main transaction divided by the source database, each main transaction comprises at least one operation, splits and classifies the main transaction by taking a table as a unit to obtain sub-transactions belonging to different packets, namely classifying the operation to the corresponding packet according to the table on which the operation depends in the main transaction, and executing the sub-transactions in each packet in parallel by using multithreading to improve the data synchronization efficiency.

There is a need to modify the packet configuration of the correlation table during the operation of the synchronization service, the modification of the packet configuration being exemplified as follows: before modification, dividing the operations of the dependence table A and the dependence table B into a first group, and dividing the operations of the dependence table C into a second group; dividing the modified operation of the dependence table A into a first group, and dividing the operation of the dependence tables B and C into a second group; the table to be modified is table B.

When the grouping configuration of the tables to be modified is modified among the groupings in the prior art, only the operation of depending on the tables to be modified, which is received newly after modification, is realized, the operation is divided into sub-transactions belonging to the same main transaction in the different groupings before modification according to the new grouping configuration, but the operation which is divided into corresponding groupings before modification but is not yet subjected to warehousing is not divided again. If a certain table to be modified is adjusted in a packet configuration multiple times in a short time (for example, the operation of the dependent table a is divided into a first packet, then into a second packet, and finally into a third packet) and there is an adjustment back and forth between packets of the same group number (for example, the operation of the dependent table a is divided into the first packet, then into the second packet, and finally into the first packet), this may result in a situation that a single main transaction involving the table to be modified is cut into a plurality of sub-transactions in different packets in the target database, and the cut sub-transactions are scattered in the plurality of packets (for example, the main transaction 1 is cut into sub-transaction 1 in the first packet and sub-transaction 2 in the second packet), or there is a plurality of cut sub-transactions in a certain packet (for example, since the operation of the dependent table a is divided into the first packet, the second packet, then into the first packet, there is sub-transaction 1 and sub-transaction 3 in the first packet, and there is sub-transaction 1 in the second packet, and sub-transaction 1 in the sub-transaction 2 is divided into sub-transaction 1 and sub-transaction 2 in the order).

When the operations in each group are executed in parallel and put in storage, due to the difference in the synchronization efficiency of the sub-transactions in different groups, the operations belonging to different sub-transactions before and after the modification of the group configuration cannot be executed according to the sequence of all the operations preset in the main transaction, so that the to-be-synchronized table exists in the target-end database for synchronization, which is originally required to be updated according to the sequence in the main transaction (namely, after a certain operation 1 updates the to-be-synchronized table, the other operation 2 continuously updates the to-be-synchronized table based on the updated to-be-synchronized table of the operation 1), after the modification of the group configuration of the to-be-synchronized table, the sequential update cannot be guaranteed (namely, the situation that the to-be-synchronized table is updated based on the operation 2 after the update of the operation 1 is continuously updated is possible), the data of the to-be-modified table in the target-end database is inconsistent with the data of the to-be-modified table in the source-end database, the consistency of the data in the database system is destroyed, and the running error of the data synchronization service is caused.

Static modifications tend to modify the packet configuration of the synchronous service directly in the configuration file, modifying the packet configuration in bulk, and the tables needed for packet configuration modification are not known to the service. In static modification, the entire synchronization service needs to be stopped, and the synchronization service is stopped for a long time, so that the packet configuration is modified in large quantities. Because the whole synchronous service needs to be stopped, that is, the execution of the warehousing thread is also stopped, there is a high possibility that a large number of operations which are divided into corresponding groups before modification but are not yet executed for warehousing exist, when the operations are executed for warehousing according to the groups in parallel after the grouping configuration is modified, the number of operations for warehousing cannot be increased according to the sequence of the operations in the main transaction of the source end, and the number of operations for strictly adhering to the execution sequence in the main transaction cannot be increased in the sub transactions of the main transaction of the same source end.

To solve the foregoing problem, an embodiment of the present invention provides a method for statically modifying a data synchronization packet, as shown in fig. 1, including:

step 10: and acquiring a target group corresponding to the sub-transaction to be cut according to the table information and the new group configuration related to all the current sub-transactions to be cut.

The table information is table information of a table on which the corresponding operation depends in a log sent from the source terminal to the target terminal. The new grouping configuration is the grouping configuration obtained from the configuration file by the synchronous service after the user modifies the configuration file. The target packet is a packet created after the packet configuration modification. The method comprises the steps that a main transaction of a source terminal corresponds to an active transaction of a target terminal, the main transaction from the source terminal is cut into at least one sub-transaction, the active transaction is a set of at least one sub-transaction, and all the active transactions are sub-transactions which do not receive a corresponding commit operation; the sub-transaction to be cut is a sub-transaction to which no end tag is added.

Step 20: and taking a table corresponding to the table information of the target packet different from the original packet as a table to be modified.

Wherein the original packet is a packet created before modification of the packet configuration; and when the table to be modified is modified for the grouping configuration, modifying the divided grouping. Traversing the sub-transactions to be cut, and matching the grouping information of the transactions in the new grouping configuration according to the table information related in the sub-transactions to be cut.

Step 30: constructing a virtual transaction of the table to be modified according to the original group of the table to be modified and the target group of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding active transaction to be cut, and warehousing all sub-transactions of the active transaction to be cut; wherein all sub-transactions of the active transaction to be cut include the corresponding sub-transaction to be cut and the corresponding virtual transaction.

The warehousing is to execute a certain operation in the target end database so as to realize data synchronization. When the active transaction to be cut is configured for the current modification group, the active transaction of the table to be modified is involved, and the active transaction needs to be cut. The following describes the relationship between an active transaction to be cut and a sub-transaction to be cut, and a specific example of an active transaction is as follows:

A1

{

g1 grouping:

G1_TRX1[ Table: T1, end tag: 0, commit LSN:0, commit number: 1]

}

Since the sub-transaction g1_trx1 does not add an end tag, the sub-transaction g1_trx1 is the sub-transaction to be cut. And determining the target packet corresponding to the sub-transaction to be cut G1_TRX1 as a G2 packet according to the table information (the table T1) related to the sub-transaction to be cut G1_TRX1 and the new packet configuration (the table T1 exists in the G2 packet). The original packet of sub-transaction g1_trx1, i.e., the G1 packet, is acquired. Since the target packet (G2 packet) and the original packet (G1 packet) are different, it is determined that the table to be modified at this time is table T1. And searching the active transaction A1, and determining the active transaction A1 as the active transaction to be cut configured by the current modification group. At this time, there is a sub-transaction g1_trx1 to be cut in the active transaction A1 to be cut, and all sub-transactions of the active transaction A1 to be cut include only the sub-transaction g1_trx1 to be cut (the virtual transaction has not yet been constructed).

The method for statically modifying the data synchronization packet according to the embodiment of the invention at least can solve the following problems: firstly, the data synchronization problem related to the table to be modified before and after the modification of the grouping configuration needs to be solved; secondly, the grouping configuration of the current system needs to be modified, so that operations contained in the logs are divided according to the modified grouping configuration after the logs of the targets are received. Since the table to be modified is unknown, table information related to all current sub-transactions to be cut needs to be used, and the table with changed packet configuration is obtained by positioning corresponding packet information in the new packet configuration and comparing the packet information with the packet information before the modification of the packet configuration so as to determine the table to be modified. And searching out all active transactions related to the table to be modified from the current active transactions, and cutting sub-transactions related to the groups and constructing virtual transactions so as to restrict the execution sequence of the groups before and after the table to be modified.

According to the method for statically modifying the data synchronization packet, according to the table information and the new packet configuration related to all the current sub-transactions to be cut, the target packet corresponding to the sub-transactions to be cut is obtained, the table corresponding to the table information of the target packet and the table information of the original packet, which are different, is used as the table to be modified, the table to be modified of the current modified packet configuration is determined, and the active transactions to be cut corresponding to the source-side main transactions to be cut are determined. According to the original group of the table to be modified and the target group of the table to be modified, constructing a virtual transaction of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding active transaction to be cut, and warehousing all sub-transactions of the active transaction to be cut. By setting the active transaction and the virtual transaction, the corresponding operations are put in storage according to the sequence of the operations in the main transaction at the target end, and the modification of the grouping configuration of the table to be modified among the groups is completed under the condition that the data synchronization is not affected. The method solves the problems that when the grouping configuration is modified, the data of the table to be modified in the target end database is inconsistent with the data of the table to be modified in the source end database because the table to be modified cannot be updated according to the sequence in the main transaction, the consistency of the data in the database system is destroyed, and the operation error of the synchronous service is caused.

According to the embodiment of the invention, the log analysis service is arranged at the source end, the analyzed operation is independently numbered by taking the main transaction of the source end as a unit, the sequential incremental performance of the operation numbers of the operation in the main transaction is ensured, and the log corresponding to the operation is filled with the table information related to the operation. Wherein, the person skilled in the art can self-specify the specific implementation mode of the operation number and the table information according to the specific data synchronization scene. The target synchronization service receives the operation in the log sent by the source, classifies the operation into corresponding active transactions according to the ID of the main transaction, extracts the table information related in the operation, locates the packet to which the operation belongs through the table information, and constructs the sub-transaction corresponding to the main transaction under the active transaction by combining the packet information.

In order to better illustrate the method for statically modifying the data synchronization packet according to the present invention, the preparation work required to be done before modifying the packet configuration according to the embodiment of the present invention is further described below, specifically, a log is received, and the log is parsed to obtain at least one operation, corresponding table information, a master transaction ID and an operation number. Judging whether a corresponding active transaction exists according to the main transaction ID, and creating the active transaction corresponding to the main transaction ID when the corresponding active transaction does not exist. Judging whether an original sub-transaction corresponding to the original group exists under the active transaction according to the table information, and creating the original sub-transaction when the original sub-transaction does not exist; the operation is divided into the original sub-transactions. Wherein, the operation number of the last operation in each original sub-transaction is taken as the commit number of the corresponding original sub-transaction.

And adding all the original sub-transactions into a submitted linked list of a corresponding original group according to the sequence from small to large of the combination of the submitted numbers of all the original sub-transactions in the active transactions and the submitted LSNs (Log sequence number, log sequence numbers) of all the original sub-transactions, and executing and warehousing all the original sub-transactions according to the submitted linked list.

When the method for statically modifying the data synchronization packet in the embodiment of the invention carries out log analysis on the source end, the operations in the main transaction of each source end are numbered according to the sequence. Since a transaction is a collection of operations, the operations in the same transaction either commit together or do not commit together. Therefore, the commit LSNs of the target terminal transactions corresponding to the same source main transaction are the same, and the operation sequences are required to be ordered according to the operation numbers in the group so as to strictly distinguish the operation sequences in the source main transaction. When the sub-transaction is executed at the target end, the consistency of the data after fault recovery is solved through the operation number of the last operation of the transaction. After the target receives the log, at least one operation in each main transaction of the source is grouped according to a table on which the operation is dependent according to the operation contained in the log, and the operation is divided into a plurality of sub-transactions. A single master transaction resulting from source-side parsing of the log involves the operation of multiple tables that are partitioned into different groupings when grouped at the target-side, which creates an overall transaction (i.e., an active transaction) to categorize the operation of the master transaction. Creating a plurality of sub-transactions under the active transaction, wherein each sub-transaction corresponds to a group, classifying the operations belonging to the same group under the sub-transaction, and recording the operation number of the last operation of each sub-transaction as the commit number of the sub-transaction. By setting the active transaction, the main transaction of the source end is managed at the target end by taking the active transaction as a set, so that the sub-transactions positioned in each group can be found and put in storage according to the sequence of operation in the main transaction under the condition that the main transaction is divided according to a dependent table and the group configuration is divided into a plurality of sub-transactions after modification. Wherein the table updated by the master transaction is a table relied on by the master transaction.

After the grouping configuration of the table to be modified is modified, the sub-transactions related to the related grouping are cut, and the sub-transactions before modification and after modification form independent sub-transactions respectively. The method comprises the steps that a target end receives logs which are all logs generated by a source end when operations are executed, so that logs of operations which are being executed and not submitted are contained, and in order to keep consistency of a database, synchronization is carried out based on the fact that the source end submits main transactions, namely when the target end receives corresponding logs, only the operations which are not submitted by the source end are divided into active transactions and corresponding groups, and when the target end receives submitted operations, data synchronization of the operations is carried out.

After the operation of the table to be modified is cut from one packet to another, sub-transactions belonging to the two packets form a dependency. If a certain table a in the database is updated by an operation in the transaction 1, the operation in the subsequent transaction 2 can complete updating of the table a based on the updated table a, then the transaction 2 is referred to as depending on the transaction 1, or a transaction dependency exists between the transaction 1 and the transaction 2. Because operations of the same sub-transaction must be guaranteed to be performed in strict order of operation numbers of operations in the main transaction, operations cut to the target packet must wait for the sub-transaction to complete binning in the original packet before binning can begin. To solve this problem, when an active transaction (corresponding to a source-side main transaction) is cut, a virtual transaction for checking the execution of an original packet is added to a target packet corresponding to the active transaction, and the target packet checks the execution completion of the original packet of the related table by executing the virtual transaction, and only when a condition is satisfied, the sub-transaction belonging to the target packet can start to execute.

In order to obtain the table information and the new packet configuration involved in all sub-transactions to be cut currently, before said step 10, it comprises:

stopping the synchronous service, storing the original grouping of each original sub-transaction and the table information related to the original sub-transaction in all the current active transactions to a check point file, and exiting the synchronous service after storing. Since the static modification cuts the large-batch modification grouping when the active transaction, in order to avoid that new operation is divided into the active transaction in the process, the active transaction is changed, and the grouping information in the last check point are compared to determine the table to be modified, the synchronous service is needed to be stopped before modification, and then the active transaction can be cut.

Because the table to be modified is unknown, after the synchronous service is stopped, a check point action is executed first, an active transaction is recovered from the check point, then the table information related to the transaction in the active transaction is used for positioning the grouping information of the table in the new grouping configuration, and the grouping information is compared with the grouping information in the last check point to obtain the table to be modified, wherein the grouping configuration of the table is changed. If the sub-transaction is matched, the obtained group is the same as the group stored by the check point, and no change is needed; if the packet obtained by sub-transaction matching is different from the packet stored by the check point, in step 20, the table corresponding to the table information of the target packet and the original packet is used as the table to be modified. And searching out all active transactions related to the table to be modified from the current active transactions, and cutting sub-transactions related to the groups and constructing virtual transactions so as to restrict the execution sequence of the groups before and after the table to be modified.

An example of an active transaction provided by an embodiment of the present invention is as follows:

A1

{

g1 grouping:

G1_TRX1[ Table: T1, end tag: 0, commit LSN:0, commit number: 1]

}

When an operation with LSN 21 is received, an active transaction A1 with ID 1 is created through the main transaction ID, at the moment T1 belongs to the G1 group, a sub-transaction G1_TRX1 belonging to the G1 group is created on the active transaction, the operation is added to the sub-transaction, and the number of the operation is taken as the commit number of the transaction.

A modification operation of grouping T1 is prepared, changing from the G1 grouping of the original grouping to the G2 grouping of the target grouping. The method comprises the steps that (a synchronous service comprises a log receiving function), the receiving of a source log is stopped, an active transaction is traversed, an original packet of each original sub-transaction in all current active transactions and table information related to the original sub-transaction are stored in a check point file, the active transaction stored in the check point is A1, the active transaction has a sub-transaction G1_TRX1, the packet is a G1 packet, and the table information related to the sub-transaction is T1. And exiting the synchronization service.

And acquiring the new packet configuration, and modifying and storing the current packet configuration of a system according to the new packet configuration so as to divide the packets according to the current packet configuration after restarting the synchronous service. For example, the sync packet configuration of T1 is modified from G1 packets (original packets) to G2 packets (target packets).

Restarting the synchronous service, and obtaining the original grouping and table information of each original sub-transaction in all the active transactions according to the latest check point file.

For example, a synchronization service is started, active transactions saved in the last checkpoint file are loaded, and the attribution of the packets of sub-transactions to be cut in the active transactions is relocated. The sub-transaction g1_trx1 in active transaction A1 is now homed to the G2 packet.

Comparing whether the information of the sub-transaction original packet and the target packet in the active transaction is consistent, wherein the target packet of the sub-transaction G1_TRX1 is a G2 packet, the target packet stored in the check point is a G1 packet, the information is inconsistent, and the table T1 to be modified needs to be switched.

And taking the original sub-transaction without the end mark in all the current active transactions as a sub-transaction to be cut, and obtaining the table information related to the sub-transaction to be cut. Wherein an end tag is used to mark sub-transactions that have been cut. Since transaction cutting needs to be performed on the active transaction currently related to the table to be modified, and modification operations of virtual transaction save group configuration are constructed, and execution sequences of groups before and after the table to be modified are constrained, sub-transactions to be cut related to the table to be modified need to be found first (i.e. active transactions related to the table to be modified are found).

After the synchronization service is restarted in static modification, the active transaction needs to be recovered from the check point, and then the grouping information of the table is positioned in the modified grouping configuration according to the table information related to the sub-transaction in the active transaction.

After the operation of the table to be modified is cut from one packet to another, the transaction execution of the two packets has a transaction dependency relationship, because the operation of the same transaction must be guaranteed to be executed strictly according to the operation number sequence of the operation in the transaction, and therefore the operation of cutting to the target packet must wait for the transaction to be executed and put into storage in the original packet after the execution and put into storage are completed. To solve this problem, a virtual transaction for performing inspection with respect to an original packet is added to a target packet corresponding to an active transaction at the time of cutting the transaction, and the target packet inspects the completion of the execution of the original packet of the related table by executing the virtual transaction, and the target packet transaction can be started to be executed only after the condition is satisfied.

To better illustrate the method of static modification data synchronization packet of the present invention, step 30 of the method of static modification data synchronization packet of the embodiment of the present invention is further refined, specifically, as shown in fig. 2, the step 30 includes:

Step 301: and taking the active transaction corresponding to the sub-transaction to be cut related to the table to be modified as the active transaction to be cut, and adding an end mark for the sub-transaction to be cut.

For example, the active transaction A1 corresponding to the sub-transaction g1_trx1 to be cut is taken as an active transaction to be cut, and an end mark is added to the sub-transaction to be cut of the original packet related to the T1 table on the active transaction A1 to be cut, and the specific steps after the adding are as follows:

A1

{

g1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

}

Wherein an end mark of 0 indicates that no end mark is added, and an end mark of 1 indicates that an end mark is added.

Step 302: and constructing a sub-transaction of the active transaction to be cut, which belongs to the target group, as a virtual transaction, and adding an end mark for the virtual transaction.

Step 303: and constructing a modification operation for dividing the table to be modified into the target group according to the original group and the target group, and adding the modification operation to the virtual transaction. The virtual transaction operation is to wait for the execution of the original grouping transaction until the current transaction submits LSN and the commit number, wherein the commit number of the virtual transaction is the operation number of the last operation of the corresponding active transaction, and the grouping configuration modification operation of the table to be modified is added to the virtual transaction. The specific implementation form of the operation number is specified by a person skilled in the art according to the specific data synchronization scenario, and is not limited herein.

Steps 302 and 303, for example, construct a virtual transaction g2_trx2 belonging to the G2 packet and dependent on the G1 packet for the new packet referred to by the T1 table on active transaction A1. If the virtual transaction g2_trx2 can successfully execute the binning based on the target-side database updated by all the sub-transactions in the group G1 after the execution binning is completed, the virtual transaction g2_trx2 is referred to as a virtual transaction g2_trx2 depending on the group G1. The commit number of the transaction is the number of the last operation of the active transaction to be cut A1, and since the active transaction to be cut has not received a commit message, both commit LSN and wait LSN are set to 0, and the modified operation of T1 to change from G1 to G2 packets is saved to the virtual transaction. Taking this state of the active transaction A1 to be cut as an example, where the active transaction A1 to be cut is shown below, all sub-transactions of the active transaction A1 to be cut include the sub-transaction g1_trx1 to be cut and the virtual transaction g2_trx2, and when a commit message of the active transaction A1 to be cut is subsequently received, the sub-transaction g1_trx1 to be cut and the virtual transaction g2_trx2 to be cut are put in storage.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G2 grouping:

G2_TRX2[ rely on G1, wait for LSN:0, end tag: 1, commit LSN:0, commit number: 1]

}

Step 304: and continuing to receive the log, receiving a commit message, and warehousing all sub-transactions of the active transaction to be cut according to the commit message. At this time, the start of the synchronization service is completed, the log is continuously received, and the newly received log is divided into packets in a new packet configuration.

To better illustrate the method of static modification of data synchronization packets according to the present invention, step 304 of the method of static modification of data synchronization packets according to an embodiment of the present invention is further detailed, specifically, as shown in fig. 3, the step 304 includes:

step 3041a: receiving a commit message, taking a sub-transaction of an active transaction corresponding to the commit message as a sub-transaction to be put in storage, and judging whether the active transaction is an active transaction to be cut. Wherein the active transaction to which the end mark is not added by the corresponding sub-transaction is the active transaction to be cut. When the transaction is not an active transaction to be cut, all the sub-transactions to be put in the database are added into the submitted linked list of the corresponding target group according to the sequence from small to large of the combination of the submitted numbers of all the sub-transactions to be put in the database and the submitted LSNs of all the sub-transactions to be put in the database, so as to put in the database.

Step 3042a: and when the transaction is an active transaction to be cut, traversing all sub transactions to be put in the database of the active transaction to be cut, and finding corresponding virtual transactions from the sub transactions to be put in the database.

Step 3043a: taking the commit LSN of the commit message as a commit LSN and a wait LSN of the virtual transaction; the commit number of the virtual transaction is the operation number of the last operation in the active transaction to be cut. The commit number is set when the virtual transaction is constructed; the wait LSN is the commit LSN of the packet that the virtual transaction needs to wait for. By setting the wait LSN, the virtual transaction (the operation of modifying the configuration of the packet) can begin executing after the sub-transaction synchronization of less than or equal to the commit LSN in the wait packet is completed.

Step 3044a: and adding all the sub-transactions to be put into the submitted linked list of the corresponding target group according to the sequence from small to large of the combination of the submitted numbers of all the sub-transactions to be put and the submitted LSNs of all the sub-transactions to be put. The operation of the same source-side main transaction is the same in commit LSN, and different in commit number, and a person of ordinary skill in the art can self-assign a combination mode of the commit LSN and the commit number according to a specific data synchronization scene. Wherein each packet corresponds to a committed linked list.

Step 3045a: and warehousing all the sub-transactions to be warehoused according to the submitted linked list and the waiting LSN.

In order to maintain the consistency of the database, before synchronization, the operation to be synchronized is distinguished according to the dependent table, so as to avoid database errors caused by the fact that sub-transactions are not put in storage in the groups relied by the groups of the sub-transactions when the sub-transactions are executed in parallel. To better illustrate the method of static modification of data synchronization packets according to the present invention, step 3045a of the method of static modification of data synchronization packets according to an embodiment of the present invention is further detailed, specifically, as shown in fig. 4, the step 3045a includes:

step 30451: and sequentially judging whether the commit LSN of the sub-transaction to be put in the committed chain table is smaller than or equal to the commit LSN of the corresponding target packet, and judging whether the commit number of the sub-transaction to be put in the warehouse is smaller than or equal to the commit number of the target packet.

For example, active transaction A1, which has received the commit message, is as follows:

A1

{

g1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:25, commit number: 1]

G1_TRX4[ rely on G2, wait for LSN:25, end tag: 1, submit LSN:25, submit number: 2]

G1_TRX5[ Table: T1, end tag: 0, commit LSN:25, commit number: 4]

G2 grouping:

G2_TRX2[ rely on G1, wait for LSN:25, end tag: 1, submit LSN:25, submit number: 1]

G2_TRX3[ Table: T1, end marker: 1, commit LSN:25, commit number: 2]

}

The execution thread polls the G1 group and the G2 group, and executes the group according to the size sequence of the combination of the commit LSN and the commit number of the transaction from the committed linked list of the G1 group and the G2 group.

Step 30452: if yes, discarding the sub-transaction to be put in storage; if not, judging whether the sub-transaction to be put in storage is a virtual transaction.

For example, the execution thread extracts the sub-transaction g1_trx1 to be put in the G1 packet, and at this time, the G1 packet does not execute the putting in the box of a certain sub-transaction to be put in the box, so that both the commit LSN and the commit number of the G1 packet are 0, the commit LSN25 of the sub-transaction g1_trx1 to be put in the box is not less than or equal to 0, and the commit number 1 is not less than or equal to 0, and the sub-transaction to be put in the box is put in the box.

Step 3053: and when the sub-transaction is a virtual transaction, according to the commit LSN and the commit number corresponding to the original packet on which the virtual transaction depends, selectively warehousing or not warehousing all sub-transactions to be warehoused in the target packet to which the virtual transaction belongs. Wherein, when the transaction is a virtual transaction, the execution of the sub-transaction in the original group corresponding to the modification is dependent; when not a virtual transaction, it may be executed directly, independent of the execution of other packets.

And sequentially judging whether the commit LSN corresponding to the original packet relied by the virtual transaction is larger than or equal to the wait LSN of the virtual transaction, and judging whether the commit number corresponding to the original packet is larger than or equal to the commit number of the virtual transaction. If not, not warehousing all sub-transactions to be warehoused in the target group to which the virtual transaction belongs in the submitted linked list; if yes, all sub-transactions to be put in the target group to which the virtual transaction belongs are put in storage according to the submitted linked list.

For example, the execution thread extracts the sub-transaction to be binned g1_trx4 of the G1 packet, which depends on the execution of the G2 packet, and is a virtual transaction, the commit LSN25 of the G2 packet is equal to the wait LSN25 of the virtual transaction g1_trx4, the condition is satisfied, the commit number 0 of the G2 packet (at this time, the G2 packet does not binning a certain sub-transaction to be binned) is not greater than or equal to the commit number 2 of the virtual transaction g1_trx4, and the condition is not satisfied, so that the g1_trx4 transaction cannot be executed, the sub-transaction to be binned in the G1 packet on the currently committed linked list should be skipped, and the sub-transaction to be binned of the G2 packet is fetched.

The execution thread fetches the to-be-binned sub-transaction g2_trx2 of the G2 packet, which depends on the execution of the G1 packet, when the G1 packet has executed g1_trx1, commit LSN of the G1 packet and commit LSN25 and commit number 1 of g1_trx1. And the waiting LSN25 of the sub-transaction to be put into storage G2 TRX2 is equal to the submitting LSN25 of the G1 group, the submitting number 1 of the G2 group is equal to the submitting number 1 of the G1 group, and the condition is met, and the sub-transaction to be put into storage G2 TRX2 is put into storage.

Step 30454: after executing the sub-transaction to be put into the warehouse each time, taking the commit LSN of the sub-transaction to be put into the warehouse as the commit LSN of the corresponding target group, and taking the commit number of the sub-transaction to be put into the warehouse as the commit number of the target group. Each group also needs to record the commit LSN and commit number of the executed transaction at the same time when executing the transaction, and is used for filtering the executed transaction after the fault, so as to ensure the consistency of the data.

For example, the execution thread fetches the g1_trx1 transaction of the G1 packet into a pool, recording the commit LSN 25 and commit number 1 of the completed transaction as the commit LSN of the target packet.

By setting the virtual transaction, when the sub-transaction to be put in the warehouse related to the table to be modified is submitted, the transaction dependency relationship executed between the original group and the target group and the execution sequence of the sub-transaction formed after cutting are ensured. By setting the waiting LSN and the submitting LSN of the virtual transaction and setting the submitting number of the virtual transaction as the operation number corresponding to the last operation of the active transaction, the operation error caused by the transaction dependency relationship among sub-transactions is avoided from occurring in the synchronous service during the process of executing the warehouse entry, and the consistency of synchronous data is recovered when the synchronous service fails for recovery.

The target end continues to receive the log sent by the source end, generates or classifies the operations in the log into corresponding active transactions, and carries out corresponding processing according to the operation types. To better illustrate the method of static modification data synchronization packet according to the present invention, step 30 of the method of static modification data synchronization packet according to an embodiment of the present invention is further detailed, specifically, as shown in fig. 5, the step 304 further includes:

step 3041b: restarting the synchronous service, continuously receiving the log, and judging the type of the operation corresponding to the log. Where there is often a dependency conflict between DML (Data Manipulation Language ) operations that belong to the same master transaction at the source and are split into different packets for parallel execution. DDL (Data Definition Language ) operations are generally free of dependency conflicts without using the method of statically modifying data sync packets of embodiments of the present invention. After receiving DDL operation, dividing the DDL operation into corresponding active transactions in a dependent table, adding the corresponding commit message to a committed linked list according to commit LSN after receiving the corresponding commit message, and executing warehousing. And when the operation is a rollback operation, releasing all sub-transactions and the active transactions under the active transaction according to the active transaction. The rollback operation of the source end withdraws all executed main transactions, and in order to maintain the consistency of the source end data and the target end data in the data synchronization, all operations of the target end related to the main transactions of the source end need to be rolled back.

Step 3042b: when the operation is a DML operation, dividing the DML operation into corresponding sub-transactions according to a main transaction ID, table information and grouping configuration of the DML operation; when the sub-transaction has added an end mark, the sub-transaction is a sub-transaction of an active transaction to be cut, a corresponding target sub-transaction is created according to the grouping configuration, and the operation number of the DML operation is used as the commit number of the target sub-transaction.

For example, upon receiving a DML operation commit LSN 22, active transaction A1 is located with its corresponding ID 1 by the master transaction ID. The corresponding sub-transaction original sub-transaction g1_trx1 and target sub-transaction g2_trx3 corresponding to the table to be modified T1 are found in the active transaction A1. Since the original sub-transaction g1_trx1 has added an end tag, i.e. the sub-transaction to be cut at the last time of the packet configuration modification, the table T1 to be modified at the moment belongs to the G2 packet according to the packet configuration, a target sub-transaction g2_trx3 belonging to the G2 packet is created on the active transaction, and the DML operation is added to the target sub-transaction, and the operation number of the operation is taken as the commit number of the target sub-transaction, specifically as follows:

A1

{

g1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G2 grouping:

G2_TRX3[ Table: T1, end tag: 0, commit LSN:0, commit number: 2]

}

Upon receipt of a DML operation with commit LSN 24, the DML operation is again appended to the sub-transaction G1_TRX5 by locating the main transaction ID to the active transaction A1 with ID 1, at which point T1 belongs to the G1 packet, and there is already a sub-transaction G1_TRX5 on the active transaction that belongs to the G1 packet, whose end tag is 0, i.e., no end tag is appended.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G1_TRX4[ rely on G2, wait for LSN:0, end tag: 1, commit LSN:0, commit number: 2]

G1_TRX5[ Table: T1, end tag: 0, commit LSN:0, commit number: 4]

G2 grouping:

G2_TRX3[ Table: T1, end marker: 1, commit LSN:0, commit number: 2]

}

Step 3043b: when the operation is a commit operation, finding a corresponding active transaction to be cut according to a log corresponding to the commit operation, adding a commit message for all sub-transactions to be put in storage of the active transaction to be cut, taking a commit LSN of the commit operation as a commit LSN of all sub-transactions to be put in storage, adding all sub-transactions to be put in storage to a committed linked list of a target group, and putting all sub-transactions to be put in storage according to the committed linked list.

When the method for statically modifying the data synchronization packet in the embodiment of the invention continues to receive the log and divides the corresponding operation, the packet is determined by the table information analyzed in the log corresponding to the operation. The packet belongs to an active transaction or an active transaction to be cut. If the table corresponding to the table information is not the table to be modified, the operation that the corresponding active transaction is not required to be cut before and after the grouping configuration is modified and the operation that is continuously received is divided into sub-transactions of the active transaction is described.

For example, upon receiving a commit operation with commit LSN of 25, locating active transaction A1 with ID 1 by the master transaction ID, traversing all sub-transactions of active transaction A1, and marking their commit LSN as that of the current commit operation, and when the sub-transactions are virtual transactions, then setting the wait LSN of the packet it waits for to be 25, and then adding to the committed linked list of the corresponding packet.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:25, commit number: 1]

G1_TRX5[ Table: T1, end tag: 0, commit LSN:25, commit number: 4]

G2 grouping: G2_TRX2[ rely on G1, wait for LSN:25, end tag: 1, submit LSN:25, submit number: 1]

G2_TRX3[ Table: T1, end marker: 1, commit LSN:25, commit number: 2]

}

Wherein the commit message is not presented herein, and the specific implementation is specified by one of ordinary skill in the art according to a specific data synchronization scenario, and is not limited herein.

Notably, upon restart recovery of the synchronization service failure, wherein:

the synchronized commit LSNs, commit numbers, and virtual transactions in each target packet are restored based on the latest checkpoint file. And when the sub-transaction is executed and put in storage after recovery, judging whether the commit LSN and the commit number corresponding to the executed sub-transaction are smaller than or equal to the commit LSN and the corresponding commit number of the last synchronized sub-transaction. If yes, not executing the sub-transaction and warehousing; and if not, executing the sub-transaction, and modifying and storing the current grouping configuration according to the modification operation in the virtual transaction. Since the checkpointing action has been performed before exiting the synchronization service upon static modification, there is no need to perform the checkpointing action again. And filtering sub-transactions with commit LSN smaller than or equal to that of the target packet and commit number smaller than or equal to that of the target packet according to the checkpoint file stored before exiting the synchronization service, so as to ensure the consistency of fault recovery data.

Example 2:

on the basis of the above embodiment 1, the embodiment of the present invention provides a specific example of a method for statically modifying a data synchronization packet so as to better understand the whole synchronization process, and will be described by taking a log information sequence shown in the following table as an example:

the source database has a table T1 (ID INT). The source end has a transaction to operate on the table T1 to generate the following sequential log, and the receiving thread forms the following numbered table after receiving:

the target synchronization service configures two groups, G1 and G2, initially, a table to be modified is configured in the G1 group, T1 to G2 groups are modified, then the G2 groups are changed back to the G1 groups, and the process is as follows:

when an operation with LSN 21 is received, an active transaction A1 with ID 1 is created through the main transaction ID, at the moment T1 belongs to the G1 group, a sub-transaction G1_TRX1 belonging to the G1 group is created on the active transaction, the operation is added to the sub-transaction, and the operation number of the operation is taken as the commit number of the sub-transaction.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end tag: 0, commit LSN:0, commit number: 1]

}

A modification operation is prepared to group T1 from G1 to G2.

The method comprises the steps of stopping receiving a source log, traversing active transactions, storing grouping information of sub-transactions and table information related to the sub-transactions into a check point file, wherein the active transaction stored in the check point file is A1, and the active transaction has a sub-transaction G1_TRX1, the grouping is G1 grouping, and the table information related to the sub-transaction is T1.

And exiting the synchronization service.

The sync packet configuration of T1 is modified from G1 packets to G2 packets.

And starting a synchronization service, loading the active transaction stored in the last check point, and relocating the sub-transaction grouping attribution without adding an end mark in the active transaction. The g1_trx1 sub-transaction in active transaction A1 is now attributed to the G2 packet.

Comparing whether the grouping information of the sub-transaction original grouping in the active transaction is consistent with that of the target grouping, wherein the new grouping of the G1-TRX 1 sub-transaction is G2, and the new grouping is G1 stored in the check point, is inconsistent, and the to-be-modified table T1 needs to switch the grouping.

An end tag is added to the sub-transaction to be cut that is referred to by the T1 table on active transaction A1.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

}

A G2 dependent G1 virtual transaction G2_TRX2 is constructed for the new packet referred to by the T1 table on active transaction A1, and T1 is saved to the virtual transaction with a modification operation from the G1 packet to the G2 packet.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G2 grouping:

}

And finishing the starting of the synchronous service.

Upon receiving the operation with LSN 22, the active transaction A1 with ID 1 is located by the main transaction ID, at which point T1 belongs to the G2 packet, a sub-transaction g2_trx3 belonging to the packet G2 is created on the active transaction, and the operation is added to the sub-transaction, and the operation number of the operation is taken as the commit number of the sub-transaction.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G2 grouping:

G2_TRX3[ Table: T1, end tag: 0, commit LSN:0, commit number: 2]

}

A modification operation is prepared to group T1 from G2 to G1.

Stopping receiving the source log, traversing the active transactions, storing grouping information of sub-transactions and table information related to the sub-transactions into a check point file, wherein the active transaction stored in the check point file is A1, the active transaction has a sub-transaction G1_TRX3 with no end mark added, the grouping is G2 grouping, and the table information related to the sub-transaction is T1.

And exiting the synchronization service.

The sync packet configuration of T1 is modified from G2 packets to G1 packets.

And starting a synchronization service, loading the active transaction stored in the last check point, and relocating the sub-transaction grouping attribution without adding an end mark in the active transaction. The g1_trx3 sub-transaction in active transaction A1 is now attributed to the G1 packet.

Comparing whether the sub-transaction grouping information in the active transaction is consistent, the grouping of the G1_TRX3 sub-transaction is G1, and the group stored in the check point file is G2, which is inconsistent, and the table T1 to be modified needs to be switched.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G2 grouping:

G2_TRX3[ Table: T1, end marker: 1, commit LSN:0, commit number: 2]

}

A virtual transaction G1_TRX4 belonging to the G1 group and depending on the G2 group is constructed for the new group related to the T1 table on the active transaction A1, the commit number of the virtual transaction is the number of the last operation of the active transaction A1, and the modification operation of changing the T1 group from G2 to G1 is saved to the virtual transaction.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G2 grouping:

G2_TRX3[ Table: T1, end marker: 1, commit LSN:0, commit number: 2]

}

And finishing the starting of the synchronous service.

Upon receiving the operation with LSN 23, the active transaction A1 with ID 1 is located by the main transaction ID, at which point T1 belongs to the G1 group, a sub-transaction g1_trx5 belonging to the group G1 is created on the active transaction, and the operation is added to the sub-transaction, and the number of the operation is taken as the commit number of the sub-transaction.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G1_TRX5[ Table: T1, end tag: 0, commit LSN:0, commit number: 3]

G2 grouping:

G2_TRX3[ Table: T1, end marker: 1, commit LSN:0, commit number: 2]

}

Upon receipt of the LSN 24 operation, the active transaction A1 with ID 1 is located by the main transaction ID, at which point T1 belongs to the G1 group, there is already a sub-transaction G1_TRX5 on the active transaction that belongs to the group G1, its end is marked 0, so the operation is also added to the sub-transaction G1_TRX5, and the number of the operation is taken as the commit number of the sub-transaction.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:0, commit number: 1]

G1_TRX5[ Table: T1, end tag: 0, commit LSN:0, commit number: 4]

G2 grouping:

G2_TRX3[ Table: T1, end marker: 1, commit LSN:0, commit number: 2]

}

Upon receiving a commit operation with LSN 25, the active transaction A1 with ID 1 is located by the master transaction ID, all sub-transactions on the active transaction A1 are traversed, and their commit LSNs are marked as LSNs for the current commit operation, and if virtual, their commit LSNs waiting for a packet are set to 25 and then added to the committed transaction chain table for the corresponding packet.

A1

{

G1 grouping:

G1_TRX1[ Table: T1, end marker: 1, commit LSN:25, commit number: 1]

G1_TRX5[ Table: T1, end tag: 0, commit LSN:25, commit number: 4]

G2_TRX3[ Table: T1, end marker: 1, commit LSN:25, commit number: 2]

}

The execution thread polls the G1 packet and the G2 packet, and executes the packets according to the size sequence of the sub-transaction commit LSN and the commit number combination from the group committed linked list.

The execution thread fetches the g1_trx1 transaction of the G1 packet, keeping track of commit LSN 25 and commit number 1 of the completed transaction it has executed. The execution thread fetches the g1_trx4 transaction of the G1 packet, skips the G1 packet (including the g1_trx4 transaction), and fetches the transaction of the G2 packet. The execution thread fetches the g2_trx2 transaction of the G2 packet and performs binning. The execution thread fetches the g2_trx3 transaction of the G2 packet, performs binning, and records commit LSN 25 and commit number 2 of the completed transaction it has executed. The pending sub-transaction of the G2 packet has been executed, and the execution thread executes the g1_trx4 transaction of the G1 packet. When the program is aborted after the execution is finished, the commit LSN of the G1 packet is 25 and the commit number is 2; the commit LSN for the G2 packet is 25 and commit number is 2. And restarting the synchronous service of the target end, and executing the sub-transaction G1_TRX5 of the G1 packet after fault recovery.

Example 3:

fig. 6 is a schematic diagram of an architecture of an apparatus for statically modifying data synchronization packets according to an embodiment of the present invention. The means for statically modifying the data synchronization packet of the present embodiment comprises one or more processors 31 and a memory 32. In fig. 6, a processor 31 is taken as an example.

The processor 31 and the memory 32 may be connected by a bus or otherwise, which is illustrated in fig. 6 as a bus connection.

The memory 32 is used as a non-volatile computer readable storage medium for storing non-volatile software programs and non-volatile computer executable programs, such as the method of statically modifying data sync packets in embodiment 1. The processor 31 performs the method of statically modifying the data sync packet by running non-volatile software programs and instructions stored in the memory 32.

The memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 32 may optionally include memory located remotely from processor 31, which may be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The program instructions/modules are stored in the memory 32 and when executed by the one or more processors 31 perform the method of statically modifying data synchronization packets in embodiment 1 described above, for example, performing the various steps shown in fig. 1-5 described above.

It should be noted that, because the content of information interaction and execution process between modules and units in the above-mentioned device and system is based on the same concept as the processing method embodiment of the present invention, specific content may be referred to the description in the method embodiment of the present invention, and will not be repeated here.

Those of ordinary skill in the art will appreciate that all or a portion of the steps in the various methods of the embodiments may be implemented by a program that instructs associated hardware, the program may be stored on a computer readable storage medium, the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims

1. A method of statically modifying a data synchronization packet, comprising:

2. The method for statically modifying a data synchronization packet according to claim 1, wherein before the obtaining the target packet corresponding to the sub-transaction to be cut according to the table information and the new packet configuration related to all the sub-transactions to be cut currently includes:

3. The method of statically modifying a data synchronization packet according to claim 2, wherein before the stopping the synchronization service, saving the original packet for each original sub-transaction in all current active transactions and the table information related to the original sub-transaction to a checkpoint file, and after saving, exiting the synchronization service comprises:

4. The method for statically modifying a data synchronization packet according to claim 1, wherein the constructing a virtual transaction of the table to be modified according to the original packet of the table to be modified and the target packet of the table to be modified, taking the virtual transaction as a sub-transaction of the corresponding transaction to be cut, and warehousing all sub-transactions of the transaction to be cut comprises:

5. The method of claim 4, wherein the continuing to receive logs, receiving commit messages, binning all sub-transactions of the active transactions to be cut according to the commit messages comprises:

6. The method of claim 5, wherein binning all sub-transactions to be binned according to the committed linked list and wait LSN comprises:

7. The method for statically modifying a data synchronization packet according to claim 6, wherein selectively binning or not binning all sub-transactions to be binned in a target packet to which a virtual transaction belongs according to a commit LSN and a commit number corresponding to an original packet on which the virtual transaction depends when the virtual transaction is a virtual transaction comprises:

8. The method of claim 4, wherein the continuing to receive logs, receiving commit messages, binning all sub-transactions of the active transactions to be cut according to the commit messages comprises:

9. A method of statically modifying a data synchronization packet according to any one of claims 1-8, wherein upon restart recovery from a synchronization service failure, wherein:

10. An apparatus for statically modifying data sync packets, comprising at least one processor and a memory, the at least one processor and the memory being coupled via a data bus, the memory storing instructions executable by the at least one processor, the instructions, when executed by the processor, for performing the method for statically modifying data sync packets as claimed in any one of claims 1 to 9.