CN110427422B - Data consistency checking method and equipment during data synchronization abnormity and storage medium - Google Patents
Data consistency checking method and equipment during data synchronization abnormity and storage medium Download PDFInfo
- Publication number
- CN110427422B CN110427422B CN201910436428.3A CN201910436428A CN110427422B CN 110427422 B CN110427422 B CN 110427422B CN 201910436428 A CN201910436428 A CN 201910436428A CN 110427422 B CN110427422 B CN 110427422B
- Authority
- CN
- China
- Prior art keywords
- transaction
- synchronization
- auxiliary table
- synchronous
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 230000001360 synchronised effect Effects 0.000 claims abstract description 114
- 230000002159 abnormal effect Effects 0.000 claims abstract description 53
- 230000008859 change Effects 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 21
- 238000005096 rolling process Methods 0.000 claims description 25
- 238000004891 communication Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 abstract description 19
- 238000005516 engineering process Methods 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 5
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000005856 abnormality Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 239000002253 acid Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a method, equipment and a storage medium for verifying data consistency when data synchronization is abnormal, wherein the method comprises the following steps: firstly, inserting a transaction ID of a synchronous transaction into an auxiliary table in a target end database; then when receiving the error notification, judging the operation of the current synchronous transaction; when the transaction change operation is carried out, whether the error notification belongs to a preset error type is judged, if the error notification belongs to the preset error type, the carried out synchronous operation is rolled back, the transaction ID of the synchronous transaction is inserted into the auxiliary table again, the transaction synchronous operation is carried out again, when the transaction change operation is carried out for the transaction, whether the auxiliary table contains the transaction ID of the synchronous transaction is inquired, and when the transaction change operation is not contained, the transaction ID of the transaction is inserted into the auxiliary table and the synchronous operation is carried out again. In the synchronization process, the transaction label is registered in the auxiliary table when the synchronization transaction starts, and if an error occurs in the synchronization process, different judgment processing is carried out according to the current operation so as to ensure the consistency of the synchronization data.
Description
Technical Field
The invention relates to the technical field of database synchronization, in particular to a method, equipment and a storage medium for verifying data consistency when data synchronization is abnormal.
Background
Among database data real-time synchronization technologies, the data real-time synchronization technology based on database log analysis is widely researched and applied due to the characteristics of supporting a heterogeneous operating system and a database platform, having small invasion to a source-end database system, having low data synchronization delay and the like. The log formats and the log contents of heterogeneous database systems are different greatly, so that the log contents of the data synchronization source database cannot be directly applied to the target database. A real-time data synchronization technology based on log analysis generally adopts the principle that incremental log content of a source end database is captured and then is sent to a target end through a network, a SQL (structured query language) reverse generation technology is used for recovering transaction operation SQL statements of the source end database at the target end, and then the recovered SQL statements are executed through a universal database access interface such as ODBC (open data base code group bus), OCI (optical code distribution interface) and the like, so that real-time data synchronization of the source end and the target end is realized.
Furthermore, the database data real-time synchronization device constructed by using the data real-time synchronization technology mainly comprises a synchronization software log reading module which reads a source database log file in real time at a synchronization system source end and captures transaction increment log content on a source database; the synchronous software log analysis module analyzes the incremental log content in real time, extracts a transaction ID, an operation table ID, the operation types of the transaction such as INSERT, UPDATE and DELETE, a transaction operation timestamp, a log LSN value, operation data and the like in the log, assembles the transaction ID, the operation table ID, the operation types of the transaction such as INSERT, UPDATE and DELETE, and assembles the transaction operation timestamp, the log LSN value and the operation data into a message with a specific format; and the synchronous software log sending module is used for sending the messages in the sending queue to a target end of the synchronous system through a TCP/IP network.
At the target end of the synchronous system, a synchronous software log receiving module receives the message sent by the source end, analyzes the content of the message, caches the operation belonging to the same transaction according to the transaction ID, and transfers the operation to a log executing module for synchronous execution when the COMMIT message of the transaction is received; and the synchronous software log execution module is used for constructing SQL statements according to the transaction operation content and synchronously executing the SQL statements in the target database.
In the database data real-time synchronization technology, when the target end data synchronization service performs data warehousing on the target end database, the situation that a network between the target end synchronization service and the target end database is abnormal or the target end database is broken down may cause repeated execution of synchronization transactions, which affects the consistency of synchronization data.
The prior art is therefore still subject to improvements and enhancements.
Disclosure of Invention
In view of the foregoing disadvantages of the prior art, an object of the present invention is to provide a method, a device, and a storage medium for verifying data consistency when data synchronization is abnormal, so as to solve the problem of inconsistent synchronization data caused by abnormal data in the current data synchronization process.
In order to achieve the purpose, the invention adopts the following technical scheme:
a data consistency checking method when data synchronization is abnormal comprises the following steps:
receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation;
when an error report notification of a target end database is received, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence;
when the current operation is the transaction change operation, judging whether the error notification belongs to a preset error type, rolling back the performed synchronization operation when the error notification belongs to the preset error type, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again;
when the current operation is a transaction submitting operation, the target end database is reconnected, whether the auxiliary table contains the transaction ID of the synchronous transaction is inquired, and when the auxiliary table does not contain the transaction ID of the synchronous transaction, the transaction ID of the synchronous transaction is inserted into the auxiliary table and the transaction synchronization operation is carried out again.
Preferably, in the method for checking data consistency when data synchronization is abnormal, the step of performing transaction synchronization operation after receiving the log message packet sent by the source end and inserting the transaction ID of the synchronization transaction in the log message packet into the auxiliary table created in the database of the target end includes:
analyzing a log message packet where the current synchronous transaction is located and acquiring a transaction ID of the synchronous transaction in the log message packet;
and performing an insertion operation in an auxiliary table created in the target end database to insert the transaction ID into the auxiliary table.
Preferably, in the method for checking data consistency when data synchronization is abnormal, the step of judging whether the error notification belongs to a preset error type when the current operation is a transaction change operation, and rolling back the performed synchronization operation and inserting the transaction ID of the synchronized transaction into the auxiliary table again and performing the transaction synchronization operation again when the error notification belongs to the preset error type includes:
when the current operation is a transaction change operation, identifying the type of an error according to an error code returned by a target end database;
judging whether the error type belongs to a preset error type or not;
and when the error type is the preset error type, disconnecting the target end database, rolling back the performed synchronization operation, and after the rolling back is completed, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again.
Preferably, in the method for checking data consistency when data synchronization is abnormal, the step of disconnecting the target-side database and rolling back the performed synchronization operation when the error type is a preset error type, and inserting the transaction ID of the synchronized transaction into the auxiliary table again after rolling back is completed and performing the transaction synchronization operation again further includes:
and when the error type is not the preset error type, executing a preset error processing flow of the synchronous transaction.
Preferably, in the data consistency checking method when data synchronization is abnormal, the preset error types include a network abnormal error and a database system level error.
Preferably, in the data consistency check method when data synchronization is abnormal, the step of reconnecting the target-side database when the current operation is a transaction commit operation, querying whether the auxiliary table contains the transaction ID of the synchronized transaction, and inserting the transaction ID of the synchronized transaction into the auxiliary table and performing the transaction synchronization operation again when the auxiliary table does not contain the transaction ID of the synchronized transaction includes:
when the current operation is a transaction submitting operation, disconnecting the target end database and reconnecting the target end database;
calling an auxiliary table in a target end database, and inquiring whether a transaction ID registered when the synchronous transaction starts exists in the auxiliary table;
and when the transaction ID does not exist in the auxiliary table, inserting the synchronous transaction ID into the auxiliary table in the target end database and restarting the transaction synchronization operation.
Preferably, in the data consistency checking method when data synchronization is abnormal, the step of inserting the synchronized transaction ID into the auxiliary table in the target-end database and restarting the transaction synchronization operation when the transaction ID does not exist in the auxiliary table further includes:
when the transaction ID exists in the secondary table, the next synchronized transaction is executed.
Preferably, in the data consistency checking method when data synchronization is abnormal, the transaction change operation includes an insert operation, an update operation, and a delete operation.
A data consistency check device when data synchronization is abnormal comprises: a processor, a memory, and a communication bus;
the memory has stored thereon a computer readable program executable by the processor;
the communication bus realizes connection communication between the processor and the memory;
the processor, when executing the computer readable program, implements the steps in the data consistency check method when the data synchronization is abnormal, as described above.
A computer readable storage medium storing one or more programs, which are executable by one or more processors, to implement the steps in the data consistency checking method when data synchronization is abnormal as described above.
The invention provides a method, equipment and a storage medium for verifying data consistency when data synchronization is abnormal, wherein the method comprises the following steps: firstly, receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation; then when receiving an error report notification of a target end database, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence; when the current operation is a transaction change operation, judging whether an error notification belongs to a preset error type, rolling back the performed synchronization operation and inserting the transaction ID of the synchronization transaction into the auxiliary table again when the error notification belongs to the preset error type, and performing the transaction synchronization operation again, when the current operation is a transaction commit operation, reconnecting the target end database, inquiring whether the auxiliary table contains the transaction ID of the synchronization transaction, and when the auxiliary table does not contain the transaction ID of the synchronization transaction, inserting the transaction ID of the synchronization transaction into the auxiliary table and performing the transaction synchronization operation again. In the synchronization process, the transaction label is registered in the auxiliary table when the synchronization transaction starts, and if an error occurs in the synchronization process, whether the transaction label is registered is judged according to the current operation, so that whether the synchronization is finished is further judged, and the consistency of the synchronization data is ensured.
Drawings
Fig. 1 is a flowchart of a data consistency checking method when data synchronization is abnormal according to the present invention.
Fig. 2 is a flowchart of the step S100 in the data consistency checking method for data synchronization abnormality according to the present invention.
Fig. 3 is a flowchart of the step S300 in the data consistency checking method for data synchronization abnormality according to the present invention.
Fig. 4 is a flowchart of the step S400 in the data consistency checking method for data synchronization abnormality according to the present invention.
FIG. 5 is a diagram illustrating an operating environment of a data consistency check program according to an embodiment of the present invention when data synchronization is abnormal.
FIG. 6 is a functional block diagram of a system for checking data consistency when a data synchronization exception is installed according to a preferred embodiment of the present invention.
Detailed Description
In view of the problem in the prior art that the inconsistent synchronous data is likely to occur when an exception occurs in the data synchronization process, the present invention aims to provide a method, a device and a storage medium for verifying the data consistency when the data synchronization is abnormal, which can ensure that the inconsistent synchronous data does not occur when the exception occurs in the data synchronization process.
In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1, the method for checking data consistency when data synchronization is abnormal according to the present invention includes the following steps:
s100, receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation.
Specifically, the source end sends a table which needs data synchronization in the form of a log message packet, so the source end needs to generate the log message packet before sending the log message packet, because incremental log information generated by the source end database cannot be directly applied to a target end database in a heterogeneous database system environment, the log information of the source end database needs to be analyzed, then a message packet with a specific format in the synchronization service is generated and stored in a sending queue to wait for sending by a log sending module, in the specific implementation, the log capturing module of the source end data synchronization service captures the incremental log data of the source end database system in real time based on a log capturing technology, then the incremental log data is transmitted to a log analysis module to be analyzed, and then the log analyzing module of the source end data synchronization service analyzes the incremental log into the specific format in the log packet, stored in the transmit queue. Preferably, the log message packet includes a transaction ID, a transaction operation type, an operation table name, column information, a data value, and the like. When the target terminal carries out data synchronization, the received synchronization transaction is classified according to the transaction ID. And then, a log sending module of the source end data synchronization service sends the log message packet in the internal sending queue to a target end data synchronization service system through a TCP/IP network based on a thread synchronization mechanism. Preferably, in order to ensure the reliability and data integrity of data transmission, the source end log sending module adopts a complete message response mechanism, the log sending module considers that data transmission is completed only after obtaining the target end confirmation message, otherwise, the data is automatically retransmitted, and after receiving the log message packets, the target end performs sequence numbering on each log message packet and simultaneously needs to return confirmation information.
Further, the target-side data synchronization service interacts with the target-side database through the driver of the database, so that the synchronization service and the database may not be on the same machine. After the grids are arranged separately, the possibility of grid failure between the grids needs to be considered; in addition, in the synchronization process, the target-side database may crash due to crash of a BUG or crash of the database due to other reasons, under these circumstances, when the synchronization service executes synchronization on the target-side database, these possible failures need to be considered, so as to prevent repeated execution or less execution of the synchronized transaction, therefore, the present invention firstly creates an auxiliary table in the target-side database, where the auxiliary table is used to store a transaction tag of the synchronized transaction, in this embodiment, the transaction tag is a transaction ID, when the synchronized transaction starts, the transaction ID of the current synchronized transaction is registered in the auxiliary table, and then when an error occurs in the synchronization process, different judgment processing is performed according to the current synchronization operation, so as to ensure consistency of the synchronized data; after the transaction ID is registered, the synchronization transaction is started, and a specific synchronization transaction is as follows:
BEGIN
INSERT INTO T VALUES(1)；
COMMIT；
END；
the structure of a specific auxiliary table is shown in the following table:
| name of field | Type of field | Description of field |
| TRXID | NUMBER | Registering transaction IDs executed during data synchronization |
Please refer to fig. 2, which is a flowchart of the step S100, including the following steps:
s101, analyzing a log message packet where the current synchronous transaction is located and acquiring a transaction ID of the synchronous transaction in the log message packet;
s102, performing an inserting operation in the auxiliary table created in the target end database to insert the transaction ID into the auxiliary table.
For the specific operation transaction, before the synchronous transaction operation is executed, an INSERT operation is executed on the secondary table, and the transaction ID is inserted into the secondary table, which specifically includes the following steps:
INSERT INTO DMHS_TRXID(TRXID)VALUES(12345)；
s200, when receiving an error report notification of a target end database, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence.
In this embodiment, when an error notification of the target-side database is not received, the synchronous transaction is executed according to the flow, then the next synchronous transaction is executed, when an error notification of the target-side database is received (i.e. an error occurs in the synchronization process), it is necessary to determine what operation the current synchronous transaction is, generally speaking, when the synchronous transaction is performed, a transaction change operation is performed first to perform data conversion, then a transaction commit operation is performed after the conversion is completed to transfer the converted data to the target-side database, so the invention needs to specifically provide a processing method after the error by determining the operation type, at different operation stages, the processing method has differences, at the transaction change operation stage, the data conversion is not completed yet, so the data conversion needs to be performed again, at the transaction commit operation stage, the data conversion is completed but the data storage may not be successful, therefore, the transaction ID needs to be verified to ensure the consistency of the synchronized data, and preferably, the transaction change operation includes an INSERT operation (INSERT operation), an UPDATE operation (UPDATE operation), and a DELETE operation (DELETE operation), and data can be inserted, updated, or deleted during synchronization.
In addition, because the types of errors are many, in order to distinguish the errors, the invention sets an error code (error code) to represent the types of the errors, if the errors occur in the synchronization process, the errors are judged and recognized through the original setting in the system, the errors are realized through the display mode of the error code, and the system quickly finds the types of the errors through the identification of the error code.
S300, when the current operation is the transaction change operation, judging whether the error notification belongs to a preset error type, rolling back the performed synchronization operation when the error notification belongs to the preset error type, inserting the transaction ID of the synchronized transaction into the auxiliary table again, and performing the transaction synchronization operation again.
In this embodiment, first, the type of the error is identified according to the error notification (in this embodiment, the error code), and if the error type belongs to the preset type, step S100 is executed again, and the specific flowchart of step S300 is shown in fig. 3.
Please refer to fig. 3, which is a flowchart of the step S300, including the following steps:
s301, when the current operation is a transaction change operation, identifying the type of an error according to an error code returned by a target end database;
s302, judging whether the error type belongs to a preset error type;
s303, when the error type is the preset error type, disconnecting the target end database, rolling back the performed synchronization operation, and after the rolling back is completed, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again.
In this embodiment, the preset error types include a network abnormal error and a database system level error, which are presented in an error code manner when an error is reported, and if such an error occurs, it indicates that data synchronization is not completed correctly when an insertion operation, an update operation, or a deletion operation is performed, at this time, in order to avoid importing abnormal data into a target database, a connection with the target database needs to be disconnected first, and then synchronization operation needs to be performed again.
Further, the step S303 further includes:
and when the error type is not the preset error type, executing a preset error processing flow of the synchronous transaction.
Specifically, because other errors can occur in the synchronization process, the invention sets the error processing flow of the synchronized transaction for the non-preset errors, the error processing flow can be set according to the actual requirements, the invention does not limit the specific error processing flow of the transaction, for example, when the non-preset error occurs, the synchronization of the current transaction is directly cancelled, the synchronization operation of the next transaction is carried out, and the like, thereby ensuring that the system can not generate a stagnation state to influence the synchronization process when the system synchronizes the transaction.
For the synchronous transaction in the above embodiment, when the synchronous transaction is executed on the target-side database to perform insert INTINTINTO T VALUES (1), if the synchronization fails, it is necessary to determine an error code returned by the target-side database to identify the type of the error, and if the error is a network anomaly or a database system-level error, it is necessary to roll the previous synchronous operation back and forth in a disconnection manner, and then re-synchronize the faulty transaction.
S400, when the current operation is a transaction submitting operation, the target end database is reconnected, whether the auxiliary table contains the transaction ID of the synchronous transaction is inquired, and when the auxiliary table does not contain the transaction ID of the synchronous transaction, the transaction ID of the synchronous transaction is inserted into the auxiliary table and the transaction synchronization operation is carried out again.
Specifically, when a commit operation is performed on a target database, a network failure occurs while waiting for the return of the commit operation, which causes an execution error, a remote target database may have completed the commit operation but cannot return to a synchronization service of the target, in which case, the synchronization service of the target needs to reconnect the target database, query a registered transaction ID in an auxiliary table to confirm whether the last transaction was successfully committed when failed, and, when the commit operation is performed on the target database, the database is abnormally halted to cause the execution error while waiting for the return of the commit operation, and the database may have completed the commit operation but has not yet come back to the synchronization service when halted, in which case, the synchronization service of the target needs to reconnect the target database, query the registered transaction ID in the auxiliary table to confirm whether the last transaction was successfully committed when failed, therefore, the invention can confirm whether the synchronous affair is successfully submitted when the fault occurs by inquiring whether the auxiliary table contains the affair ID, and can carry out the affair synchronization again when the fault occurs, so as to ensure the consistency of the synchronous data.
Please refer to fig. 4, which is a flowchart of the step S400, including the following steps:
s401, when the current operation is a transaction submitting operation, disconnecting the target end database and reconnecting the target end database;
s402, calling an auxiliary table in a target end database, and inquiring whether a transaction ID registered when the synchronous transaction starts exists in the auxiliary table;
s403, when the transaction ID does not exist in the auxiliary table, inserting the synchronous transaction ID into the auxiliary table in the target end database and restarting the transaction synchronization operation.
The invention confirms whether the synchronous data is successfully submitted or not by inquiring the transaction ID, and ensures the Consistency of the synchronous data under the condition of grid or database failure by utilizing the ACID characteristics (Atomicity, Consistency, Isolation and Durability) of the transaction when the auxiliary table and the table to be synchronized are in the same transaction.
Preferably, the step S403 further includes:
when the transaction ID exists in the secondary table, the next synchronized transaction is executed.
In other words, when the transaction ID is determined to exist in the auxiliary table, the present invention can confirm that the synchronization has been successful, thereby ending the synchronization of the current transaction and transferring to the synchronization of the next transaction, so as to ensure the consistency of the synchronization data.
For the synchronous transaction in the above specific embodiment, when the transaction commit operation is executed on the target end database, if the execution fails, the connection needs to be disconnected and the target end database needs to be reconnected, then whether the auxiliary table contains the transaction is queried, and the query SELECT TRXID FROM DMHS _ TRXID WHERE TRXID is executed as 12345; if a result set with the transaction ID of 12345 is found, which indicates that the transaction is successfully synchronized, ending the synchronization of the current transaction and turning to the synchronization of the next transaction; otherwise the transaction is resynchronized.
As shown in fig. 5, based on the data consistency verification method when data synchronization is abnormal, the present invention further provides a data consistency verification device when data synchronization is abnormal, where the data consistency verification device when data synchronization is abnormal may be a mobile terminal, a desktop computer, a notebook computer, a palm computer, a server, or other computing devices. The data consistency checking device comprises a processor 10, a memory 20 and a display 30 when the data synchronization is abnormal. FIG. 5 shows only some of the components of the data consistency check apparatus in the event of a data synchronization anomaly, but it is to be understood that not all of the shown components need be implemented, and that more or fewer components may be implemented instead.
The storage 20 may be an internal storage unit of the data consistency check device when the data synchronization is abnormal in some embodiments, for example, a hard disk or a memory of the data consistency check device when the data synchronization is abnormal. In other embodiments, the memory 20 may also be an external storage device of the data consistency check device when the data synchronization is abnormal, for example, a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, which is equipped on the data consistency check device when the data synchronization is abnormal.
Further, the memory 20 may include both an internal storage unit of the data consistency check device and an external storage device when the data synchronization is abnormal. The memory 20 is used for storing application software installed in the data consistency check device when the data synchronization is abnormal, and various types of data, such as program codes of the data consistency check device when the data synchronization is abnormal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In an embodiment, the memory 20 stores a data consistency check program 40 when the data synchronization is abnormal, and the data consistency check program 40 may be executed by the processor 10 when the data synchronization is abnormal, so as to implement the data consistency check method when the data synchronization is abnormal according to the embodiments of the present application.
The processor 10 may be a Central Processing Unit (CPU), a microprocessor or other data Processing chip in some embodiments, and is used to run program codes stored in the memory 20 or process data, such as performing a data consistency check method when the data synchronization is abnormal.
The display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch panel, or the like in some embodiments. The display 30 is used for displaying information of the data consistency check device when the data synchronization is abnormal and displaying a visual user interface. The components 10-30 of the data consistency check device communicate with each other via the system bus when the data synchronization is abnormal.
In one embodiment, the following steps are implemented when the processor 10 executes the data consistency check program 40 when the data synchronization in the memory 20 is abnormal:
receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation;
when an error report notification of a target end database is received, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence;
when the current operation is the transaction change operation, judging whether the error notification belongs to a preset error type, rolling back the performed synchronization operation when the error notification belongs to the preset error type, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again;
when the current operation is a transaction submitting operation, the target end database is reconnected, whether the auxiliary table contains the transaction ID of the synchronous transaction is inquired, and when the auxiliary table does not contain the transaction ID of the synchronous transaction, the transaction ID of the synchronous transaction is inserted into the auxiliary table and the transaction synchronization operation is carried out again.
The transaction change operation includes an insert operation, an update operation, and a delete operation.
Further, the step of performing transaction synchronization operation after receiving the log message packet sent by the source end and inserting the transaction ID of the synchronization transaction in the log message packet into the auxiliary table created in the database of the target end includes:
analyzing a log message packet where the current synchronous transaction is located and acquiring a transaction ID of the synchronous transaction in the log message packet;
and performing an insertion operation in an auxiliary table created in the target end database to insert the transaction ID into the auxiliary table.
Further, the step of determining whether the error notification belongs to a preset error notification type when the current operation is a transaction change operation, and rolling back the performed synchronization operation and inserting the transaction ID of the synchronized transaction into the auxiliary table again and performing the transaction synchronization operation again when the error notification belongs to the preset error type includes:
when the current operation is a transaction change operation, identifying the type of an error according to an error code returned by a target end database;
judging whether the error type belongs to a preset error type or not;
and when the error type is the preset error type, disconnecting the target end database, rolling back the performed synchronization operation, and after the rolling back is completed, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again.
Further, when the error type is a preset error type, the step of disconnecting the target-side database and rolling back the performed synchronization operation, and after the rolling back is completed, re-inserting the transaction ID of the synchronized transaction into the auxiliary table and re-performing the transaction synchronization operation further includes:
and when the error type is not the preset error type, executing a preset error processing flow of the synchronous transaction.
Further, the preset error types include network exception errors and database system level errors.
Further, when the current operation is a transaction commit operation, the step of reconnecting the target-side database, querying whether the auxiliary table contains the transaction ID of the synchronized transaction, and when the auxiliary table does not contain the transaction ID of the synchronized transaction, inserting the transaction ID of the synchronized transaction into the auxiliary table and performing the transaction synchronization operation again includes:
when the current operation is a transaction submitting operation, disconnecting the target end database and reconnecting the target end database;
calling an auxiliary table in a target end database, and inquiring whether a transaction ID registered when the synchronous transaction starts exists in the auxiliary table;
and when the transaction ID does not exist in the auxiliary table, inserting the synchronous transaction ID into the auxiliary table in the target end database and restarting the transaction synchronization operation.
Further, the step of inserting the synchronized transaction ID into the auxiliary table in the target end database and resuming the transaction synchronization operation when the transaction ID does not exist in the auxiliary table further includes:
when the transaction ID exists in the secondary table, the next synchronized transaction is executed.
Please refer to fig. 6, which is a functional block diagram of a system for verifying data consistency when a data synchronization exception is installed according to a preferred embodiment of the present invention. In this embodiment, the system for installing the data consistency check program when the data synchronization is abnormal may be divided into one or more modules, and the one or more modules are stored in the memory 20 and executed by one or more processors (in this embodiment, the processor 10) to complete the present invention. For example, in fig. 6, the system of installing the data consistency check program when the data synchronization is abnormal may be divided into a transaction tag insertion module 21, an operation judgment module 22, a change operation error processing module 23, and a commit operation error processing module 24. The module referred to in the present invention refers to a series of computer program instruction segments capable of performing specific functions, and is more suitable than a program for describing the execution process of the data consistency check program in the data consistency check device when the data synchronization is abnormal. The following description will specifically describe the functionality of the modules 21-24.
The transaction tag insertion module 21 is configured to receive a log message packet sent by a source end, insert a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a database of a target end, and then perform a transaction synchronization operation;
an operation judgment module 22, configured to, when an error notification of the target-side database is received, judge what operation the current synchronous transaction is in, where the operation includes a transaction change operation and a transaction commit operation that are executed in sequence;
a change operation error processing module 23, configured to, when the current operation is a transaction change operation, determine whether the error notification belongs to a preset error type, and roll back the performed synchronization operation when the error notification belongs to the preset error type, and insert the transaction ID of the synchronized transaction into the auxiliary table again and perform a transaction synchronization operation again;
and the commit operation error processing module 24 is configured to, when the current operation is a transaction commit operation, reconnect the target-side database, query whether the auxiliary table contains the transaction ID of the synchronized transaction, and insert the transaction ID of the synchronized transaction into the auxiliary table and perform a transaction synchronization operation again when the auxiliary table does not contain the transaction ID of the synchronized transaction.
Wherein the transaction change operation comprises an insert operation, an update operation and a delete operation.
Further, the transaction tag inserting module 21 specifically includes:
the transaction label acquisition unit is used for analyzing the log message packet where the current synchronous transaction is located and acquiring the transaction ID of the synchronous transaction in the log message packet;
and the transaction label inserting unit is used for executing an inserting operation in the auxiliary table created in the target end database so as to insert the transaction ID into the auxiliary table.
The change operation error processing module 23 specifically includes:
the error receiving unit is used for identifying the type of an error according to an error code returned by the target end database when the current operation is a transaction change operation;
the error type judging unit is used for judging whether the error type belongs to a preset error type or not;
and the first error processing unit is used for disconnecting the target end database and rolling back the performed synchronization operation when the error type is a preset error type, and inserting the transaction ID of the synchronization transaction into the auxiliary table again after the rolling back is finished and performing the transaction synchronization operation again.
Preferably, the first error processing unit is further configured to execute a preset error processing procedure of the synchronized transaction when the error type is not a preset error type.
Preferably, the preset error types include network exception errors and database system level errors.
The commit operation error processing module 24 specifically includes:
the connection disconnection unit is used for disconnecting the connection with the target end database and reconnecting the target end database when the current operation is a transaction submission operation;
the query unit is used for calling an auxiliary table in a target end database and querying whether a transaction ID registered when the synchronous transaction starts exists in the auxiliary table;
and the second error processing unit inserts the synchronous transaction ID into the auxiliary table in the target end database and restarts to perform transaction synchronization operation when the transaction ID does not exist in the auxiliary table.
Preferably, the second error handling unit is further configured to execute a next synchronization transaction when the transaction ID exists in the auxiliary table.
In summary, in the data consistency checking method, device and storage medium for data synchronization exception provided by the present invention, the method includes: firstly, receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation; then when receiving an error report notification of a target end database, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence; when the current operation is a transaction change operation, judging whether an error notification belongs to a preset error type, rolling back the performed synchronization operation and inserting the transaction ID of the synchronization transaction into the auxiliary table again when the error notification belongs to the preset error type, and performing the transaction synchronization operation again, when the current operation is a transaction commit operation, reconnecting the target end database, inquiring whether the auxiliary table contains the transaction ID of the synchronization transaction, and when the auxiliary table does not contain the transaction ID of the synchronization transaction, inserting the transaction ID of the synchronization transaction into the auxiliary table and performing the transaction synchronization operation again. In the synchronization process, the transaction label is registered in the auxiliary table when the synchronization transaction starts, and if an error occurs in the synchronization process, whether the transaction label is registered is judged according to the current operation, so that whether the synchronization is finished is further judged, and the consistency of the synchronization data is ensured.
Of course, it will be understood by those skilled in the art that all or part of the processes of the methods of the above embodiments may be implemented by a computer program instructing relevant hardware (such as a processor, a controller, etc.), and the program may be stored in a computer readable storage medium, and when executed, the program may include the processes of the above method embodiments. The storage medium may be a memory, a magnetic disk, an optical disk, etc.
It is to be understood that the invention is not limited to the examples described above, but that modifications and variations may be effected thereto by those of ordinary skill in the art in light of the foregoing description, and that all such modifications and variations are intended to be within the scope of the invention as defined by the appended claims.
Claims (7)
1. A data consistency checking method when data synchronization is abnormal is characterized by comprising the following steps:
receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation;
when an error report notification of a target end database is received, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence;
when the current operation is the transaction change operation, judging whether the error notification belongs to a preset error type, rolling back the performed synchronization operation when the error notification belongs to the preset error type, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again; the preset error types comprise network abnormal errors and database system level errors;
when the current operation is a transaction submitting operation, the target end database is reconnected, a result set corresponding to the transaction ID is inquired after the transaction ID of the synchronous transaction registered in the auxiliary table is inquired, and when the result set of the transaction ID is not inquired, the transaction ID of the synchronous transaction is reinserted into the auxiliary table and the transaction synchronizing operation is carried out again;
the receiving source end sends a log message packet, and the step of performing transaction synchronization operation after inserting the transaction ID of the synchronization transaction in the log message packet into an auxiliary table created in a target end database comprises the following steps:
analyzing a log message packet where the current synchronous transaction is located and acquiring a transaction ID of the synchronous transaction in the log message packet;
performing an insert operation in an auxiliary table created in a target-side database to insert the transaction ID into the auxiliary table;
when the current operation is a transaction submitting operation, the target end database is reconnected, the result set corresponding to the transaction ID is inquired after the transaction ID of the synchronous transaction registered in the auxiliary table is inquired, and when the result set of the transaction ID is not inquired, the transaction ID of the synchronous transaction is reinserted into the auxiliary table and the transaction synchronizing operation is carried out again, the steps comprise:
when the current operation is a transaction submitting operation, disconnecting the target end database and reconnecting the target end database;
calling an auxiliary table in a target end database, and inquiring a result set corresponding to the transaction ID after inquiring the transaction ID registered when the synchronous transaction starts in the auxiliary table;
and when the result set of the transaction ID is not inquired, reinserting the transaction ID of the synchronous transaction into the auxiliary table and performing the transaction synchronization operation again.
2. The method according to claim 1, wherein the step of determining whether the error notification belongs to a preset error type when the current operation is a transaction change operation, and rolling back the performed synchronization operation and inserting the transaction ID of the synchronized transaction into the auxiliary table again and performing the transaction synchronization operation again when the error notification belongs to the preset error type comprises:
when the current operation is a transaction change operation, identifying the type of an error according to an error code returned by a target end database;
judging whether the error type belongs to a preset error type or not;
and when the error type is the preset error type, disconnecting the target end database, rolling back the performed synchronization operation, and after the rolling back is completed, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again.
3. The method for checking data consistency when data synchronization is abnormal according to claim 2, wherein the step of disconnecting the target database and rolling back the performed synchronization operation when the error type is a preset error type, and reinserting the transaction ID of the synchronized transaction into the auxiliary table and performing the transaction synchronization operation again after the rolling back is completed further comprises:
and when the error type is not the preset error type, executing a preset error processing flow of the synchronous transaction.
4. The method for checking data consistency when data synchronization is abnormal according to claim 1, wherein the step of reinserting the transaction ID of the synchronized transaction into the auxiliary table and resuming the transaction synchronization operation when the result set of the transaction ID is not queried further comprises:
when the result set of the transaction ID is queried, the next synchronized transaction is executed.
5. The method for checking data consistency when data synchronization is abnormal according to claim 1, wherein the transaction change operation comprises an insert operation, an update operation and a delete operation.
6. A data consistency check device when data synchronization is abnormal is characterized by comprising: a processor, a memory, and a communication bus;
the memory has stored thereon a computer readable program executable by the processor;
the communication bus realizes connection communication between the processor and the memory;
the processor, when executing the computer readable program, implements the steps of the data consistency checking method when data synchronization is abnormal according to any one of claims 1 to 5.
7. A computer-readable storage medium, storing one or more programs, which are executable by one or more processors, to implement the steps of the data consistency checking method when data synchronization is abnormal according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910436428.3A CN110427422B (en) | 2019-05-23 | 2019-05-23 | Data consistency checking method and equipment during data synchronization abnormity and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910436428.3A CN110427422B (en) | 2019-05-23 | 2019-05-23 | Data consistency checking method and equipment during data synchronization abnormity and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110427422A CN110427422A (en) | 2019-11-08 |
| CN110427422B true CN110427422B (en) | 2020-10-30 |
Family
ID=68407549
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910436428.3A Active CN110427422B (en) | 2019-05-23 | 2019-05-23 | Data consistency checking method and equipment during data synchronization abnormity and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110427422B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111221800A (en) * | 2019-12-19 | 2020-06-02 | 中国平安人寿保险股份有限公司 | Database migration method and device, electronic equipment and storage medium |
| CN111367705B (en) * | 2020-03-31 | 2023-08-04 | 中国工商银行股份有限公司 | SQL error reporting processing method and device |
| CN111737331B (en) * | 2020-06-22 | 2023-08-18 | 中国银行股份有限公司 | Transaction consistency processing method and system for database and object storage |
| CN112307118B (en) * | 2020-09-30 | 2024-03-22 | 武汉达梦数据库股份有限公司 | Method for guaranteeing data consistency based on log analysis synchronization and synchronization system |
| CN113761056A (en) * | 2021-04-07 | 2021-12-07 | 西安京迅递供应链科技有限公司 | Distributed service processing method, device and system |
| CN113282393A (en) * | 2021-06-04 | 2021-08-20 | 银盛支付服务股份有限公司 | Method for solving task scheduling facing to multiple portrait label jobs |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105338093A (en) * | 2015-11-16 | 2016-02-17 | 中国建设银行股份有限公司 | Data synchronizing method and system |
| US9436752B2 (en) * | 2003-10-02 | 2016-09-06 | Progress Software Corporation | High availability via data services |
| CN106484807A (en) * | 2016-09-23 | 2017-03-08 | 北京许继电气有限公司 | Build the method and system of autonomous controlled data storehouse dual-active or many aggregated structures living |
| CN106775959A (en) * | 2016-12-06 | 2017-05-31 | 上海亿账通互联网科技有限公司 | Distributed transaction processing method and system |
| CN109241185A (en) * | 2018-08-27 | 2019-01-18 | 武汉达梦数据库有限公司 | A kind of method and data synchronization unit that data are synchronous |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9965359B2 (en) * | 2014-11-25 | 2018-05-08 | Sap Se | Log forwarding to avoid deadlocks during parallel log replay in asynchronous table replication |
| CN109657007A (en) * | 2018-12-17 | 2019-04-19 | 武汉达梦数据库有限公司 | Database in phase based on asynchronous submission executes method and apparatus parallel |
-
2019
- 2019-05-23 CN CN201910436428.3A patent/CN110427422B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9436752B2 (en) * | 2003-10-02 | 2016-09-06 | Progress Software Corporation | High availability via data services |
| CN105338093A (en) * | 2015-11-16 | 2016-02-17 | 中国建设银行股份有限公司 | Data synchronizing method and system |
| CN106484807A (en) * | 2016-09-23 | 2017-03-08 | 北京许继电气有限公司 | Build the method and system of autonomous controlled data storehouse dual-active or many aggregated structures living |
| CN106775959A (en) * | 2016-12-06 | 2017-05-31 | 上海亿账通互联网科技有限公司 | Distributed transaction processing method and system |
| CN109241185A (en) * | 2018-08-27 | 2019-01-18 | 武汉达梦数据库有限公司 | A kind of method and data synchronization unit that data are synchronous |
Non-Patent Citations (4)
| Title |
|---|
| "Synchronization and Recovery in an Embedded Database System for Read-Only Transactions";Subhash Bhalla et al.;《International Conference on Embedded and Ubiquitous Computing》;20051231;全文 * |
| "基于事务日志的数据同步模型研究";钟容 等;《科技创新与应用》;20160430(第10期);全文 * |
| "基于操作日志的完井数据同步模型";吴雅娟 等;《计算机系统应用》;20150527;第24卷(第5期);全文 * |
| "数据库同步技术的研究与实现";刘娟娟 等;《软件工程》;20170131;第20卷(第1期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110427422A (en) | 2019-11-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110427422B (en) | Data consistency checking method and equipment during data synchronization abnormity and storage medium | |
| CN110347746B (en) | Heterogeneous database synchronous data consistency checking method and device | |
| CN107517110B (en) | Single board configuration self-recovery method and device in distributed system | |
| CN107357595B (en) | Bus-based multi-driver firmware online updating system | |
| CN111865466B (en) | Time synchronization method, device, time-given equipment and storage medium | |
| CN111400104B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN110969417A (en) | Government affair item synchronization method, device, system, computer equipment and storage medium | |
| CN108170551B (en) | Crawler system based front-end and back-end error processing method, server and storage medium | |
| CN103678423A (en) | Data file input system, device and method | |
| CN102841825A (en) | Drive backup method and device and drive restoration method and device | |
| CN106874126A (en) | Host process method for detecting abnormality in a kind of software development | |
| CN110222119A (en) | A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database | |
| CN113573344A (en) | SMF session detection method based on 5G and terminal | |
| CN112068963B (en) | Distributed scheduling system of embedded system | |
| CN111400085A (en) | Transaction rollback exception processing method and device, computer equipment and storage medium | |
| US8285775B2 (en) | Expedited transaction failure handling by leveraging a reliable message transport protocol to assist detection of discarded processing | |
| CN114090050A (en) | Robot software remote automatic updating method, system and related products | |
| CN109669867B (en) | Test apparatus, automated test method, and computer-readable storage medium | |
| CN103312728B (en) | Method of data synchronization between a kind of storage system, system and device | |
| CN115470031A (en) | Abnormal event correction method, device, equipment and medium | |
| CN110611583B (en) | System service method, server and storage medium | |
| CN116630044A (en) | Abnormal transaction processing method, device, equipment and storage medium | |
| CN117453825A (en) | Data synchronization method, device, medium and storage equipment | |
| CN114816485A (en) | Version updating method and device applied to battery swapping system, electronic equipment and storage medium | |
| CN116450180A (en) | Remote upgrading method for terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Patentee after: Wuhan dream database Co.,Ltd. Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Patentee before: WUHAN DAMENG DATABASE Co.,Ltd. |
|
| CB03 | Change of inventor or designer information |
Inventor after: Sun Feng Inventor after: Yu Yuanlan Inventor after: Zhao Jiawei Inventor before: Fu Quan Inventor before: Sun Feng Inventor before: Yu Yuanlan Inventor before: Zhao Jiawei |
|
| CB03 | Change of inventor or designer information | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220914 Address after: 430073 16-19 / F, building C3, future science and technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province Patentee after: Wuhan dream database Co.,Ltd. Patentee after: HUAZHONG University OF SCIENCE AND TECHNOLOGY Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Patentee before: Wuhan dream database Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230725 Address after: 16-19/F, Building C3, Future Science and Technology Building, No. 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province, 430206 Patentee after: Wuhan dream database Co.,Ltd. Address before: 430073 16-19 / F, building C3, future science and technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province Patentee before: Wuhan dream database Co.,Ltd. Patentee before: HUAZHONG University OF SCIENCE AND TECHNOLOGY |
|
| TR01 | Transfer of patent right |