CN116610752A - Transactional distributed data synchronization method, device, system and storage medium - Google Patents
Transactional distributed data synchronization method, device, system and storage medium Download PDFInfo
- Publication number
- CN116610752A CN116610752A CN202310579061.7A CN202310579061A CN116610752A CN 116610752 A CN116610752 A CN 116610752A CN 202310579061 A CN202310579061 A CN 202310579061A CN 116610752 A CN116610752 A CN 116610752A
- Authority
- CN
- China
- Prior art keywords
- snapshot
- data
- transaction
- submitting
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 238000012545 processing Methods 0.000 claims abstract description 40
- 230000008569 process Effects 0.000 claims abstract description 39
- 238000011084 recovery Methods 0.000 claims abstract description 13
- 238000004891 communication Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 11
- 230000001360 synchronised effect Effects 0.000 claims description 9
- 239000000758 substrate Substances 0.000 claims 2
- 238000010586 diagram Methods 0.000 description 14
- 230000007246 mechanism Effects 0.000 description 11
- 230000002159 abnormal effect Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Retry When Errors Occur (AREA)
Abstract
The invention provides a transactional distributed data synchronization method, device, system and storage medium, which are used for solving the technical problem that data consistency cannot be ensured in the process of synchronizing data from a data source to a database table without a main key by using a Flink. The invention divides the process of the flank cluster for carrying out distributed data synchronization into a pre-commit stage and a formal commit stage, creates a snapshot in the pre-commit stage, commits snapshot data to a target database in a pre-commit mode of database transactions, and formally commits the snapshot data to a target database table through the formal commit stage after the snapshot is successfully created. According to the technical scheme, the processing state of the snapshot is recorded, the submitting state of the transaction is recorded, and under the condition of a Flink fault, the recovery of the data synchronization task is carried out according to the recorded snapshot and the transaction state. The invention can realize the consistency of the data synchronization of the Flink from the distributed data source to the relation database table without the main key.
Description
Technical Field
The present invention relates to the field of communications and cloud computing technologies, and in particular, to a transactional distributed data synchronization method, device, system, and storage medium.
Background
Kafka is an open source stream processing platform developed by the Apache software foundation, written by Scala and Java. Kafka is a high throughput distributed publish-subscribe messaging system. It is a common requirement to achieve data synchronization from a distributed publish-subscribe messaging system (e.g., kafka) to a relational database (e.g., mySQL).
A Flink is a framework and distributed processing engine for stateful computation on both unbounded (stream processing) and bordered (batch processing) data streams. Flink is a common tool to achieve Kafka data synchronization to relational databases. The general data processing flow in a link includes one or more of an input operator task (source), a data processing operator task (transform), and an output operator task (sink).
In the processing flow of synchronizing data in a Kafka cluster to a relational database by using a flank, abnormal phenomena such as restarting of the cluster, rescheduling of tasks and the like usually occur easily due to the distributed characteristics of the flank cluster, manually developed processing logic and the like, and the problem of consistency of source and destination data is usually caused after the cluster is abnormal, for example, repeated service data exists in an output database, lost service data exists in the output database and the like.
Through the stream processing capability and the internal mechanism of the Flink, such as checkpoint (checkpoint) snapshot and other technologies, the consistency of Kafka to MySQL data synchronization can be improved, and particularly under the condition that a corresponding main key exists in a destination MySQL library table, the consistency of the synchronous data can be realized through idempotent SQL in MySQL. However, a primary key is not defined in the MySQL library table of the destination end, and when the Flink cluster is used for synchronizing data from the data source Kafka to MySQL, the problem of data loss or data repetition can easily occur under the abnormal condition of the cluster.
Disclosure of Invention
In view of the above, the present invention provides a transactional distributed data synchronization method, apparatus, system, and storage medium, which are used for solving the technical problem that data consistency cannot be ensured in the process of synchronizing data from a data source to a database table without a primary key by using a flank.
Based on one aspect of the embodiment of the invention, the invention provides a transactional distributed data synchronization method, which is applied to a Flink cluster, and comprises the following steps:
the method comprises the steps that a flank cluster is used for synchronizing data in a distributed data source to a target keyless relational database table, namely a target database table, and the process is divided into a pre-submitting stage and a formal submitting stage;
In the pre-commit stage, snapshot creation and pre-commit of snapshot data are completed, and an output operator task sink is responsible for pre-commit of the snapshot data to a target database table and records a snapshot completion state and a transaction pre-commit state;
in the formal commit phase, the job manager Jobmanager is responsible for instructing sink operator tasks to formally commit snapshot data to the destination database table after the snapshot is created successfully and recording transaction commit results.
Further, the pre-commit phase includes:
the job manager Jobmanager marks the currently synchronized snapshot data by injecting checkpoint boundaries;
the input operator task source records snapshot offsets after detecting checkpoint boundaries;
after detecting a check point boundary, the sink operator task is responsible for caching current snapshot data and pre-submitting the current snapshot data to a target database table and recording a transaction pre-submitting state;
each operator task sends snapshot completion feedback to the Jobmanager after the snapshot data is successfully processed, and the sink operator task sends pre-submission result feedback to the Jobmanager after the snapshot data is successfully submitted;
and after the Jobmanager receives snapshot completion feedback of all operator tasks and pre-submitting result feedback of sink operator tasks, recording a snapshot success state.
Further, the step of formally submitting the stage comprises the following steps:
after Jobmanager judges that creation is successful and pre-submission is successful according to snapshot completion feedback and pre-submission result feedback, snapshot completion information is sent to all operator tasks;
and after receiving the snapshot completion message, the output operator task sink formally submits snapshot data to a target database table and records a transaction submitting result whether the transaction formally submits successfully or not.
Further, if the flank cluster fails during the pre-commit phase and results in unsuccessful snapshot creation, the method further includes the following failure recovery steps:
jobmanager obtains snapshot information of a snapshot which is successfully created last time from a state storage back end;
jobmanager coordinates the task manager to resume the data synchronization task;
acquiring a transaction identification ID and snapshot data of a snapshot which is successfully created last time from a state storage back end according to the snapshot information by a sink operator task;
and the sink operator task acquires a transaction submitting state according to the transaction identification ID, verifies whether the snapshot data of the snapshot which is successfully created in the last time is successfully submitted to a target database table, and if so, the source operator task acquires the snapshot data offset to be consumed next time, and resumes the data synchronization process.
Further, if the flank cluster fails in the pre-commit stage and the last snapshot creation is successful, the method further includes the following failure recovery steps:
jobmanager obtains snapshot information of a snapshot which is successfully created last time;
jobmanager coordinates the task manager to resume execution of the data synchronization task;
acquiring a transaction identification ID and snapshot data of a snapshot which is successfully created last time according to the snapshot information by a sink operator task;
the sink operator task acquires the transaction submitting state according to the transaction ID and judges whether the corresponding snapshot data is formally submitted to the target database table;
under the condition that the transaction is not formally submitted, the sink operator task submits snapshot data of the snapshot which is successfully created last time to a target database table;
the source operator task acquires the next snapshot data offset to be consumed and continues to perform data synchronization.
Further, the Jobmanager and each operator task record snapshot information in a state storage back end, wherein the state storage back end is a distributed file system; the distributed data source is a distributed publishing and subscribing message system; the sink operator task records the transaction state in a distributed memory database.
Based on another aspect of the embodiment of the invention, the invention also provides a transactional distributed data synchronization system, which comprises a Flink cluster, a state storage back end and a distributed memory database;
the Flink cluster comprises a job manager Jobmanager and a task manager Taskanager, wherein at least an input operator task source and an output operator task sink are operated in the task manager Taskanager;
the Flink cluster is used for synchronizing data in the distributed data source into a target database table which is a target database table without a main key relation, and the data synchronization process is divided into a pre-submission stage and a formal submission stage;
in the pre-submitting stage, snapshot creation and pre-submitting of snapshot data are completed, and an output operator sink operator task is responsible for pre-submitting the snapshot data to a target database table and recording a snapshot completion state and a transaction pre-submitting state;
in the formal submitting stage, the job manager Jobmanager is responsible for instructing a sink operator task to formally submit snapshot data to a target database table after the snapshot is successfully created and recording a transaction submitting result;
the state storage back end is used for storing snapshot information and caching snapshot data;
the distributed memory database is used for storing transaction states.
Further, in the pre-commit phase:
the Jobmanager marks currently synchronized snapshot data through injecting check point boundaries;
the source operator task records snapshot offsets after detecting a checkpoint boundary;
after detecting a check point boundary, the sink operator task is responsible for caching current snapshot data and pre-submitting the current snapshot data to a target database table and recording a transaction pre-submitting state;
each operator task sends snapshot completion feedback to the Jobmanager after the snapshot data is successfully processed, and the sink operator task sends pre-submission result feedback to the Jobmanager after the snapshot data is successfully submitted;
the Jobmanager is also used for receiving snapshot completion feedback of all operator tasks and pre-submitting result feedback of sink operator tasks and recording snapshot success states.
Further, in the formal commit phase:
the Jobmanager is also used for sending snapshot completion information to all operator tasks after the creation success and the pre-submission success are judged according to snapshot completion feedback and pre-submission result feedback;
and after receiving the snapshot completion message, the output operator task sink formally submits snapshot data to a target database table and records a transaction submission result of whether the transaction formally submits successfully or not.
Based on an aspect of the embodiments of the present invention, the present invention further provides a transactional distributed data synchronization apparatus, where the apparatus is configured to synchronize data in a distributed data source to a destination keyless relational database table, that is, a destination database table, through a link cluster, where the apparatus includes:
the pre-submitting module is used for completing the processing steps of the pre-submitting stage in the data synchronization process and completing snapshot creation and pre-submitting of snapshot data; the method comprises the steps of pre-submitting snapshot data to a target database table through an output operator task sink and recording a snapshot completion state and a transaction pre-submitting state;
the formal submitting module is used for completing the processing steps of the formal submitting stage in the data synchronization process and formally submitting the snapshot data in a target database table in a transaction mode; and after the snapshot is successfully created, the job manager Jobmanager instructs the sink operator task to formally submit snapshot data to the target database table, record the transaction submission state and record the transaction submission result.
The device provided by the invention can be realized in a mode of software, hardware or a combination of software and hardware. When implemented as a software module, the program code of the software module is loaded into a storage medium of the device, and the program code in the storage medium is read and executed by a processor, thereby implementing the functions of the respective constituent modules in the apparatus.
Based on an aspect of the embodiment of the invention, the invention further provides electronic equipment, which comprises a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium complete communication with each other through the communication bus;
a storage medium storing a computer program;
a processor for performing one or more steps of the transactional distributed data synchronization method as described above when executing a computer program stored on a storage medium.
The invention provides a transactional distributed data synchronization method, device, system and storage medium, which are used for solving the technical problem that data consistency cannot be ensured in the process of synchronizing data from a data source to a database table without a main key by using a Flink. The invention divides the process of the flank cluster for carrying out distributed data synchronization into a pre-commit stage and a formal commit stage, creates a snapshot in the pre-commit stage, commits snapshot data to a target database in a pre-commit mode of database transactions, and formally commits the snapshot data to a target database table through the formal commit stage after the snapshot is successfully created. According to the technical scheme, the processing state of the snapshot is recorded, the submitting state of the transaction is recorded, and under the condition of a Flink fault, the recovery of the data synchronization task is carried out according to the recorded snapshot and the transaction state. The invention can realize the consistency of the data synchronization of the Flink from the distributed data source to the relation database table without the main key.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will briefly describe the drawings required to be used in the embodiments of the present invention or the description in the prior art, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings of the embodiments of the present invention for a person having ordinary skill in the art.
FIG. 1 is a schematic diagram of a process for implementing data synchronization based on a Flink two-phase distributed transaction state mechanism in an embodiment of the invention;
FIG. 2A is a diagram illustrating a pre-commit and pre-commit process initiated during a pre-commit phase according to an embodiment of the present invention;
FIG. 2B is a diagram illustrating a pre-commit phase confirming completion of a pre-commit phase according to an embodiment of the present invention;
FIG. 2C is a diagram illustrating a second stage of the present invention, namely a formal commit stage, according to an embodiment of the present invention;
FIG. 3A is a schematic diagram of an exception recovery procedure in the case that the first-stage snapshot creation is unsuccessful according to an embodiment of the present invention;
FIG. 3B is a schematic diagram of an exception recovery procedure in case of successful first-stage snapshot creation according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device for implementing a transactional distributed data synchronization method according to an embodiment of the invention.
Detailed Description
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the invention. As used in this embodiment of the invention, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be understood that although the terms first, second, third, etc. may be used in embodiments of the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one from another or similar information, entity or step, but not to describe a particular sequence or order. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. Furthermore, the word "if" as used may be interpreted as "at … …" or "at … …" or "in response to a determination". The "and/or" in the present invention is merely an association relationship describing the association object, and indicates that three relationships may exist, for example, a and/or B may indicate: there are three cases, a alone, a and B together, and B alone, wherein a, B may be singular or plural. Also, in the description of the present invention, unless otherwise indicated, "a plurality" means two or more than two. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.
The Flink is a distributed stream processing framework, can execute any stream data program in a parallel and pipeline mode, and can execute batch processing and stream processing programs when the Flink is in pipeline operation. The Checkpoint snapshot mechanism of the link is the cornerstone of its reliability. When a task fails during operation, the task can be restored to a state before the failure according to the Checkpoint snapshot information, and then the operation of the task is restored from the state. In the Flink, a distributed snapshot algorithm is adopted by a Checkpoint snapshot mechanism, so that the semantics of just Once (exact on) inside the Flink program are ensured, and the consistency of data inside the Flink is ensured.
The Flink framework comprises two important components, namely a management coordination component JobManager (job manager) and an execution process component TaskManager (task manager), and the Flink framework also conforms to the Master-Slave framework design principle, wherein JobManager is equivalent to a Master node of the whole cluster, and only one active JobManager, taskManager is equivalent to a Slave node of the whole cluster.
JobManager is responsible for scheduling of the entire Flink cluster task and management of resources, and TaskManager is responsible for specific task execution and resource application and management of the corresponding task on each node. JobManager obtains submitted applications from the client, allocates corresponding TaskSlot resources for the submitted applications according to the use condition of task slots TaskSlot on the TaskManager in the cluster, and instructs the TaskManager to start the applications obtained from the client.
The client side compiles and packages the compiled Flink application, submits the compiled Flink application to the JobManager, and then the JobManager distributes tasks to the task manager nodes with the resources according to the resource conditions of the task manager registered in the JobManager, and then starts and runs the tasks.
In a scene of realizing data synchronization from Kafka to a relational database by using a Flink, a snapshot offset of a 'state storage back end' for storing service data processed by a source operator task in the Flink cluster is added simultaneously based on a Flink frame native snapshot Checkpoint mechanism, the snapshot offset is a data starting position consumed by the source operator task from Kafka, the sink operator task is connected with the database, and the processed service data is output and written into a database table. And when the cluster or task abnormality is encountered, the Flink is restored to the state when the previous snapshot is successful, and the processing of the next snapshot data is continued, but the implementation process suffers from the following two problems:
problem one: if the snapshot is unsuccessful but partial data is written into the database table, at the moment, the flank cluster abnormality occurs, the task is restarted, the snapshot is recovered again from the last time, and at the moment, the data which are put in storage can be repeatedly processed, so that repeated data are inserted into the database table.
And a second problem: if the snapshot is successful, but snapshot data is not completely written into the database, a flank cluster abnormality occurs at the moment, and after a task is restarted, the next snapshot data processing is continued from the last successful snapshot, in this case, a part of service data is lost because part of snapshot data is not put in storage.
Therefore, based on the checkpoint snapshot mechanism of the link native, in the above case, only the data consistency between the data source and the link can be ensured, while in the sink output terminal, in the case that the destination database table has no primary key, the consistency between the data source and the service data in the destination database table cannot be ensured.
The invention provides a transactional distributed data synchronization scheme for solving the technical problem that data consistency cannot be ensured in the process of synchronizing data from a data source to a database table without a primary key by using a Flink. The core idea of the technical scheme of the invention is as follows: dividing the process of using the Flink cluster to perform distributed data synchronization into a pre-commit stage and a formal commit stage, wherein a sink operator task commits snapshot data to a target database in a pre-commit mode of a database transaction in the pre-commit stage, and after receiving snapshot success feedback of all operator tasks, jobManager records the snapshot success state and notifies the sink operator task to formally commit the snapshot data to the target database table, and after the data is successfully committed, the sink operator task records the transaction commit state. According to the technical scheme, the processing state of the snapshot is recorded, the submitting state of the transaction is recorded, and under the condition of a Flink fault, the recovery of the data synchronization task is carried out according to the recorded snapshot and the transaction state. The invention can realize the consistency of the data synchronization of the Flink from the distributed data source (such as a distributed publish-subscribe message system, kafka and the like) to the tables of the keyless relational database (such as MySQL).
Based on the basic idea of the invention, a specific implementation of the invention is described below in connection with the accompanying drawings and the specific embodiments.
FIG. 1 is a schematic diagram of a process for implementing data synchronization based on a Flink two-phase distributed transaction state mechanism in accordance with an embodiment of the present invention. The embodiment uses a Flink to divide a data synchronization process based on a distributed transaction state mechanism for realizing a distributed data source to a keyless relational database table into two stages, wherein the first stage is a pre-commit stage and the second stage is a formal commit stage. The coordination management component JobManager in the Flink cluster is used as a coordinator to coordinate the creation of the snapshot and the submission of the transaction in two phases, and each operator task managed by the execution process component TaskManager in the Flink cluster is used as a participant to be responsible for data synchronization (data acquisition, data processing and data output). The JobManager initiates the second stage processing steps after completing the pre-commit phase of the first stage. JobManager and TaskManager cooperate to accomplish distributed data synchronization based on a transaction mechanism.
The invention can ensure the consistency of synchronous data in the Flink and the data consistency of the Sink output end through the transactional writing process of two stages, thereby ensuring the consistency of the data from a data source to a database table without a main key relation.
The following describes the processing procedure in two stages in detail, respectively.
FIG. 2A is a diagram illustrating a pre-commit and pre-commit process initiated during a pre-commit phase according to an embodiment of the present invention; FIG. 2B is a diagram illustrating a pre-commit phase confirming completion of a pre-commit phase according to an embodiment of the present invention. The data source for data synchronization in the figure is Kafka, and the invention is not limited to the type and specific form of the data source, for example, the data source can also be other distributed message subscription and release components, other data acquisition systems and the like. The destination end of the data synchronization in the figure is a MySQL database table without a primary key index (without a primary key), and the invention is not limited to the type of the destination relational database, for example, mySQL, SQLServer, oracle and the like. In the figure, jobManager and TaskManager are self-components of the Flink engine, the distributed file system HDFS is used as a state storage back end for caching snapshot offset, snapshot data and the like, and the distributed memory database Redis is a newly introduced component for caching transaction state information and the like.
The processing performed in the first stage based on fig. 2A and 2B includes:
step 200, a job manager in the link cluster injects a check point boundary into a data input operator Source to trigger a snapshot creation process;
The job manager Jobmanager in the Flink cluster is responsible for scheduling and executing the data synchronization job, and after the client submits the data synchronization job to the Jobmanager, the Jobmanager schedules the job tasks of the data synchronization job to operators (Source, map, sink) of the task manager Taskanager to run the operator tasks.
The link itself provides a Checkpoint snapshot function, which is implemented by injecting a Checkpoint boundary, and the Checkpoint boundary can be regarded as a snapshot mark, which is mainly used to distinguish between two Checkpoint snapshots of the Checkpoint before and after, where a snapshot corresponds to a batch of service data that can be processed by multiple task operators respectively. Checkpointing is a special data stream triggered by a Jobmanager component that starts with a data entry operator (source) task to inject a checkpointing boundary that flows along with the entire data processing flow as a normal data stream, so that each operator detects the checkpointing boundary and processes accordingly.
Step 201, recording snapshot offset after detecting a checkpoint boundary in the execution process of a flank cluster data input operator task source;
And after receiving an instruction of injecting Jobmanager into a check point boundary from Kafka consumption data, the data input source operator task marks the consumed data with a snapshot, and records the snapshot offset corresponding to the check point boundary (namely the position information of the currently consumed Kafka data) to a state storage back end.
The state storage backend used in the embodiments of the present invention may be a distributed file system (for example Hadoop Distributed File System, HDFS), or may be other types of distributed file systems or databases, which is not limited by the present invention.
Step 202, after detecting a check point boundary, a data output operator task sink in the flank cluster starts a transaction, and caches snapshot data and a transaction ID;
and after detecting a check point boundary, each Sink operator task starts a database transaction for the current snapshot data, sequences the snapshot data marked by the check point boundary, and sequences and caches the snapshot data and the transaction identification ID to a state storage back end for use in cluster fault recovery. The sink operator task may also use other storage locations to cache snapshot data of the sequence number, and not store the snapshot state information such as snapshot offset.
Step 203, pre-submitting current snapshot data to a target database table by a sink operator task and recording a pre-submitting state of a transaction;
and after the pre-submission of the snapshot data is completed, the Sink operator task also needs to record the pre-submission state of the snapshot data in the distributed memory database dis, namely, mark the transaction state of the part of the snapshot data responsible for the Sink operator task as the pre-submission state (non-submission mode), and need to record the success or failure result of the transaction ID and the pre-submission.
The transaction in the invention is consistent with the concept of the transaction in the database, when one transaction is started in the database, before the Commit Commit instruction is executed, the adding/deleting/modifying operation of the database table is not actually effective, the current transaction can be rolled back to the state before the transaction is started at any time through the roller back Rollback instruction, and the processing procedure of the pre-Commit stage in the invention is equivalent to the processing procedure before the Commit Commit instruction is executed in the database transaction.
In the invention, the Sink operator task writes the snapshot data into the target database in a pre-writing mode of the database transaction, and the snapshot data is in a state that an informal Commit is submitted to a target database table although the snapshot data is actually written into the database. Only after all sink operator tasks complete the pre-Commit of all snapshot data of the current batch and the transaction pre-Commit status is recorded, the Jobmanager can further trigger the sink operator tasks to execute the Commit Commit operation of formally committing the snapshot data to the target database table.
Step 204, after finishing the processing of the pre-submitting stage of the snapshot data respectively responsible for each operator task in the flank cluster, sending snapshot completion feedback and pre-submitting result feedback to the management coordination component;
step 205, after receiving snapshot completion feedback and pre-submission result feedback of all operator task feedback, the management coordination component records the snapshot success state of the injected checkpoint boundary marks;
this step corresponds to the example of fig. 2B, where after each operator task completes the processing of the current snapshot-related pre-commit phase, snapshot completion feedback and pre-commit result feedback are sent to the management coordination component Jobmanager.
For example, after completing the record of the snapshot offset to the state storage backend, the source operator task may send snapshot completion feedback to the Jobmanager; after the sink operator task finishes caching the snapshot data at the back end of the state storage, snapshot completion feedback is sent to the Jobmanager, and after the sink operator task finishes pre-submitting the snapshot data and records the pre-submitting state, pre-submitting result feedback is sent to the Jobmanager.
And after receiving feedback of all operators about the current snapshot and the pre-submitted state, the management coordination component Jobmanager judges that all operator tasks successfully complete respective processing steps according to feedback results, and records a snapshot success state to a state storage back end, wherein the snapshot success state information can comprise a checkpoint boundary mark, a snapshot identifier, snapshot time and the like. The Jobmanager can confirm the first stage completion only after each operator task successfully completes the snapshot creation and the Sink operator task successfully completes the pre-commit of the synchronization data and records the transaction state. If the operator task does not successfully complete snapshot creation or pre-submission of snapshot data, the first stage of the current snapshot, namely the pre-submission stage, is indicated to be failed in processing, so that data synchronization of two-stage transaction states of the current snapshot is failed, and corresponding fault processing steps are needed to be executed at the moment.
In some embodiments of the present invention, source and sink operator tasks will typically be available, and normally there may be multiple Map data processing operators between source and sink for processing data synchronization. Map operator tasks generally belong to stateless operators and may not record state information.
FIG. 2C is a schematic diagram illustrating a second stage of the present invention, namely, a formal commit stage, according to an embodiment of the present invention, the process of executing the second stage based on FIG. 2C includes:
step 206, the job manager sends snapshot completion information to all operator tasks;
after the job manager Jobmanager writes the state of the snapshot creation success to the state store backend, a snapshot complete message is sent to all operators (Source, map, sink) indicating that each operator task pre-commit phase is complete.
Step 207, after receiving the snapshot completion message, the sink operator task formally submits the current snapshot data to the target database table and records the transaction commit status as a commit completion status;
after receiving the notification message of the pre-Commit stage completion, the sink operator task formally commits the snapshot data Commit responsible for the sink operator task to the target database table, records the successful state information of the transaction Commit to the distributed memory database rediss after the Commit is successful, so that the snapshot data corresponding to the injected checkpoint parting line is successfully synchronized to the target relational database table, and the second stage process based on the distributed transaction state mechanism is successfully completed.
The invention adopts two-stage distributed data synchronization based on a transaction mechanism, which has the important technical effects that the data consistency from Kafka to a target database can be realized under the condition of a Flink cluster fault, and if the Flink cluster breaks down, the conditions of normal feedback of all operators on the current snapshot, operator feedback creation snapshot failure, sink operator task feedback pre-submission failure and the like are not received by Jobmanager within preset time. The failure of the snapshot data synchronization marked by the checkpoints may occur in the pre-commit phase or in the formal commit phase, and the failure recovery method is described below for both cases.
Abnormal case one: the data synchronization operation process is in a first-stage pre-submitting stage when fault is interrupted, and the current Checkpoint snapshot is unsuccessful
Whether the snapshot is successfully created is judged by judging whether the Jobmanager receives snapshot completion feedback information which is sent by all operator tasks and indicates that the snapshot is successfully created, if the snapshot completion feedback information of all operator tasks is received, the Jobmanager judges that the snapshot is successfully created, and at the moment, the Jobmanager records snapshot creation success state record information to the state storage back end. In some cases, the Map operator task is stateless, and the snapshot completion feedback message may not be sent to the Jobmanager, where the Jobmanager mainly determines whether the snapshot creation is successful or not by determining whether the snapshot completion feedback sent by all source and sink operator tasks is received.
FIG. 3A is a schematic diagram of an abnormal recovery step in the case that the first-stage snapshot creation is unsuccessful in an embodiment of the present invention, if a Flink cluster fails before Jobmanager records status record information of the snapshot creation success, the following steps are performed after the cluster is restarted:
step 310, the Jobmanager obtains snapshot information of a last created successful snapshot (a previous successful snapshot of a last failed snapshot) from the state storage backend;
the snapshot information of the last successfully created snapshot may include corresponding checkpoint boundary information (snapshot identification), snapshot offset, snapshot creation time, etc.
Step 311, jobmanager coordinates and schedules the task to resume the execution of the data synchronization task;
step 312, the sink operator task obtains the transaction ID and snapshot data of the snapshot which is successfully created last time from the state storage back end according to the snapshot information;
step 313, the sink operator task acquires the transaction commit status according to the transaction ID, and verifies that the transaction commit status is committed;
the method comprises the steps that a sink operator task acquires cached serialized snapshot data from a state storage rear end, the transaction ID and the snapshot data are recovered through deserialization, whether transaction submission corresponding to a snapshot which is created successfully last time is successful or not is judged from redis according to the transaction ID, if so, the subsequent step is continuously executed, if not, the corresponding snapshot data are submitted to a target database table, and then the subsequent step is executed.
In step 314, the source operator task acquires the shift of the snapshot data (the latest failed snapshot data) to be consumed next from the state storage back end, and resumes the data synchronization process of creating the snapshot and the like.
FIG. 3B is a schematic diagram of an exception recovery step in the case of successful snapshot creation in the first stage according to an embodiment of the present invention, if a flank cluster fails after the Jobmanager records that the last snapshot creation was successful, in this case, a part of the sink operator task commit transactions may be successful, but the second stage has not been completed successfully, the following steps are performed after the cluster is restarted:
step 320, the Jobmanager obtains snapshot information of the snapshot which is successfully created last time from the state storage back end;
step 321, jobmanager coordinates and schedules the task to resume the execution of the data synchronization task;
step 322, the sink operator task acquires the transaction ID and snapshot data of the snapshot which is successfully created last time from the state storage back end according to the snapshot information;
step 323, the sink operator task obtains the transaction commit status according to the transaction ID;
step 324, in the uncommitted state of the transaction (in the target database table in which the snapshot data is not formally committed), the sink operator task commits the uncommitted transaction after the recovery to the target database table;
In step 325, the source operator task acquires the next starting point offset to be consumed (the offset of different data blocks in the same snapshot) from the state storage back end (through the deserialization operation) to continue data synchronization.
A snapshot identified by a checkpoint boundary corresponds to a batch of data in Kafka, multiple sources may consume the batch of data in Kafka in parallel, a source operator task may process multiple data blocks in the batch of data belonging to the same snapshot, each operator task may identify a snapshot based on the checkpoint boundary, and by the data block identification, which data block is processed.
If the Flink cluster fails in the second stage of the snapshot data synchronization process, the snapshot creation is successful, so that the processing process is similar to that of FIG. 3B, a sink operator task judges whether the transaction is submitted successfully according to the transaction ID, if not, the transaction is formally submitted, and if so, the snapshot data of the same batch in Kafka is continuously consumed to execute data synchronization.
Fig. 4 is a schematic structural diagram of an electronic device for implementing the transactional distributed data synchronization method according to an embodiment of the present invention, where the device 400 includes: a processor 410 such as a Central Processing Unit (CPU), a communication bus 420, a communication interface 440, and a memory 430. Wherein the processor 410 and the memory 430 may communicate with each other via a communication bus 420. The memory 430 has stored therein a computer program which, when executed by the processor 410, performs the functions of one or more steps of the transactional distributed data synchronization method provided by the present invention.
Memory refers to a device for storing computer programs and/or data based on some storage medium, which may be a Volatile Memory (VM) or a Non-Volatile Memory (NVM). The memory is an internal memory for directly exchanging data with the processor, and can read and write data at any time, and has high speed, and is used as a storage medium for temporary data of an operating system and other running programs. The memory may be synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), or the like. The nonvolatile memory is a memory using a persistent storage medium, and has a large capacity and can store data permanently, and may be a storage class memory (Storage Class Memory, SCM), a Solid State Disk (SSD), a NAND flash memory, a magnetic Disk, or the like. SCM is a common name for new storage medium between memory and flash memory, and is a composite storage technology combining persistent storage characteristic and memory characteristic, and has access speed slower than that of DRAM and SSD hard disk.
The processor may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
It should be appreciated that embodiments of the invention may be implemented or realized by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory (or referred to as non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Furthermore, the operations of the processes described in the present invention may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes (or variations and/or combinations thereof) described herein may be performed under control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications), by hardware, or combinations thereof, collectively executing on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable computing platform, including, but not limited to, a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and so forth. Aspects of the invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and/or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, which when read by a computer, is operable to configure and operate the computer to perform the processes described herein. Further, the machine readable code, or portions thereof, may be transmitted over a wired or wireless network. When such media includes instructions or programs that, in conjunction with a microprocessor or other data processor, implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media. The invention also includes the computer itself when programmed according to the methods and techniques of the present invention.
The foregoing is merely exemplary of the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (13)
1. A method for transactional distributed data synchronization, the method being applied to a Flink cluster, the method comprising:
the method comprises the steps that a flank cluster is used for synchronizing data in a distributed data source to a target keyless relational database table, namely a target database table, and the process is divided into a pre-submitting stage and a formal submitting stage;
in the pre-commit stage, snapshot creation and pre-commit of snapshot data are completed, and an output operator task sink is responsible for pre-commit of the snapshot data to a target database table and records a snapshot completion state and a transaction pre-commit state;
in the formal commit phase, the job manager Jobmanager is responsible for instructing sink operator tasks to formally commit snapshot data to the destination database table after the snapshot is created successfully and recording transaction commit results.
2. The method of claim 1, wherein the pre-commit phase comprises:
The job manager Jobmanager marks the currently synchronized snapshot data by injecting checkpoint boundaries;
the input operator task source records snapshot offsets after detecting checkpoint boundaries;
after detecting a check point boundary, the sink operator task is responsible for caching current snapshot data and pre-submitting the current snapshot data to a target database table and recording a transaction pre-submitting state;
each operator task sends snapshot completion feedback to the Jobmanager after the snapshot data is successfully processed, and the sink operator task sends pre-submission result feedback to the Jobmanager after the snapshot data is successfully submitted;
and after the Jobmanager receives snapshot completion feedback of all operator tasks and pre-submitting result feedback of sink operator tasks, recording a snapshot success state.
3. The method of claim 2, wherein the step of formally submitting the phase comprises:
after Jobmanager judges that creation is successful and pre-submission is successful according to snapshot completion feedback and pre-submission result feedback, snapshot completion information is sent to all operator tasks;
and after receiving the snapshot completion message, the output operator task sink formally submits snapshot data to a target database table and records a transaction submitting result whether the transaction formally submits successfully or not.
4. The method of claim 2, wherein if the Flink cluster fails during the pre-commit phase and results in unsuccessful snapshot creation, the method further comprises the step of failure recovery:
jobmanager obtains snapshot information of a snapshot which is successfully created last time from a state storage back end;
jobmanager coordinates the task manager to resume the data synchronization task;
acquiring a transaction identification ID and snapshot data of a snapshot which is successfully created last time from a state storage back end according to the snapshot information by a sink operator task;
and the sink operator task acquires a transaction submitting state according to the transaction identification ID, verifies whether the snapshot data of the snapshot which is successfully created in the last time is successfully submitted to a target database table, and if so, the source operator task acquires the snapshot data offset to be consumed next time, and resumes the data synchronization process.
5. The method of claim 3, wherein if the flank cluster failed during the pre-commit phase and the last snapshot creation was successful, the method further comprises the step of recovering from the failure:
jobmanager obtains snapshot information of a snapshot which is successfully created last time;
jobmanager coordinates the task manager to resume execution of the data synchronization task;
Acquiring a transaction identification ID and snapshot data of a snapshot which is successfully created last time according to the snapshot information by a sink operator task;
the sink operator task acquires the transaction submitting state according to the transaction ID and judges whether the corresponding snapshot data is formally submitted to the target database table;
under the condition that the transaction is not formally submitted, the sink operator task submits snapshot data of the snapshot which is successfully created last time to a target database table;
the source operator task acquires the next snapshot data offset to be consumed and continues to perform data synchronization.
6. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the Jobmanager and each operator task record snapshot information in a state storage back end, wherein the state storage back end is a distributed file system;
the distributed data source is a distributed publish-subscribe messaging system.
7. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the sink operator task records the transaction state in a distributed memory database.
8. The transactional distributed data synchronization system is characterized by comprising a Flink cluster, a state storage back end and a distributed memory database;
the Flink cluster comprises a job manager Jobmanager and a task manager Taskanager, wherein at least an input operator task source and an output operator task sink are operated in the task manager Taskanager;
The Flink cluster is used for synchronizing data in the distributed data source into a target database table which is a target database table without a main key relation, and the data synchronization process is divided into a pre-submission stage and a formal submission stage;
in the pre-submitting stage, snapshot creation and pre-submitting of snapshot data are completed, and an output operator sink operator task is responsible for pre-submitting the snapshot data to a target database table and recording a snapshot completion state and a transaction pre-submitting state;
in the formal submitting stage, the job manager Jobmanager is responsible for instructing a sink operator task to formally submit snapshot data to a target database table after the snapshot is successfully created and recording a transaction submitting result;
the state storage back end is used for storing snapshot information and caching snapshot data;
the distributed memory database is used for storing transaction states.
9. The system of claim 8, wherein in the pre-commit phase:
the Jobmanager marks currently synchronized snapshot data through injecting check point boundaries;
the source operator task records snapshot offsets after detecting a checkpoint boundary;
after detecting a check point boundary, the sink operator task is responsible for caching current snapshot data and pre-submitting the current snapshot data to a target database table and recording a transaction pre-submitting state;
Each operator task sends snapshot completion feedback to the Jobmanager after the snapshot data is successfully processed, and the sink operator task sends pre-submission result feedback to the Jobmanager after the snapshot data is successfully submitted;
the Jobmanager is also used for receiving snapshot completion feedback of all operator tasks and pre-submitting result feedback of sink operator tasks and recording snapshot success states.
10. The system of claim 9, wherein in the formal commit phase:
the Jobmanager is also used for sending snapshot completion information to all operator tasks after the creation success and the pre-submission success are judged according to snapshot completion feedback and pre-submission result feedback;
and after receiving the snapshot completion message, the output operator task sink formally submits snapshot data to a target database table and records a transaction submission result of whether the transaction formally submits successfully or not.
11. A transactional distributed data synchronization apparatus for synchronizing data in a distributed data source to a destination keyless relational database table, i.e., a destination database table, via a flank cluster, the apparatus comprising:
the pre-submitting module is used for completing the processing steps of the pre-submitting stage in the data synchronization process and completing snapshot creation and pre-submitting of snapshot data; the method comprises the steps of pre-submitting snapshot data to a target database table through an output operator task sink and recording a snapshot completion state and a transaction pre-submitting state;
The formal submitting module is used for completing the processing steps of the formal submitting stage in the data synchronization process and formally submitting the snapshot data in a target database table in a transaction mode; and after the snapshot is successfully created, the job manager Jobmanager instructs the sink operator task to formally submit snapshot data to the target database table, record the transaction submission state and record the transaction submission result.
12. An electronic device is characterized by comprising a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium are communicated with each other through the communication bus;
a storage medium storing a computer program;
a processor for carrying out the method steps of any one of claims 1-7 when executing a computer program stored on a storage medium.
13. A storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310579061.7A CN116610752A (en) | 2023-05-19 | 2023-05-19 | Transactional distributed data synchronization method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310579061.7A CN116610752A (en) | 2023-05-19 | 2023-05-19 | Transactional distributed data synchronization method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116610752A true CN116610752A (en) | 2023-08-18 |
Family
ID=87683073
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310579061.7A Pending CN116610752A (en) | 2023-05-19 | 2023-05-19 | Transactional distributed data synchronization method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116610752A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117171213A (en) * | 2023-09-11 | 2023-12-05 | 上海轴芯科技有限公司 | Big data supervision system and method based on heterogeneous computer system |
| CN117763052A (en) * | 2024-02-22 | 2024-03-26 | 浩鲸云计算科技股份有限公司 | data synchronization method and system for charging multi-center memory database |
-
2023
- 2023-05-19 CN CN202310579061.7A patent/CN116610752A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117171213A (en) * | 2023-09-11 | 2023-12-05 | 上海轴芯科技有限公司 | Big data supervision system and method based on heterogeneous computer system |
| CN117171213B (en) * | 2023-09-11 | 2024-02-02 | 上海轴芯科技有限公司 | Big data supervision system and method based on heterogeneous computer system |
| CN117763052A (en) * | 2024-02-22 | 2024-03-26 | 浩鲸云计算科技股份有限公司 | data synchronization method and system for charging multi-center memory database |
| CN117763052B (en) * | 2024-02-22 | 2024-05-10 | 浩鲸云计算科技股份有限公司 | Data synchronization method and system for charging multi-center memory database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11914572B2 (en) | Adaptive query routing in a replicated database environment | |
| US10275507B2 (en) | Replication of a relational database | |
| CN116610752A (en) | Transactional distributed data synchronization method, device, system and storage medium | |
| Zhou et al. | Foundationdb: A distributed unbundled transactional key value store | |
| US9128881B2 (en) | Recovery for long running multithreaded processes | |
| Wang et al. | Lineage stash: fault tolerance off the critical path | |
| US20110082832A1 (en) | Parallelized backup and restore process and system | |
| US20160378785A1 (en) | Distributed work flow using database replication | |
| WO2017181931A1 (en) | Method and device for processing distributed transaction | |
| US9652492B2 (en) | Out-of-order execution of strictly-ordered transactional workloads | |
| CN111753013A (en) | Distributed transaction processing method and device | |
| WO2021238273A1 (en) | Message fault tolerance method and system based on spark streaming computing framework | |
| CN109063005B (en) | Data migration method and system, storage medium and electronic device | |
| CN104657158A (en) | Method and device for processing business in business system | |
| US20140236887A1 (en) | Data seeding optimization for database replication | |
| CN109783578B (en) | Data reading method and device, electronic equipment and storage medium | |
| US11874816B2 (en) | Lock free distributed transaction coordinator for in-memory database participants | |
| CN115562911B (en) | Virtual machine data backup method, device, system, electronic equipment and storage medium | |
| Mendizabal et al. | High performance recovery for parallel state machine replication | |
| US6772367B1 (en) | Software fault tolerance of concurrent programs using controlled re-execution | |
| WO2022242372A1 (en) | Object processing method and apparatus, computer device, and storage medium | |
| CN111352704A (en) | Distributed global transaction processing system and method based on policy management | |
| US20240134840A1 (en) | Distributed table lock application methods, apparatuses, storage media, and electronic devices | |
| CN113986450A (en) | Virtual machine backup method and device | |
| Azim et al. | Efficient mode changes in multi-mode systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |