CN112434043A - Data synchronization method, device, electronic equipment and medium - Google Patents
Data synchronization method, device, electronic equipment and medium Download PDFInfo
- Publication number
- CN112434043A CN112434043A CN202011404776.1A CN202011404776A CN112434043A CN 112434043 A CN112434043 A CN 112434043A CN 202011404776 A CN202011404776 A CN 202011404776A CN 112434043 A CN112434043 A CN 112434043A
- Authority
- CN
- China
- Prior art keywords
- cdc
- data
- change
- timestamp
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data synchronization method, a data synchronization device, electronic equipment and a medium, and relates to the technical field of storage. The method comprises the following steps: acquiring a new synchronization relation, wherein the new synchronization relation is used for indicating that data in a source table of a source database is synchronized to a target table of a target database, then capturing a log file of the source database by a CDC producer calling a database log analysis component through data change, extracting change data of the source table from the log file, storing the extracted change data in a middleware, further acquiring the change data of the source table from the middleware through a CDC consumer based on the new synchronization relation, and synchronizing the change data of the source table to the target table. The data synchronization between the databases can be realized, and the configuration is simpler.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data synchronization method and apparatus, an electronic device, and a medium.
Background
MySQL is a Relational Database Management System (RDBMS) which is widely applied at present, and operation records of MySQL Database are all recorded in binary log (binlog). If the database is down to cause data loss, data redoing and data recovery can be carried out by analyzing the binlog, and data synchronization of the database can be realized based on the binlog in the process of data backup in the database.
The Databus is a databank change capturing system, can pull the change of a databank in real time, but after the change of the pull databank, the data synchronization from a source databank to a target databank can not be realized directly based on the change of the databank, the data synchronization can be realized only by carrying out complex application installation on the source databank and a target databank, and the configuration is complex.
Disclosure of Invention
Embodiments of the present invention provide a data synchronization method, an apparatus, an electronic device, and a medium, which can implement data synchronization between databases and are configured more easily. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a data synchronization method, including:
acquiring a newly added synchronization relationship, wherein the newly added synchronization relationship is used for indicating that data in a source table of a source database is synchronized to a target table of a target database;
capturing a log file of the source database by a CDC (data change Capture) producer through data change, calling a database log analysis component to analyze the log file of the source database, and extracting change data of the source table from the log file;
storing the extracted change data in the middleware;
and based on the newly added synchronization relationship, acquiring the change data of the source table from the middleware through a CDC consumer, and synchronizing the change data of the source table to the target table.
In a possible implementation manner, a plurality of new synchronization relationships exist, each new synchronization relationship corresponds to one CDC consumer, and each source database corresponds to one CDC producer; before the invoking, by the data change capturing CDC producer, a database log parsing component to parse the log file of the source database, and extracting the changed data of the source table from the log file, the method further comprises:
restarting the synchronous task of the source database;
aiming at each newly added synchronization relationship, synchronizing the data in the current time source table to the target table through the CDC consumer corresponding to the newly added synchronization relationship, and acquiring the initialization timestamp of the current time;
determining the maximum value of the initialization timestamp from the initialization timestamps of CDC consumers corresponding to each newly-added synchronization relationship through the CDC producer, and acquiring the maximum value and the minimum value of the change timestamp of the transaction record in the middleware, wherein each transaction record in the middleware corresponds to one change timestamp and is used for indicating the timestamp for starting execution of the transaction to which the change data in the transaction record belongs;
calling the database log analysis component through the CDC producer, and extracting changed data from a log file of the source database by using the system change identification serial number SCN corresponding to the minimum value of the change timestamp;
if the change timestamp of the extracted change data is between the maximum value of the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, storing the extracted change data in the middleware;
the capturing CDC producer through data change calls a database log analysis component to analyze the log file of the source database, and the extracting of the changed data of the source table from the log file comprises the following steps:
and calling a database log analysis component by the CDC producer, and extracting the changed data of the source table from the SCN corresponding to the maximum value of the initialization timestamp in the log file of the source database.
In one possible implementation, if a change timestamp of the extracted change data is between the maximum change timestamp value and the maximum initialization timestamp value, and a source table to which the extracted change data belongs is in a table name set of the CDC producer, the storing the extracted change data in the middleware includes:
if the change timestamp of the extracted change data is between the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, saving the extracted change data into the belonging transaction object;
when the transaction object is committed, saving the transaction object as a transaction record in the middleware.
In one possible implementation, before the invoking, by the data change capture CDC producer, a database log parsing component to parse a log file of the source database, extracting change data of the source table from the log file, the method further includes:
for each CDC consumer corresponding to each newly added synchronization relationship, sequentially comparing the change timestamp of each change data in the middleware with the initialization timestamp of the CDC consumer;
when it is determined by the comparison that the change timestamp of the first target change data is greater than the initialization timestamp of the CDC consumer, determining a location of the target change data in the middleware as an initial offset value for the CDC consumer relative to the middleware;
the step of acquiring the change data of the source table from the middleware through a CDC consumer based on the newly added synchronization relationship, and synchronizing the change data of the source table to the target table comprises:
and based on the newly-added synchronization relationship, acquiring the change data of the source table from the middleware by the CDC consumer corresponding to the newly-added synchronization relationship from the initial offset value of the CDC consumer corresponding to the newly-added synchronization relationship relative to the middleware, and synchronizing the acquired change data of the source table to the target table.
In a possible implementation manner, after the obtaining of the new synchronization relationship, the method further includes:
judging whether a synchronous task corresponding to the source database exists or not;
if the synchronization task corresponding to the source database does not exist, creating the synchronization task corresponding to the source database, storing the newly-added synchronization relationship in a synchronization relationship set corresponding to the source table, and adding the table name of the source table to a table name set of the CDC producer;
and if the synchronization task corresponding to the source database exists and the synchronization relationship set corresponding to the source table does not include the newly added synchronization relationship, storing the newly added synchronization relationship in the synchronization relationship set corresponding to the source table.
In a second aspect, an embodiment of the present application provides a data synchronization apparatus, including:
an obtaining module, configured to obtain a newly added synchronization relationship, where the newly added synchronization relationship is used to instruct to synchronize data in a source table of a source database to a target table of a target database;
the extraction module is used for capturing a log file of the source database by a CDC (data change control) producer through data change, calling a database log analysis component to analyze the log file of the source database, and extracting the change data of the source table from the log file;
the storage module is used for storing the extracted change data in the middleware;
and the synchronization module is used for acquiring the change data of the source table from the middleware through a CDC consumer based on the newly added synchronization relationship and synchronizing the change data of the source table to the target table.
In a possible implementation manner, a plurality of new synchronization relationships exist, each new synchronization relationship corresponds to one CDC consumer, and each source database corresponds to one CDC producer; the device further comprises:
the restarting module is used for restarting the synchronous task of the source database;
the synchronization module is further used for synchronizing the total data in the current time source table to the target table through the CDC consumer corresponding to each newly added synchronization relationship and acquiring the initialization timestamp of the current time;
a determining module, configured to determine, by the CDC producer, a maximum value of an initialization timestamp from the initialization timestamps of the CDC consumers corresponding to each newly added synchronization relationship, and obtain a maximum value of a change timestamp and a minimum value of the change timestamp of a transaction record in the middleware, where each transaction record in the middleware corresponds to one change timestamp, and is used to indicate a timestamp at which a transaction to which change data in the transaction record belongs starts to be executed;
the extraction module is further configured to invoke the database log analysis component through the CDC producer, and extract changed data from the system change identifier serial number SCN corresponding to the minimum change timestamp in the log file of the source database;
the storage module is further configured to store the extracted change data in the middleware if a change timestamp of the change data extracted by the extraction module is between the maximum value of the change timestamp and the maximum value of the initialization timestamp, and a source table to which the extracted change data belongs is in the table name set of the CDC producer;
the extraction module is specifically configured to invoke a database log analysis component by the CDC producer, and start to extract the changed data of the source table from the SCN corresponding to the maximum value of the initialization timestamp in the log file of the source database.
In a possible implementation manner, the storage module is specifically the same as:
if the change timestamp of the extracted change data is between the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, saving the extracted change data into the belonging transaction object;
when the transaction object is committed, saving the transaction object as a transaction record in the middleware.
In a possible implementation manner, the determining module is further configured to, for each CDC consumer corresponding to each newly added synchronization relationship, sequentially compare a change timestamp of each change data in the middleware with an initialization timestamp of the CDC consumer; and when it is determined by the comparison that the change timestamp of the first target change data is greater than the initialization timestamp of the CDC consumer, determining a location of the target change data in the middleware as an initial offset value of the CDC consumer relative to the middleware;
the synchronization module is specifically configured to, based on the newly added synchronization relationship, obtain, from the initial offset value of the CDC consumer corresponding to the newly added synchronization relationship with respect to the middleware, change data of the source table from the middleware, and synchronize the obtained change data of the source table to the target table, through the CDC consumer corresponding to the newly added synchronization relationship.
In one possible implementation, the apparatus further includes:
the judging module is used for judging whether a synchronous task corresponding to the source database exists or not;
a creating module, configured to create a synchronization task corresponding to the source database if a synchronization task corresponding to the source database does not exist, store the newly-added synchronization relationship in a synchronization relationship set corresponding to the source table, and add the table name of the source table to the table name set of the CDC producer;
and the storage module is used for storing the newly added synchronization relationship in the synchronization relationship set corresponding to the source table if the synchronization task corresponding to the source database exists and the synchronization relationship set corresponding to the source table does not include the newly added synchronization relationship.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any one of the steps of the data synchronization method when executing the program stored in the memory.
In a fourth aspect, the present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the data synchronization method described in the first aspect.
In a fifth aspect, embodiments of the present application further provide a computer program product containing instructions, which when run on a computer, cause the computer to perform the data synchronization method described in the first aspect.
By adopting the technical scheme, after the newly-added synchronization relationship is obtained, the log file of the source database can be analyzed by the database log analysis component through the CDC producer, so that the change data of the source table is extracted from the log file, the extracted change data is stored in the middleware, the change data of the source table is obtained from the middleware through the CDC consumer, and the change data of the source table is synchronized to the target table.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is an exemplary schematic diagram of a data synchronization system provided in an embodiment of the present application;
fig. 2 is a schematic structural diagram of a data synchronization system according to an embodiment of the present application;
fig. 3 is a flowchart of a data synchronization method according to an embodiment of the present application;
FIG. 4 is a flowchart illustrating a method for initializing a CDC consumer and a CDC producer in a data synchronization method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For convenience of understanding, an application scenario of the embodiment of the present application is first described. The embodiment of the application can be applied to a data synchronization scene between databases, for example, the embodiment can be used for realizing backup of data in the databases. The embodiment of the application can realize data synchronization among databases of the same type and also can realize data synchronization among databases of different types.
As an example, an embodiment of the present application provides a data synchronization system in a possible application scenario, and as shown in fig. 1, the data synchronization system includes a user operation page, a Mysql database as a source database, an Oracle database, a Mysql database, and an sqserver database as a target database, and further includes a real-time synchronization module for executing the data synchronization method provided in the embodiment of the present application. The real-time synchronization module can comprise an Oracle real-time synchronization module, a Mysql real-time synchronization module and an SQLsever real-time synchronization module.
The Oracle real-time synchronization module is used for synchronizing data in the Mysql database serving as a source database to the Oracle database;
the Mysql real-time synchronization module is used for synchronizing data in a Mysql database serving as a source database to a Mysql database serving as a target database;
the SQLsever real-time synchronization module is used for synchronizing data in the Mysql database serving as the source database to the SQLsever database serving as the target database.
The user operation page may be a web page in the electronic device, and the user may configure the synchronization relationship through the web page, so that the real-time synchronization module performs data synchronization based on the synchronization relationship configured by the user.
An architecture diagram of a data synchronization system is further provided in an embodiment of the present application, as shown in fig. 2, the data synchronization system includes: kafka stores middleware, Mybatis. At the application level, the data synchronization system further comprises a web layer, a Service layer (Service layer), and a data access object (Dao) layer.
The web layer is used for providing a user operation page for a user, the user can configure a synchronization relationship through the user operation page, and the user operation page can also be used for displaying the statistical information of the synchronization task resources.
The Service layer is configured to implement the processing logic of the real-time synchronization module in fig. 1, and is specifically configured to implement the data synchronization method provided in the embodiment of the present application.
And the Dao layer is used for encapsulating access methods and the like of the database so as to ensure the persistence of the task information.
Mybatis is a persistent layer framework used to support storage procedures and also to ensure that task information persists.
And the kafka storage middleware is used for storing the change data extracted from the source table of the source database in the data synchronization process. The middleware in the embodiment of the present application is not limited to the kafka storage middleware, and may also be other message queue middleware, such as RabbitMQ, ActiveMQ, and the like.
The data synchronization method provided by the embodiment of the present application is described in detail below.
As shown in fig. 3, an embodiment of the present application provides a data synchronization method, where the method includes:
s301, acquiring a new synchronization relationship, wherein the new synchronization relationship is used for indicating that data in a source table of a source database is synchronized to a target table of a target database.
The newly added synchronization relationship may be a synchronization relationship configured by a user through a web page, and the synchronization relationship may include an IP address, a port, a database name, and a source table name of a source database, and an IP address, a port, a database name, and a target table name of a target database. Optionally, a user name and password may also be included. If the database supports schema, the synchronization relationship may also include schema information of the source database and the target database.
In the embodiment of the present application, a source data table in a source database is simply referred to as a source table, and a target data table in a target database is simply referred to as a target table.
S302, a producer calls a database log analysis component to analyze a log file of a source database through Data Change capture (CDC), and changed Data of a source table is extracted from the log file.
Wherein, the CDC producer (producer) is an extraction task and can call the database log analysis component to extract the change data of the source table based on the synchronous relation.
In the case that the source database is the Mysql database, the database log parsing component may be a log parsing component of the Mysql database: open-responder. It can be understood that if the source database is another database, the log analysis components of the other databases may be used, so as to implement data synchronization between databases of different types, without requiring complex and special configuration, and having low invasiveness to the source database and less influence on normal operation of the source database.
The database log parsing component may parse binlog of the source database in which data change records of the source database are recorded in the form of event records. For each event record in the binlog, an attribute may be maintained that indicates the position of the event record in the binlog, which identifies a sequence Number (SCN) for a System Change.
And S303, storing the extracted change data in the middleware.
In this embodiment of the present application, the change data of the source table extracted from the log file by the CDC producer may be stored by means of a middleware, specifically, kafka topic, which is a high-performance distributed message queue, and kafka topic, which is a topic of kafka, and is used for distinguishing different types of messages, where each partition in the kafka topic is a message queue corresponding to one type of message. In the embodiment of the application, in order to ensure the continuity of the extracted data, only one partition is set for the kafka topoic, and the messages in the kafka topoc are in transaction units. That is to say, the kafka topic in the embodiment of the present application includes transaction records arranged in time sequence, and each transaction record includes change data corresponding to one data change in the source database.
S304, based on the newly added synchronization relationship, the CDC consumer acquires the change data of the source table from the middleware, and synchronizes the change data of the source table to the target table.
By adopting the technical scheme, after the newly-added synchronization relationship is obtained, the log file of the source database can be analyzed by the database log analysis component through the CDC producer, so that the change data of the source table is extracted from the log file, the extracted change data is stored in the middleware, the change data of the source table is obtained from the middleware through the CDC consumer, and the change data of the source table is synchronized to the target table.
In the embodiment of the present application, data synchronization may be performed based on a synchronization task of a database, where the synchronization task is an integral of all synchronization relationships corresponding to one source database, and therefore after an additional synchronization relationship is obtained, the additional synchronization relationship needs to be added to the synchronization task. Therefore, after the new synchronization relationship is obtained in S301, it is further determined whether a synchronization task corresponding to the source database exists, and based on the determination result, the following two situations are specifically included:
in case one, the synchronization task corresponding to the source database does not exist.
And if the synchronous task corresponding to the source database does not exist, creating the synchronous task corresponding to the source database, storing the newly-added synchronous relationship in the synchronous relationship set corresponding to the source table, and adding the table name of the source table to the table name set of the CDC producer.
After the synchronization task corresponding to the source database is created, each source table of the source database corresponds to one synchronization relationship set, and at this time, the synchronization relationship set corresponding to each source table is empty. The newly added synchronization relationship may then be stored in a synchronization relationship set corresponding to the source table of the newly added synchronization relationship.
In addition, a CDC producer is also created for the source database, that is, an extraction task is created, and a CDC consumer is created for each newly added synchronization relationship, that is, a consumption task is created for each newly added synchronization relationship. After creating the CDC producer for the source database, adding the table name of the source table with the newly added synchronization relationship into a table name set of the CDC so that the CDC producer calls a database log analysis component to extract the change data of the source table corresponding to the table name in the table name set from a log file of the source database.
Each CDC consumer may then be initialized such that each CDC consumer synchronizes a full amount of data in the current time source table to the target table based on the synchronization relationship and records a respective initialization timestamp.
After the CDC consumers complete initialization, the CDC producer is initialized, and the minimum value of the initialization timestamps of the CDC consumers is obtained.
And then calling a database log analysis component by the CDC producer, and extracting the changed data from the minimum value of the initialization time stamp of the log file of the source database. Specifically, the CDC producer may invoke the database log parsing component to parse the event records in the log records of the source database, so as to obtain a timestamp (timestamp) of each event record, and if the timestamp of a certain event record is found to be smaller than the minimum value of the initialization timestamp, and the timestamp of the next event record adjacent to the event record is found to be larger than the minimum value of the initialization timestamp, determine that the SCN corresponding to the event record is the starting SCN, and extract the change data from the event record. The time stamp of an event record is the time stamp of the beginning of execution of the event included in the event record.
The extracted change data is stored in the middleware in the form of a transaction record, the transaction record includes the change data and a change timestamp corresponding to the change data, and the change timestamp is consistent with the timestamp of the event record of the log file, that is, the change timestamp is the timestamp of the start execution of the event to which the change data belongs.
For example, the starting SCN is 3, the time stamp included in the event record corresponding to the starting SCN in the log file is 10:00:00, and the data change operation corresponding to the event record is executed from 10:00: 00. After the CDC producer extracts the changed data based on the event record through the database log analysis component, a new transaction record is added in the middleware, and the transaction record comprises the extracted data and the change timestamp of 10:00: 00.
When a CDC producer extracts a piece of change data, whether the table name to which the change data belongs is in the table name set of the CDC producer needs to be judged; if the data is not in the table name set of the data, discarding the changed data; if the data is in the table name set of the data, the changed data is temporarily stored in the form of a transaction record. Then judging whether the change time stamp of the changed data is larger than the maximum change time stamp in the middleware, if so, storing the transaction record of the changed data in the middleware after the event record corresponding to the changed data is submitted; if not, the transaction record is discarded.
Further, the CDC consumer may obtain, from the middleware, change data of the source table subsequent to the own initialization timestamp based on the own initialization timestamp, and synchronize the change data to the target table based on the synchronization relationship.
And in case two, the synchronization task corresponding to the source database exists.
And if the synchronization task corresponding to the source database exists and the synchronization relationship set corresponding to the source table does not comprise the newly added synchronization relationship, storing the newly added synchronization relationship in the synchronization relationship set corresponding to the source table.
In this embodiment of the present application, if a synchronization task corresponding to a source database exists, all synchronization relationship sets corresponding to the synchronization task are traversed, and whether a synchronization relationship set corresponding to a source table in a newly added synchronization relationship exists is determined.
And if the synchronization relationship set corresponding to the source table in the newly added synchronization relationship does not exist or the synchronization relationship set corresponding to the source table in the newly added synchronization relationship is empty, directly adding the newly added synchronization relationship to the synchronization relationship set corresponding to the source table, and adding the table name of the source table to the table name set of the CDC producer.
If the synchronization relationship set corresponding to the source table in the newly added synchronization relationship exists, traversing the newly added synchronization relationship set corresponding to the source table, and judging whether all the newly added synchronization relationships corresponding to the source table are not in the synchronization relationship set; if so, adding all the newly added synchronization relations corresponding to the source table into a synchronization relation set corresponding to the source table; and if not, displaying the repeatedly configured synchronization relation error information on the web page.
It can be understood that, if a synchronization task corresponding to a source database to which a synchronization relationship is newly added exists, a CDC producer corresponding to the source database already exists currently.
After the new synchronization relationship is stored in the synchronization relationship set corresponding to the source table, a CDC consumer may be created for the new synchronization relationship, where a plurality of new synchronization relationships may exist, each new synchronization relationship corresponds to one CDC consumer, and each source database corresponds to one CDC producer. After the CDC consumer is created for the newly added synchronization relationship, the CDC consumer and the CDC producer need to be initialized. Before the above-mentioned S302 is executed, and the CDC producer calls the database log analysis component to analyze the log file of the source database, and extracts the changed data of the source table from the log file, the initialization process is described as follows, and as shown in fig. 4, the initialization process includes:
s401, restarting a synchronization task of the source database.
S402, aiming at each newly added synchronization relationship, synchronizing the data in the current time source table to the target table in a full-scale mode through the CDC consumer corresponding to the newly added synchronization relationship, and acquiring the initialization timestamp of the current time.
And S403, determining the maximum value of the initialization timestamp from the initialization timestamps of the CDC consumers corresponding to the newly-added synchronization relationships through the CDC producer, and acquiring the maximum value and the minimum value of the change timestamp of the transaction record in the middleware.
Each transaction record in the middleware corresponds to a change timestamp, and the change timestamp is used for indicating the timestamp of starting execution of the event to which the change data in the transaction record belongs.
And S404, calling a database log analysis component through the CDC producer, and extracting the changed data from the SCN corresponding to the minimum value of the change timestamp in the log file of the source database.
And S405, if the change timestamp of the extracted change data is between the maximum value of the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, storing the extracted change data in the middleware.
If the extracted time stamp of the changed data is between the maximum value of the changed time stamp and the maximum value of the initialized time stamp, it is indicated that the data in the source database is changed during the restart of the synchronization task of the source database (within the time period from the stop of the synchronization task to the completion of the initialization of the CDC consumer), the changed data of the period of time is not stored in the middleware, in order to complete the synchronization of the changed data of the period of time, the CDC producer needs to extract the changed data between the maximum value of the changed time stamp and the maximum value of the initialized time stamp, and needs to judge whether the changed data is in the own table name set or not every time one changed data is extracted; if the data is not in the table name set of the data, discarding the changed data; if the data is in the table name set of the user, the changed data is stored in the middleware.
Specifically, if the change timestamp of the extracted change data is between the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, the extracted change data is saved in the belonging transaction object. When the transaction object is committed, the transaction object is saved as a transaction record in the middleware. The transaction object includes the extracted change data and a timestamp of when the transaction to which the extracted change data belongs begins execution.
By adopting the method, the CDC consumer and the CDC producer are initialized, so that the CDC producer can extract the changed data in the source database in the restarting process of the synchronous task of the source data, the integrity of the extracted data is ensured, and the condition of data loss can be avoided.
In addition, in the embodiment of the present application, one source table may have a synchronization relationship with a plurality of target tables, but since the CDC producer extracts the change data of the source table from the log file and then stores the change data of the source table in one middleware, the synchronization progress may be different for each target table, and therefore the CDC consumer needs to initialize with respect to the middleware. That is, before the CDC producer invokes the database log parsing component to parse the log file of the source database through data change capture and extracts the changed data of the source table from the log file, the method further comprises:
and aiming at the CDC consumer corresponding to each newly added synchronization relationship, sequentially comparing the change timestamp of each change data in the middleware with the initialization timestamp of the CDC consumer.
When it is determined by the comparison that the change timestamp of the first target change data is greater than the initialization timestamp of the CDC consumer, the position of the target change data in the middleware is taken as an initial offset value (offset) of the CDC consumer with respect to the middleware.
For example, if the initialization timestamp of the CDC consumer is 10:00:15, the change timestamps of the change data included in the middleware are, in order: 10:00:00, 10:00:05, 10:00:10, 10:00:16, 10:00:20, the change data having the change timestamp of 10:00:16 may be the target change data corresponding to the CDC, and the CDC consumer may perform data synchronization from the change data having the change timestamp of 10:00: 16.
By the method, a plurality of CDC consumers corresponding to the same source table can share one middleware, namely, the changed data corresponding to one source table is only stored in one middleware, and data redundancy does not occur. Meanwhile, the synchronization task can be suspended, and after the synchronization task is restarted, each CDC consumer can continue to perform data synchronization based on the initial offset value of the CDC consumer, so that the CDC consumer can avoid repeatedly consuming data, and the phenomenon of synchronous data loss can be avoided.
Based on the initialization flow shown in fig. 4, after the initialization of the CDC consumer and the CDC producer is completed, in step S302, the CDC producer calls the database log analysis component to analyze the log file of the source database, and extracts the change data of the source table from the log file, which may be specifically implemented as:
and calling a database log analysis component by the CDC producer, and starting to extract the changed data of the source table from the SCN corresponding to the maximum value of the initialization timestamp in the log file of the source database. It will be appreciated that the CDC producer extracts from the log file the change data for each source table in its own maintained set of table names.
Since the change data between the maximum value of the change timestamp and the maximum value of the initialization timestamp is already stored in the middleware during the initialization process of the CDC producer, after the synchronization task is restarted, the change data can be continuously extracted from the SCN corresponding to the maximum value of the initialization timestamp, and if the source table to which the extracted change data belongs is in the table name set of the CDC producer, the extracted change data is stored in the middleware in the unit of transaction to ensure the continuity of the data.
Correspondingly, based on the newly added synchronization relationship, the CDC consumer acquires the change data of the source table from the middleware, and synchronizes the change data of the source table to the target table, which can be specifically realized as follows:
and based on the newly added synchronization relationship, acquiring the change data of the source table from the middleware by the CDC consumer corresponding to the newly added synchronization relationship from the initial offset value of the CDC consumer corresponding to the newly added synchronization relationship relative to the middleware, and synchronizing the acquired change data of the source table to the target table.
After the synchronization task of the source database is restarted, the CDC producer can continuously extract the change data, if the source table to which the extracted change data belongs is in the table name set of the CDC producer, the extracted change data is stored in the middleware, and then the CDC consumer continuously acquires the change data from the middleware, synchronizes the acquired change data of the source table to the target table based on the synchronization relation corresponding to the CDC consumer, so that real-time data synchronization is realized.
The method provided by the embodiment of the application can realize the real-time synchronization relationship configuration through the web page, is simple to operate, and is convenient for a user to monitor the synchronization task information in real time. In addition, the method and the device do not need complex configuration operation, only the log record of the source database needs to be started, and the method and the device are simple and convenient to implement.
Corresponding to the foregoing method embodiment, an embodiment of the present application further provides a data synchronization apparatus, as shown in fig. 5, the apparatus includes:
an obtaining module 501, configured to obtain a new synchronization relationship, where the new synchronization relationship is used to instruct to synchronize data in a source table of a source database to a target table of a target database;
an extraction module 502, configured to capture, by data change, a log file of the source database, which is parsed by a database log parsing component invoked by the CDC producer, and extract changed data of the source table from the log file;
a storage module 503, configured to store the extracted change data in the middleware;
and a synchronization module 504, configured to obtain, by the CDC consumer, change data of the source table from the middleware based on the newly added synchronization relationship, and synchronize the change data of the source table to the target table.
Optionally, a plurality of newly added synchronization relationships exist, each newly added synchronization relationship corresponds to one CDC consumer, and each source database corresponds to one CDC producer; the device also includes:
the restarting module is used for restarting the synchronous task of the source database;
the synchronization module 504 is further configured to synchronize, for each newly added synchronization relationship, the total amount of data in the source table at the current time to the target table through the CDC consumer corresponding to the newly added synchronization relationship, and obtain an initialization timestamp at the current time;
the determining module is used for determining the maximum value of the initialization timestamp from the initialization timestamps of the CDC consumers corresponding to the newly-added synchronization relationships through the CDC producer, and acquiring the maximum value and the minimum value of the change timestamp of the transaction record in the middleware, wherein each transaction record in the middleware corresponds to one change timestamp and is used for indicating the timestamp for starting execution of the transaction to which the change data in the transaction record belongs;
the extraction module 502 is further configured to invoke a database log analysis component by the CDC producer, and extract changed data from the SCN corresponding to the minimum value of the change timestamp in the log file of the source database;
the storage module 503 is further configured to store the extracted change data in the middleware if the change timestamp of the change data extracted by the extraction module 502 is between the maximum value of the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer;
optionally, the extraction module 502 is specifically configured to invoke a database log analysis component by the CDC producer, and start to extract the changed data of the source table from the SCN corresponding to the maximum value of the initialization timestamp in the log file of the source database.
Optionally, the storage module 503 is specifically the same as:
if the change timestamp of the extracted change data is between the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, storing the extracted change data into the belonging transaction object;
when a transaction object is committed, the transaction object is saved as a transaction record in the middleware.
Optionally, the determining module is further configured to, for each CDC consumer corresponding to each newly added synchronization relationship, sequentially compare the change timestamp of each change data in the middleware with the initialization timestamp of the CDC consumer; and when it is determined by the comparison that the change timestamp of the first target change data is greater than the initialization timestamp of the CDC consumer, determining a location of the target change data in the middleware as an initial offset value of the CDC consumer relative to the middleware.
Optionally, the synchronization module 504 is specifically configured to, based on the new synchronization relationship, obtain, by the CDC consumer corresponding to the new synchronization relationship, change data of the source table from the middleware starting from an initial offset value of the CDC consumer corresponding to the new synchronization relationship with respect to the middleware, and synchronize the obtained change data of the source table to the target table.
Optionally, the apparatus further comprises:
the judging module is used for judging whether a synchronous task corresponding to the source database exists or not;
the creating module is used for creating the synchronization task corresponding to the source database if the synchronization task corresponding to the source database does not exist, storing the newly-added synchronization relationship in the synchronization relationship set corresponding to the source table, and adding the table name of the source table to the table name set of the CDC producer;
and the storage module is used for storing the newly-added synchronization relationship in the synchronization relationship set corresponding to the source table if the synchronization task corresponding to the source database exists and the synchronization relationship set corresponding to the source table does not include the newly-added synchronization relationship.
An embodiment of the present invention further provides an electronic device, as shown in fig. 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604,
a memory 603 for storing a computer program;
the processor 601 is configured to implement the method steps in the above method embodiments when executing the program stored in the memory 603.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program realizes the steps of any one of the above data synchronization methods when executed by a processor.
In a further embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the data synchronization methods of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device, the electronic apparatus, and the media embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and the relevant points can be referred to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (12)
1. A method of data synchronization, comprising:
acquiring a newly added synchronization relationship, wherein the newly added synchronization relationship is used for indicating that data in a source table of a source database is synchronized to a target table of a target database;
capturing a log file of the source database by a CDC (data change Capture) producer through data change, calling a database log analysis component to analyze the log file of the source database, and extracting change data of the source table from the log file;
storing the extracted change data in the middleware;
and based on the newly added synchronization relationship, acquiring the change data of the source table from the middleware through a CDC consumer, and synchronizing the change data of the source table to the target table.
2. The method of claim 1, wherein there are a plurality of new synchronization relationships, each new synchronization relationship corresponding to a CDC consumer, and each source database corresponding to a CDC producer; before the invoking, by the data change capturing CDC producer, a database log parsing component to parse the log file of the source database, and extracting the changed data of the source table from the log file, the method further comprises:
restarting the synchronous task of the source database;
aiming at each newly added synchronization relationship, synchronizing the data in the current time source table to the target table through the CDC consumer corresponding to the newly added synchronization relationship, and acquiring the initialization timestamp of the current time;
determining the maximum value of the initialization timestamp from the initialization timestamps of CDC consumers corresponding to each newly-added synchronization relationship through the CDC producer, and acquiring the maximum value and the minimum value of the change timestamp of the transaction record in the middleware, wherein each transaction record in the middleware corresponds to one change timestamp and is used for indicating the timestamp for starting execution of the transaction to which the change data in the transaction record belongs;
calling the database log analysis component through the CDC producer, and extracting changed data from a log file of the source database by using the system change identification serial number SCN corresponding to the minimum value of the change timestamp;
if the change timestamp of the extracted change data is between the maximum value of the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, storing the extracted change data in the middleware;
the capturing CDC producer through data change calls a database log analysis component to analyze the log file of the source database, and the extracting of the changed data of the source table from the log file comprises the following steps:
and calling a database log analysis component by the CDC producer, and extracting the changed data of the source table from the SCN corresponding to the maximum value of the initialization timestamp in the log file of the source database.
3. The method of claim 2, wherein saving the extracted change data in the middleware if a change timestamp of the extracted change data is between the change timestamp maximum value and the initialization timestamp maximum value and a source table to which the extracted change data belongs is in a set of table names of the CDC producer comprises:
if the change timestamp of the extracted change data is between the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, saving the extracted change data into the belonging transaction object;
when the transaction object is committed, saving the transaction object as a transaction record in the middleware.
4. The method of claim 2 or 3, wherein before said invoking a database log parsing component by a data change capture CDC producer to parse a log file of said source database, extracting change data of said source table from said log file, said method further comprises:
for each CDC consumer corresponding to each newly added synchronization relationship, sequentially comparing the change timestamp of each change data in the middleware with the initialization timestamp of the CDC consumer;
when it is determined by the comparison that the change timestamp of the first target change data is greater than the initialization timestamp of the CDC consumer, determining a location of the target change data in the middleware as an initial offset value for the CDC consumer relative to the middleware;
the step of acquiring the change data of the source table from the middleware through a CDC consumer based on the newly added synchronization relationship, and synchronizing the change data of the source table to the target table comprises:
and based on the newly-added synchronization relationship, acquiring the change data of the source table from the middleware by the CDC consumer corresponding to the newly-added synchronization relationship from the initial offset value of the CDC consumer corresponding to the newly-added synchronization relationship relative to the middleware, and synchronizing the acquired change data of the source table to the target table.
5. The method of claim 1, wherein after the obtaining of the new synchronization relationship, the method further comprises:
judging whether a synchronous task corresponding to the source database exists or not;
if the synchronization task corresponding to the source database does not exist, creating the synchronization task corresponding to the source database, storing the newly-added synchronization relationship in a synchronization relationship set corresponding to the source table, and adding the table name of the source table to a table name set of the CDC producer;
and if the synchronization task corresponding to the source database exists and the synchronization relationship set corresponding to the source table does not include the newly added synchronization relationship, storing the newly added synchronization relationship in the synchronization relationship set corresponding to the source table.
6. A data synchronization apparatus, comprising:
an obtaining module, configured to obtain a newly added synchronization relationship, where the newly added synchronization relationship is used to instruct to synchronize data in a source table of a source database to a target table of a target database;
the extraction module is used for capturing a log file of the source database by a CDC (data change control) producer through data change, calling a database log analysis component to analyze the log file of the source database, and extracting the change data of the source table from the log file;
the storage module is used for storing the extracted change data in the middleware;
and the synchronization module is used for acquiring the change data of the source table from the middleware through a CDC consumer based on the newly added synchronization relationship and synchronizing the change data of the source table to the target table.
7. The apparatus of claim 6, wherein there are a plurality of new synchronization relationships, each new synchronization relationship corresponding to one CDC consumer, and each source database corresponding to one CDC producer; the device further comprises:
the restarting module is used for restarting the synchronous task of the source database;
the synchronization module is further used for synchronizing the total data in the current time source table to the target table through the CDC consumer corresponding to each newly added synchronization relationship and acquiring the initialization timestamp of the current time;
a determining module, configured to determine, by the CDC producer, a maximum value of an initialization timestamp from the initialization timestamps of the CDC consumers corresponding to each newly added synchronization relationship, and obtain a maximum value of a change timestamp and a minimum value of the change timestamp of a transaction record in the middleware, where each transaction record in the middleware corresponds to one change timestamp, and is used to indicate a timestamp at which a transaction to which change data in the transaction record belongs starts to be executed;
the extraction module is further configured to invoke the database log analysis component through the CDC producer, and extract changed data from the system change identifier serial number SCN corresponding to the minimum change timestamp in the log file of the source database;
the storage module is further configured to store the extracted change data in the middleware if a change timestamp of the change data extracted by the extraction module is between the maximum value of the change timestamp and the maximum value of the initialization timestamp, and a source table to which the extracted change data belongs is in the table name set of the CDC producer;
the extraction module is specifically configured to invoke a database log analysis component by the CDC producer, and start to extract the changed data of the source table from the SCN corresponding to the maximum value of the initialization timestamp in the log file of the source database.
8. The apparatus of claim 7, wherein the storage module is specifically identical to:
if the change timestamp of the extracted change data is between the change timestamp and the maximum value of the initialization timestamp, and the source table to which the extracted change data belongs is in the table name set of the CDC producer, saving the extracted change data into the belonging transaction object;
when the transaction object is committed, saving the transaction object as a transaction record in the middleware.
9. The apparatus according to claim 7 or 8,
the determining module is further configured to, for each CDC consumer corresponding to each newly added synchronization relationship, sequentially compare a change timestamp of each change data in the middleware with an initialization timestamp of the CDC consumer; and when it is determined by the comparison that the change timestamp of the first target change data is greater than the initialization timestamp of the CDC consumer, determining a location of the target change data in the middleware as an initial offset value of the CDC consumer relative to the middleware;
the synchronization module is specifically configured to, based on the newly added synchronization relationship, obtain, from the initial offset value of the CDC consumer corresponding to the newly added synchronization relationship with respect to the middleware, change data of the source table from the middleware, and synchronize the obtained change data of the source table to the target table, through the CDC consumer corresponding to the newly added synchronization relationship.
10. The apparatus of claim 6, further comprising:
the judging module is used for judging whether a synchronous task corresponding to the source database exists or not;
a creating module, configured to create a synchronization task corresponding to the source database if a synchronization task corresponding to the source database does not exist, store the newly-added synchronization relationship in a synchronization relationship set corresponding to the source table, and add the table name of the source table to the table name set of the CDC producer;
and the storage module is used for storing the newly added synchronization relationship in the synchronization relationship set corresponding to the source table if the synchronization task corresponding to the source database exists and the synchronization relationship set corresponding to the source table does not include the newly added synchronization relationship.
11. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
12. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011404776.1A CN112434043B (en) | 2020-12-02 | 2020-12-02 | Data synchronization method, device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011404776.1A CN112434043B (en) | 2020-12-02 | 2020-12-02 | Data synchronization method, device, electronic equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112434043A true CN112434043A (en) | 2021-03-02 |
| CN112434043B CN112434043B (en) | 2022-12-20 |
Family
ID=74692276
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011404776.1A Active CN112434043B (en) | 2020-12-02 | 2020-12-02 | Data synchronization method, device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112434043B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905618A (en) * | 2021-04-06 | 2021-06-04 | 浙江网商银行股份有限公司 | Data processing method and device |
| CN113111129A (en) * | 2021-04-16 | 2021-07-13 | 挂号网（杭州）科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113449043A (en) * | 2021-07-21 | 2021-09-28 | 中国人民解放军61932部队 | Data synchronization method and device, computer equipment and storage medium |
| CN114020850A (en) * | 2022-01-05 | 2022-02-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120059749A1 (en) * | 2010-09-08 | 2012-03-08 | Alcatel-Lucent Bell Nv | Method and apparatus for facilitating interim billing for ims session using time-based interim accounting message to determine interim processing trigger |
| CN102760126A (en) * | 2011-04-26 | 2012-10-31 | 阿里巴巴集团控股有限公司 | Data synchronization method in distributive system, and distributive system |
| CN106682140A (en) * | 2016-12-20 | 2017-05-17 | 华北计算技术研究所（中国电子科技集团公司第十五研究所） | Multi-system user incremental synchronization method based on timestamps and mapping strategies |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
| US20180365146A1 (en) * | 2017-06-20 | 2018-12-20 | International Business Machines Corporation | Identification of software components with event persistency across operation interruptions |
| CN109241175A (en) * | 2018-06-28 | 2019-01-18 | 东软集团股份有限公司 | Method of data synchronization, device, storage medium and electronic equipment |
| CN109379363A (en) * | 2018-10-25 | 2019-02-22 | 北京开普云信息科技有限公司 | A kind of single-sign-on integrated approach and system based on intensive platform |
| CN110309161A (en) * | 2019-06-06 | 2019-10-08 | 新华三大数据技术有限公司 | A kind of method of data synchronization, device and server |
| CN110309231A (en) * | 2019-07-12 | 2019-10-08 | 焦点科技股份有限公司 | A kind of method of data synchronization and system across computer room |
-
2020
- 2020-12-02 CN CN202011404776.1A patent/CN112434043B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120059749A1 (en) * | 2010-09-08 | 2012-03-08 | Alcatel-Lucent Bell Nv | Method and apparatus for facilitating interim billing for ims session using time-based interim accounting message to determine interim processing trigger |
| CN102760126A (en) * | 2011-04-26 | 2012-10-31 | 阿里巴巴集团控股有限公司 | Data synchronization method in distributive system, and distributive system |
| CN106682140A (en) * | 2016-12-20 | 2017-05-17 | 华北计算技术研究所（中国电子科技集团公司第十五研究所） | Multi-system user incremental synchronization method based on timestamps and mapping strategies |
| US20180365146A1 (en) * | 2017-06-20 | 2018-12-20 | International Business Machines Corporation | Identification of software components with event persistency across operation interruptions |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
| CN109241175A (en) * | 2018-06-28 | 2019-01-18 | 东软集团股份有限公司 | Method of data synchronization, device, storage medium and electronic equipment |
| CN109379363A (en) * | 2018-10-25 | 2019-02-22 | 北京开普云信息科技有限公司 | A kind of single-sign-on integrated approach and system based on intensive platform |
| CN110309161A (en) * | 2019-06-06 | 2019-10-08 | 新华三大数据技术有限公司 | A kind of method of data synchronization, device and server |
| CN110309231A (en) * | 2019-07-12 | 2019-10-08 | 焦点科技股份有限公司 | A kind of method of data synchronization and system across computer room |
Non-Patent Citations (4)
| Title |
|---|
| YANG ZHANG; JINGJUN ZHANG; DONGWEN ZHANG: ""Implementing and Testing Producer-Consumer Problem Using Aspect-Oriented Programming"", 《IEEE》 * |
| 刘伟等: "异构数据库集成中的变化捕获方案设计", 《计算机应用研究》 * |
| 刘洋等: "数据库同步技术中变化数据捕获模式的研究与比较", 《科技视界》 * |
| 崔伟等: "分布式系统中数据同步机制的研究与实现", 《计算机工程与设计》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905618A (en) * | 2021-04-06 | 2021-06-04 | 浙江网商银行股份有限公司 | Data processing method and device |
| CN113111129A (en) * | 2021-04-16 | 2021-07-13 | 挂号网（杭州）科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113449043A (en) * | 2021-07-21 | 2021-09-28 | 中国人民解放军61932部队 | Data synchronization method and device, computer equipment and storage medium |
| CN114020850A (en) * | 2022-01-05 | 2022-02-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112434043B (en) | 2022-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112434043B (en) | Data synchronization method, device, electronic equipment and medium | |
| CN108804299B (en) | Application program exception handling method and device | |
| CN110309161B (en) | Data synchronization method and device and server | |
| CN111177165B (en) | Method, device and equipment for detecting data consistency | |
| CN112380093A (en) | Operation and maintenance processing method and device and computer equipment | |
| CN108255620B (en) | Service logic processing method, device, service server and system | |
| WO2019109518A1 (en) | Data list uploading method and terminal thereof | |
| CN112115200B (en) | Data synchronization method, device, electronic equipment and readable storage medium | |
| CN110737719B (en) | Data synchronization method, device, equipment and computer readable storage medium | |
| WO2019057193A1 (en) | Data deletion method and distributed storage system | |
| CN107040576A (en) | Information-pushing method and device, communication system | |
| US10915510B2 (en) | Method and apparatus of collecting and reporting database application incompatibilities | |
| CN115757150A (en) | Production environment testing method, device, equipment and storage medium | |
| CN110018932B (en) | Method and device for monitoring container magnetic disk | |
| CN111078418B (en) | Operation synchronization method, device, electronic equipment and computer readable storage medium | |
| CN113238815A (en) | Interface access control method, device, equipment and storage medium | |
| CN117033058A (en) | Analysis method, device, equipment and medium for software crash data | |
| CN109189864B (en) | Method, device and equipment for determining data synchronization delay | |
| CN116302652A (en) | System alarm information processing method and device and electronic equipment | |
| CN116186082A (en) | Data summarizing method based on distribution, first server and electronic equipment | |
| CN115237889A (en) | Database switching method and device, storage medium and computer equipment | |
| CN114518984A (en) | Reporting method and device of embedded point information, storage medium and terminal equipment | |
| US10037242B2 (en) | Failure detection in a processing system | |
| CN113434509B (en) | Increment index updating method and device, storage medium and electronic equipment | |
| CN115203063B (en) | Playback method and system of production flow re-running risk program based on real-time recording |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |