CN113434600A - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN113434600A CN113434600A CN202110742129.XA CN202110742129A CN113434600A CN 113434600 A CN113434600 A CN 113434600A CN 202110742129 A CN202110742129 A CN 202110742129A CN 113434600 A CN113434600 A CN 113434600A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- data table
- message queue
- updating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data synchronization method and a device, wherein the method comprises the following steps: and acquiring configuration information required by data synchronization, wherein the configuration information at least comprises a message queue address and a table name of a target data table. The target data table is any data table in the source database with the updated internal data. And acquiring the update data of the target data table from the source database according to the table name of the target data table. And storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the target end monitors that the message queue contains the update data, the target end updates the data of the data table to be updated in the database of the target end by using the update data. And determining the data table to be updated according to the table name of the target data table. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data synchronization method and apparatus.
Background
As businesses increase, the businesses of many enterprises are distributed in different locations, resulting in databases that handle different businesses also being distributed in different locations. Enterprises require business data of different databases to be consistent in real time so as to ensure business continuity.
Currently, data synchronization can be performed by providing data transmission interfaces between different databases. However, the throughput of each data transmission interface is limited, and if the data to be processed exceeds the processing upper limit of the data transmission interface, the system may be crashed.
Disclosure of Invention
In order to solve the technical problem, the present application provides a data synchronization method and apparatus, which are used for implementing data synchronization between different databases.
In order to achieve the above purpose, the technical solutions provided in the embodiments of the present application are as follows:
an embodiment of the present application provides a data synchronization method, where the method is applied to a source end, and the method includes:
acquiring configuration information required by data synchronization; the configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in a source database with updated internal data;
acquiring the update data of the target data table from the source database according to the table name of the target data table;
storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when a target end monitors that the message queue contains the update data, the target end updates the data of the data table to be updated in the target end database by using the update data; and the data table to be updated is determined according to the table name of the target data table.
Optionally, before the obtaining the configuration information required for data synchronization, the method further includes:
responding to the service request of the source end, and generating a specified event applied to a target data table;
and based on the specified event, executing an operation corresponding to the specified event on the target data table by using a data trigger configured by the source database so as to complete data updating of the source database.
Optionally, the obtaining configuration information required for data synchronization includes:
after the configuration service in the configuration service management platform determines the configuration information required by data synchronization, the configuration information required by the data synchronization is acquired.
Optionally, after the obtaining, from the source database, the update data of the target data table according to the table name of the target data table, and before the storing, in the message queue corresponding to the message queue address, the update data and the table name of the target data table, the method further includes:
performing format conversion on the update data and the table name of the target data table by using a source end adapter to obtain target update data; the data format of the target updating data is the data format allowed by the message queue;
the storing the update data and the table name of the target data table in the message queue corresponding to the message queue address so that when the destination terminal monitors that the message queue has the update data, the updating data is used to update the data of the data table to be updated in the destination terminal database, including:
and storing the target updating data in a message queue corresponding to the message queue address, so that when a target end monitors that the target updating data exists in the message queue, a target end adapter is used for carrying out data format conversion on the target updating data to obtain the updating data and the table name of the target data table, and the updating data is used for updating the data of the data table to be updated in the target end database.
Optionally, the method further comprises:
and recording an operation log by using the data trigger.
The embodiment of the application also provides a data synchronization method, which is applied to a destination terminal and comprises the following steps:
monitoring a message queue;
when the table name of the target data table and the updating data of the target data table exist in the message queue, updating the data of the data table to be updated of the target end database by using the updating data; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; and the updating data of the target data table is acquired by the source terminal from the source terminal database according to the table name of the target data table.
Optionally, when the table name of the target data table and the update data of the target data table are in the message queue, updating the data of the data table to be updated of the destination database by using the update data, including:
when the message queue has target updating data, a target end adapter is used for carrying out data format conversion on the target updating data; the source end utilizes a source end adapter to perform format conversion on the table name of the target data table and the update data of the target data table to obtain the target update data;
acquiring a table name of the target data table and update data of the target data table;
and updating the data of the data table to be updated of the destination database by using the updating data.
Optionally, the obtaining the table name of the target data table and the update data of the target data table includes:
acquiring a normal table name of a target data table and normal updating data of the target data table;
the updating the data of the data table to be updated of the destination-side database by using the update data includes:
and updating the data of the data table to be updated of the destination database by using the normal updating data of the target data table.
An embodiment of the present application further provides a data synchronization apparatus, where the apparatus is applied to a source end, and the apparatus includes:
the first acquisition unit is used for acquiring configuration information required by data synchronization; the configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in a source database with updated internal data;
the second obtaining unit is used for obtaining the updating data of the target data table from the source database according to the table name of the target data table;
the storage unit is used for storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the target end monitors that the message queue contains the update data, the data of the data table to be updated in the target end database is updated by using the update data; and the data table to be updated is determined according to the table name of the target data table.
The embodiment of the present application further provides a data synchronization apparatus, where the apparatus is applied to a destination, and the apparatus includes:
the monitoring unit is used for monitoring the message queue;
the updating unit is used for updating the data of the data table to be updated of the target end database by using the updating data when the table name of the target data table and the updating data of the target data table exist in the message queue; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; and the updating data of the target data table is acquired from the source database according to the table name of the target data table.
According to the technical scheme, the method has the following beneficial effects:
in the embodiment of the application, configuration information required by data synchronization is acquired, wherein the configuration information at least comprises a message queue address and a table name of a target data table. The target data table is any data table in the source database with the updated internal data. And acquiring the update data of the target data table from the source database according to the table name of the target data table. And storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the target end monitors that the message queue contains the update data, the target end updates the data of the data table to be updated in the database of the target end by using the update data. And determining the data table to be updated according to the table name of the target data table. The source end of the embodiment of the application places the transmitted data in the message queue, and when the destination end monitors the updated data in the message queue, the updated data in the message queue can be extracted, so that the data synchronization of the destination end database is carried out based on the updated data. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of an exemplary application scenario of a data synchronization method provided in an embodiment of the present application;
fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present application;
fig. 3 is a flowchart of another data synchronization method provided in an embodiment of the present application;
fig. 4 is a flowchart of a data synchronization method according to an embodiment of the present scenario;
fig. 5 is a schematic diagram of a data synchronization apparatus according to an embodiment of the present application;
fig. 6 is a schematic diagram of a data synchronization apparatus according to an embodiment of the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, embodiments accompanying the drawings are described in detail below.
In order to facilitate understanding of the data synchronization method provided in the embodiment of the present application, an application scenario of the embodiment of the present application is described below with reference to fig. 1. Fig. 1 is a schematic diagram of an exemplary application scenario of a data synchronization method according to an embodiment of the present application.
As shown in fig. 1, an exemplary application scenario of a data synchronization method provided in an embodiment of the present application relates to a source end 1 and a destination end 2. As an example, the source peer 1 and the destination peer 2 are different traffic sides. The source end 1 corresponds to a source end database, and the destination end 2 corresponds to a destination end database. The data in the source end database and the destination end database are required to keep synchronous so as to ensure the continuity of the source end 1 service and the destination end 2 service.
When the source terminal 1 receives the service request, if the service request relates to updating data of the source terminal database. The server of the source 1 will update the data in the source database according to the content of the service request. Since the data in the source database needs to be synchronized with the data in the destination database, the source 1 needs to send the updated data to the destination 2.
In this embodiment of the present application, a source end 1 obtains configuration information required for data synchronization, where the configuration information at least includes a message queue address and a table name of a target data table. The target data table is any data table in the source database with the updated internal data. The source end 1 obtains the update data of the target data table from the source end database according to the table name of the target data table. Then, the source end 1 stores the update data and the table name of the target data table in the message queue corresponding to the message queue address.
The destination terminal 2 monitors the message queue, and when monitoring that the message queue has the update data and the table name of the target data table, the destination terminal 2 updates the data of the data table to be updated in the destination terminal database by using the update data. And the data table to be updated is determined according to the table name of the target data table. The table name of the data table to be updated corresponds to the table name of the target data table. As an example, the table name of the data table to be updated is the same as the table name of the target data table.
Those skilled in the art will appreciate that the schematic diagram shown in fig. 1 is merely one example in which embodiments of the present application may be implemented and that the scope of applicability of embodiments of the present application is not limited in any way by this framework.
It is noted that the source 101 and destination 102 may be any user equipment now existing, developing, or later developed that is capable of interacting with each other via any form of wired and/or wireless connection (e.g., Wi-Fi, LAN, cellular, coaxial cable, etc.), including but not limited to: smart wearable devices, smart phones, non-smart phones, tablets, laptop personal computers, desktop personal computers, minicomputers, midrange computers, mainframe computers, and the like, either now in existence, under development, or developed in the future. The embodiments of the present application are not limited in any way in this respect. It should also be noted that the source terminal 101 and the destination terminal 102 in the embodiment of the present application may be an example of an existing, developing or future developed device capable of performing the above operations. The embodiments of the present application are not limited in any way in this respect.
Referring to fig. 2, fig. 2 is a flowchart of a data synchronization method provided by an embodiment of the present application, where the method is applied to a source end, and as shown in fig. 2, the method includes S201-S203:
s201: acquiring configuration information required by data synchronization; the configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in the source database with the updated internal data.
The source end obtains configuration information required by data synchronization of the source end database and the target end database, and performs subsequent data synchronization based on the configuration information. The configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in the source database with the updated internal data.
It will be appreciated that the message queue address is the address at which the message queue is located. The message queue can be queried according to the message queue address. A message queue may be understood as a container that holds messages during the transmission of a message, for example, when system a writes a message into the message queue, system B may read the written message from the message queue. The message queue is deployed independently as a separate middleware product. It should be noted that the type of the message queue and the deployment position of the message queue are not limited, and may be selected and deployed according to needs.
Any data table updated in the source database is called a target data table. The data table in the database has a table name, and the data table corresponding to the table name can be determined according to the table name.
In specific implementation, the obtaining of configuration information required for data synchronization includes: after the configuration service in the configuration service management platform determines the configuration information required by data synchronization, the configuration information required by the data synchronization is acquired. The configuration information is configured for the configuration service of the configuration service management platform, and after the configuration information is determined, the configuration service is sent to the source server.
It should be noted that, before the configuration information required for data synchronization is obtained, the target data table in the database has already completed data updating. Specifically, the process of updating the data in the target data table includes:
responding to a service request of a source end, and generating a specified event applied to a target data table;
based on the specified event, executing the operation corresponding to the specified event on the target data table by using the data trigger configured by the source database so as to complete the data updating of the source database.
Wherein the specified event applied to the target data table corresponds to the service request. As an example, a business request is to change data corresponding to "employee name", the data corresponding to "employee name" is located on a target data table, and in response to the business request, data is generated specifying an event as to change the data corresponding to "employee name" in the target data table. When a specific event occurs in the target data table, the data trigger performs an operation corresponding to the specific event, for example, the data trigger changes data corresponding to "employee name" in the target data table.
In addition, after the data updating in the target data table is completed, the operation log is also recorded by using the data trigger. The operation log is a data record for some business operations. The changed content and related information of the data in the database can be clearly known according to the operation log.
S202: and acquiring the update data of the target data table from the source database according to the table name of the target data table.
And after the server of the source end acquires the configuration information, acquiring the update data of the target data table from the source end database according to the table name of the target data table in the configuration information.
Specifically, the source server determines the target data table according to the table name of the target data table, and further extracts updated data from the target data table. For example, the updated data corresponding to "employee name".
S203: storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the target end monitors that the message queue has the update data, the target end updates the data of the data table to be updated in the database of the target end by using the update data; and the data table to be updated is determined according to the table name of the target data table.
And the source server stores the update data and the table name of the target data table in a message queue corresponding to the message queue address. And the destination terminal monitors the message queue, and when the message queue is monitored to contain the updated data and the table name of the target data table, the updated data and the table name of the target data table are taken out from the message queue.
And the target end determines a data table to be updated of the target end database by using the table name of the target data table. The data table to be updated of the destination database corresponds to the target data table, and as an example, the table name of the data table to be updated is the same as the table name of the target data table, and the data table to be updated and the target data table are tables recording the same data.
Further, the destination terminal updates the data in the data table to be updated by using the update data, so that the target data table and the data in the data table to be updated are synchronized. For example, the update data is data corresponding to the updated "employee name", and the data corresponding to the "employee name" in the data table to be updated is updated by using the data.
It can be understood that, the mode of replacing the data transmission interface with the message queue improves the buffer capacity when different databases are synchronized, and improves the timeliness of data transmission to a certain extent.
After S202 and before S203, the method further includes:
performing format conversion on the update data and the table name of the target data table by using a source end adapter to obtain target update data; the data format of the target updating data is the data format allowed by the message queue.
It can be understood that the messages in the message queue must satisfy the data format of the message queue, so when the data format of the update data and the data format of the table name of the target data table are different from the data format required by the message queue, format conversion needs to be performed on the update data and the table name of the target data table by using the source adapter to obtain the target update data. The data format of the target updating data is the data format allowed by the message queue.
Thus, the method for updating the data table to be updated in the database of the destination end by using the update data when the destination end monitors that the message queue has the update data includes the steps of:
and storing the target updating data in the message queue corresponding to the message queue address so that when the target end monitors that the target updating data exists in the message queue, the target end adapter is used for carrying out data format conversion on the target updating data to obtain the updating data and the table name of the target data table, and the updating data is used for updating the data of the data table to be updated in the target end database.
It should be noted that, when the destination retrieves the target update data from the message queue, since the data format of the target update data is different from the data format of the data in the destination database, the destination adapter needs to perform data format conversion on the target update data to obtain the update data and the table name of the target data table. Further, the destination terminal updates the data of the data table to be updated in the destination terminal database by using the update data.
In the embodiment of the application, configuration information required by data synchronization is acquired, wherein the configuration information at least comprises a message queue address and a table name of a target data table. The target data table is any data table in the source database with the updated internal data. And acquiring the update data of the target data table from the source database according to the table name of the target data table. And storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the target end monitors that the message queue contains the update data, the target end updates the data of the data table to be updated in the database of the target end by using the update data. And determining the data table to be updated according to the table name of the target data table. The source end of the embodiment of the application places the transmitted data in the message queue, and when the destination end monitors the updated data in the message queue, the updated data in the message queue can be extracted, so that the data synchronization of the destination end database is carried out based on the updated data. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
Referring to fig. 3, fig. 3 is a flowchart of another data synchronization method provided in the embodiment of the present application, where the method is applied to a destination, and as shown in fig. 3, the method includes S301-S302:
s301: a message queue is monitored.
The destination monitors the message queue.
A message queue may be understood as a container for holding messages during message transmission, for example, after system a writes a message into the message queue, system B may read the written message from the message queue. The message queue is deployed independently as a separate middleware product. It should be noted that the type of the message queue and the deployment position of the message queue are not limited, and may be selected and deployed according to needs.
S302: when the table name of the target data table and the updating data of the target data table exist in the message queue, updating the data of the data table to be updated in the target database by using the updating data; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; and the source end acquires the update data of the target data table from the source end database according to the table name of the target data table.
When the target end monitors that the table name of the target data table and the updating data of the target data table exist in the message queue, the target end updates the data of the data table to be updated in the target end database by using the updating data.
And the data table to be updated is determined according to the table name of the target data table. As an example, the table name of the data table to be updated is the same as the table name of the target data table. The target data table is any data table in the source database with updated internal data, and the updated data of the target data table is obtained by the source end from the source database according to the table name of the target data table.
It should be noted that, when there are a table name of the target data table and update data of the target data table in the message queue, updating the data of the data table to be updated in the destination database by using the update data, including:
when the target updating data exists in the message queue, the data format conversion is carried out on the target updating data by using a target end adapter; the source end utilizes the source end adapter to perform format conversion on the table name of the target data table and the update data of the target data table to obtain the target update data;
acquiring a table name of a target data table and update data of the target data table;
and updating the data of the data table to be updated of the destination database by using the updating data.
It can be understood that the messages in the message queue must satisfy the data format of the message queue, and when the data format of the update data and the data format of the table name of the target data table are different from the data format required by the message queue, format conversion needs to be performed on the update data and the table name of the target data table by using the source end adapter to obtain the target update data. And the data format of the target updating data is the data format allowed by the message queue.
When the target end takes out the target update data from the message queue, because the data format of the target update data is different from the data format of the data in the target end database, the target end adapter needs to be used for carrying out data format conversion on the target update data, and the update data and the table name of the target data table are obtained. Further, the destination terminal updates the data of the data table to be updated in the destination terminal database by using the update data.
In specific implementation, the obtaining of the table name of the target data table and the update data of the target data table includes:
acquiring a normal table name of a target data table and normal updating data of the target data table;
updating the data of the data table to be updated of the destination database by using the update data, including:
and updating the data of the data table to be updated of the destination database by using the normal update data of the target data table.
When the destination converts the data format of the target update data by using the destination adapter, there is a possibility that an abnormal situation such as a conversion error occurs and abnormal data is generated. The abnormal data comprises an abnormal table name of the target data table and/or abnormal updating data of the target data table. And subsequently storing the abnormal data in an abnormal database, and analyzing and further processing the abnormal data. Accordingly, the normal data includes the normal table name of the target data table and the normal update data of the corresponding target data table. And the target updates the data of the data table to be updated of the target database by using the normal updating data of the target data table. At this time, the data table to be updated is obtained according to the normal table name of the target data table.
In the embodiment of the application, the destination monitors the message queue. When the target end monitors that the table name of the target data table and the updating data of the target data table exist in the message queue, the data of the data table to be updated in the database of the target end are updated by the updating data; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; the update data of the target data table is that the source end of the embodiment of the application, which is acquired from the source end database according to the table name of the target data table, places the transmitted data in the message queue, and when the target end monitors the update data in the message queue, the update data in the message queue can be extracted, so that the data synchronization of the target end database is performed based on the update data. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
Referring to fig. 4, fig. 4 is a flowchart of a data synchronization method provided in the present scenario embodiment. As shown in fig. 4, in response to a service request of the source, the source server generates a specified event applied to data table 1, data table 2 and data table 3, respectively. It is understood that the target data table in the above embodiments may be data table 1, data table 2 or data table 3.
Based on the specified event on the data table 1, the data trigger in the data table 1 performs an operation corresponding to the specified event on the data table 1 to complete the data update of the data table 1. The data update process of data table 2 and data table 3 is similar to that of data table 1. In addition, the operation log can be recorded by using the data trigger.
The configuration service in the configuration service management platform generates configuration information required by data which needs to be synchronized between the source end database and the destination end database, and sends the configuration information to the server of the source end. The configuration information includes at least the message queue address, and the table names of data table 1, data table 2, and data table 3.
After the source server obtains the configuration information required by data synchronization, the source server obtains the updated data in the data table 1, the data table 2 and the data table 3 from the source database according to the table names of the data table 1, the data table 2 and the data table 3.
The source end adapter performs format conversion on the update data in the data table 1, the data table 2 and the data table 3 and the table names corresponding to the data table 1, the data table 2 and the data table 3 to obtain target update data. The data format of the target updating data is the data format allowed by the message queue.
And storing the target updating data in the message queue corresponding to the message queue address. The destination monitors the message queue. And when the destination end monitors that the target updating data exist in the message queue, the destination end adapter is used for carrying out data format conversion on the target updating data. And analyzing the converted data, and if the corresponding table names and the updated data of the data table 1, the data table 2 and the data table 3 are normal, updating the data in the corresponding data table 1, the data table 2 and the data table 3 in the destination end database by using the normal updated data in the data table 1, the data table 2 and the data table 3.
And if the table names and/or the updated data corresponding to the data table 1, the data table 2 and the data table 3 are abnormal, acquiring abnormal data, wherein the abnormal data comprises the abnormal table names and/or the abnormal updated data. And subsequently storing the abnormal data in an abnormal database, and analyzing and further processing the abnormal data.
The source end of the embodiment of the application places the transmitted data in the message queue, and when the destination end monitors the updated data in the message queue, the updated data in the message queue can be extracted, so that the data synchronization of the destination end database is carried out based on the updated data. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
Referring to fig. 5, fig. 5 is a schematic diagram of a data synchronization apparatus according to an embodiment of the present application. The device is applied to the source end, the device includes:
a first obtaining unit 501, configured to obtain configuration information required for data synchronization; the configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in a source database with updated internal data;
a second obtaining unit 502, configured to obtain, from the source database, update data of the target data table according to the table name of the target data table;
a storage unit 503, configured to store the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the destination monitors that the message queue has the update data, the destination updates the data of the data table to be updated in the destination database with the update data; and the data table to be updated is determined according to the table name of the target data table.
Optionally, in some implementations of embodiments of the present application, the apparatus further includes:
a generating unit, configured to generate a specified event applied to a target data table in response to a service request of the source end before the configuration information required for data synchronization is acquired;
and the execution unit is used for executing the operation corresponding to the specified event on the target data table by using the data trigger configured by the source database based on the specified event so as to complete the data updating of the source database.
Optionally, in some implementations of the embodiments of the present application, the first obtaining unit 501 is specifically configured to:
after the configuration service in the configuration service management platform determines the configuration information required by data synchronization, the configuration information required by the data synchronization is acquired.
Optionally, in some implementations of embodiments of the present application, the apparatus further includes:
a format conversion unit, configured to, after obtaining the update data of the target data table from the source database according to the table name of the target data table, perform format conversion on the update data and the table name of the target data table by using a source adapter before storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, and obtain target update data; the data format of the target updating data is the data format allowed by the message queue;
the storage unit 503 is specifically configured to:
and storing the target updating data in a message queue corresponding to the message queue address, so that when a target end monitors that the target updating data exists in the message queue, a target end adapter is used for carrying out data format conversion on the target updating data to obtain the updating data and the table name of the target data table, and the updating data is used for updating the data of the data table to be updated in the target end database.
Optionally, in some implementations of embodiments of the present application, the apparatus further includes:
and the recording unit is used for recording the operation log by utilizing the data trigger.
The source end of the embodiment of the application places the transmitted data in the message queue, and when the destination end monitors the updated data in the message queue, the updated data in the message queue can be extracted, so that the data synchronization of the destination end database is carried out based on the updated data. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
Referring to fig. 6, fig. 6 is a schematic diagram of a data synchronization apparatus according to an embodiment of the present application. The device is applied to a destination, the device comprising:
a monitoring unit 601, configured to monitor a message queue;
an updating unit 602, configured to update, when a table name of a target data table and update data of the target data table exist in the message queue, data of a data table to be updated of the target database by using the update data; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; and the updating data of the target data table is acquired from the source database according to the table name of the target data table.
Optionally, in some implementations of embodiments of the present application, the updating unit 602 includes:
the format conversion subunit is used for performing data format conversion on the target update data by using a destination end adapter when the target update data exists in the message queue; the target updating data is obtained by the source end by using a source end adapter to perform format conversion on the table name of the target data table and the updating data of the target data table;
the acquisition subunit is used for acquiring the table name of a target data table and the update data of the target data table;
and the updating subunit is used for updating the data of the data table to be updated of the destination database by using the updating data.
Optionally, in some implementations of the embodiments of the present application, the obtaining subunit is specifically configured to:
acquiring a normal table name of a target data table and normal updating data of the target data table;
the update subunit is specifically configured to:
and updating the data of the data table to be updated of the destination database by using the normal updating data of the target data table.
The source end of the embodiment of the application places the transmitted data in the message queue, and when the destination end monitors the updated data in the message queue, the updated data in the message queue can be extracted, so that the data synchronization of the destination end database is carried out based on the updated data. According to the embodiment of the application, the message queue replaces the existing database data transmission interface, the buffering capacity of different database data in synchronization is improved, and the timeliness of data transmission is improved to a certain extent.
As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present application may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the method according to the embodiments or some parts of the embodiments of the present application.
It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. The method disclosed by the embodiment corresponds to the system disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the system part for description.
It should also be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data synchronization method is applied to a source end, and the method comprises the following steps:
acquiring configuration information required by data synchronization; the configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in a source database with updated internal data;
acquiring the update data of the target data table from the source database according to the table name of the target data table;
storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when a target end monitors that the message queue contains the update data, the target end updates the data of the data table to be updated in the target end database by using the update data; and the data table to be updated is determined according to the table name of the target data table.
2. The method of claim 1, wherein before the obtaining the configuration information required for data synchronization, the method further comprises:
responding to the service request of the source end, and generating a specified event applied to a target data table;
and based on the specified event, executing an operation corresponding to the specified event on the target data table by using a data trigger configured by the source database so as to complete data updating of the source database.
3. The method according to claim 1, wherein the obtaining of the configuration information required for data synchronization comprises:
after the configuration service in the configuration service management platform determines the configuration information required by data synchronization, the configuration information required by the data synchronization is acquired.
4. The method of claim 1, wherein after the obtaining of the update data of the target data table from the source database according to the table name of the target data table, before the storing of the update data and the table name of the target data table in the message queue corresponding to the message queue address, the method further comprises:
performing format conversion on the update data and the table name of the target data table by using a source end adapter to obtain target update data; the data format of the target updating data is the data format allowed by the message queue;
the storing the update data and the table name of the target data table in the message queue corresponding to the message queue address so that when the destination terminal monitors that the message queue has the update data, the updating data is used to update the data of the data table to be updated in the destination terminal database, including:
and storing the target updating data in a message queue corresponding to the message queue address, so that when a target end monitors that the target updating data exists in the message queue, a target end adapter is used for carrying out data format conversion on the target updating data to obtain the updating data and the table name of the target data table, and the updating data is used for updating the data of the data table to be updated in the target end database.
5. The method of claim 2, further comprising:
and recording an operation log by using the data trigger.
6. A data synchronization method, wherein the method is applied to a destination, and the method comprises:
monitoring a message queue;
when the table name of a target data table and the updating data of the target data table exist in the message queue, updating the data of the data table to be updated of the target database by using the updating data; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; and the updating data of the target data table is acquired by the source terminal from the source terminal database according to the table name of the target data table.
7. The method as claimed in claim 6, wherein when there is a table name of a target data table and update data of the target data table in the message queue, updating data of a data table to be updated of the destination database with the update data, includes:
when the message queue has target updating data, a target end adapter is used for carrying out data format conversion on the target updating data; the source end utilizes a source end adapter to perform format conversion on the table name of the target data table and the update data of the target data table to obtain the target update data;
acquiring a table name of the target data table and update data of the target data table;
and updating the data of the data table to be updated of the destination database by using the updating data.
8. The method of claim 7, wherein the obtaining of the table name of the target data table and the update data of the target data table comprises:
acquiring a normal table name of a target data table and normal updating data of the target data table;
the updating the data of the data table to be updated of the destination-side database by using the update data includes:
and updating the data of the data table to be updated of the destination database by using the normal updating data of the target data table.
9. A data synchronization apparatus, wherein the apparatus is applied to a source end, the apparatus comprising:
the first acquisition unit is used for acquiring configuration information required by data synchronization; the configuration information at least comprises a message queue address and a table name of a target data table; the target data table is any data table in a source database with updated internal data;
the second obtaining unit is used for obtaining the updating data of the target data table from the source database according to the table name of the target data table;
the storage unit is used for storing the update data and the table name of the target data table in a message queue corresponding to the message queue address, so that when the target end monitors that the message queue contains the update data, the data of the data table to be updated in the target end database is updated by using the update data; and the data table to be updated is determined according to the table name of the target data table.
10. A data synchronization apparatus, wherein the apparatus is applied to a destination, the apparatus comprising:
the monitoring unit is used for monitoring the message queue;
the updating unit is used for updating the data of the data table to be updated of the target end database by using the updating data when the table name of the target data table and the updating data of the target data table exist in the message queue; the data table to be updated is determined according to the table name of the target data table; the target data table is any data table in a source database with updated internal data; and the updating data of the target data table is acquired from the source database according to the table name of the target data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110742129.XA CN113434600B (en) | 2021-06-30 | 2021-06-30 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110742129.XA CN113434600B (en) | 2021-06-30 | 2021-06-30 | Data synchronization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113434600A true CN113434600A (en) | 2021-09-24 |
| CN113434600B CN113434600B (en) | 2023-06-09 |
Family
ID=77758420
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110742129.XA Active CN113434600B (en) | 2021-06-30 | 2021-06-30 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113434600B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590719A (en) * | 2021-09-27 | 2021-11-02 | 北京奇虎科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN115033647A (en) * | 2022-08-11 | 2022-09-09 | 杭州湖畔网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN115982285A (en) * | 2023-03-10 | 2023-04-18 | 北京集度科技有限公司 | Data processing method, device and computer readable storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6073177A (en) * | 1997-08-05 | 2000-06-06 | Sterling Software, Inc. | Dynamic method for connecting a client to a server application |
| CN101860470A (en) * | 2009-04-13 | 2010-10-13 | 中国联合网络通信集团有限公司 | File synchronous processing method, synchronous gateway and file synchronizing system |
| US9804935B1 (en) * | 2015-01-26 | 2017-10-31 | Intel Corporation | Methods for repairing a corrupted database to a new, correct state by selectively using redo and undo operations |
| CN107332886A (en) * | 2017-06-20 | 2017-11-07 | 福建中金在线信息科技有限公司 | Method of data synchronization, device, system, electronic equipment and readable storage medium storing program for executing |
| CN107948316A (en) * | 2017-12-25 | 2018-04-20 | 北京搜狐新媒体信息技术有限公司 | A kind of file synchronisation method, device and equipment |
| CN109325077A (en) * | 2018-08-03 | 2019-02-12 | 北京马上慧科技术有限公司 | A kind of system that number storehouse in real time is realized based on canal and kafka |
| CN109582731A (en) * | 2018-10-18 | 2019-04-05 | 恒峰信息技术有限公司 | A kind of real time data synchronization method and system |
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
| CN110597914A (en) * | 2019-09-18 | 2019-12-20 | 北京思维造物信息科技股份有限公司 | Data transmission system, method, device and equipment |
| CN111240828A (en) * | 2019-12-30 | 2020-06-05 | 苏州大学 | Synchronous communication system and method of embedded real-time operating system |
| CN111262923A (en) * | 2020-01-14 | 2020-06-09 | 深圳壹账通智能科技有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN111752910A (en) * | 2020-06-24 | 2020-10-09 | 上海微盟企业发展有限公司 | Data synchronization method, system and related device for heterogeneous platform |
| CN112307037A (en) * | 2019-07-26 | 2021-02-02 | 北京京东振世信息技术有限公司 | Data synchronization method and device |
| CN112925656A (en) * | 2021-01-11 | 2021-06-08 | 华南理工大学 | Multi-protocol automation equipment monitoring system based on message queue and control method thereof |
-
2021
- 2021-06-30 CN CN202110742129.XA patent/CN113434600B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6073177A (en) * | 1997-08-05 | 2000-06-06 | Sterling Software, Inc. | Dynamic method for connecting a client to a server application |
| CN101860470A (en) * | 2009-04-13 | 2010-10-13 | 中国联合网络通信集团有限公司 | File synchronous processing method, synchronous gateway and file synchronizing system |
| US9804935B1 (en) * | 2015-01-26 | 2017-10-31 | Intel Corporation | Methods for repairing a corrupted database to a new, correct state by selectively using redo and undo operations |
| CN107332886A (en) * | 2017-06-20 | 2017-11-07 | 福建中金在线信息科技有限公司 | Method of data synchronization, device, system, electronic equipment and readable storage medium storing program for executing |
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
| CN107948316A (en) * | 2017-12-25 | 2018-04-20 | 北京搜狐新媒体信息技术有限公司 | A kind of file synchronisation method, device and equipment |
| CN109325077A (en) * | 2018-08-03 | 2019-02-12 | 北京马上慧科技术有限公司 | A kind of system that number storehouse in real time is realized based on canal and kafka |
| CN109582731A (en) * | 2018-10-18 | 2019-04-05 | 恒峰信息技术有限公司 | A kind of real time data synchronization method and system |
| CN112307037A (en) * | 2019-07-26 | 2021-02-02 | 北京京东振世信息技术有限公司 | Data synchronization method and device |
| CN110597914A (en) * | 2019-09-18 | 2019-12-20 | 北京思维造物信息科技股份有限公司 | Data transmission system, method, device and equipment |
| CN111240828A (en) * | 2019-12-30 | 2020-06-05 | 苏州大学 | Synchronous communication system and method of embedded real-time operating system |
| CN111262923A (en) * | 2020-01-14 | 2020-06-09 | 深圳壹账通智能科技有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN111752910A (en) * | 2020-06-24 | 2020-10-09 | 上海微盟企业发展有限公司 | Data synchronization method, system and related device for heterogeneous platform |
| CN112925656A (en) * | 2021-01-11 | 2021-06-08 | 华南理工大学 | Multi-protocol automation equipment monitoring system based on message queue and control method thereof |
Non-Patent Citations (2)
| Title |
|---|
| WU JIE-MING 等: ""Research and Design of Smart Client Data Synchronization Engine"", 《PROCEEDINGS OF THE 2009 INTERNATIONAL WORKSHOP ON INFORMATION SECURITY AND APPLICATION》 * |
| 徐梓荐 等: ""分布式异构数据库数据同步工具"", 《软件学报》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590719A (en) * | 2021-09-27 | 2021-11-02 | 北京奇虎科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113590719B (en) * | 2021-09-27 | 2022-03-22 | 北京奇虎科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN115033647A (en) * | 2022-08-11 | 2022-09-09 | 杭州湖畔网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN115982285A (en) * | 2023-03-10 | 2023-04-18 | 北京集度科技有限公司 | Data processing method, device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113434600B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113434600B (en) | Data synchronization method and device | |
| JP6745884B2 (en) | Data synchronization method, device and system | |
| US8874091B2 (en) | Automatic device capabilities change notification | |
| CN105787077B (en) | Data synchronization method and device | |
| US20120323846A1 (en) | Sending Category Information | |
| KR101685215B1 (en) | Automatic discovery of alternate mailboxes | |
| US20150256504A1 (en) | Distributed synchronization data in a message management service | |
| JP2012516584A (en) | Method and apparatus for tracking management data changes | |
| CN102255866A (en) | Method and device for downloading data | |
| US20140013007A1 (en) | Access log management method | |
| CN111338834B (en) | Data storage method and device | |
| CN112328413A (en) | Application service calling method, device and system | |
| CN111241189A (en) | Method and device for synchronizing data | |
| CN110677443A (en) | Data transmitting and receiving method, transmitting end, receiving end, system and storage medium | |
| JP4616114B2 (en) | Mobile communication terminal device, mobile communication terminal program, server device, server program, and main mobile communication terminal device | |
| CN112925848B (en) | Data synchronization method, device, equipment and readable storage medium | |
| CN113220645B (en) | Linux-compatible Android file display method and device | |
| CN111106966B (en) | Information processing method and device, equipment and storage medium | |
| US20120023074A1 (en) | Scalable rule-based data synchronization systems and methods | |
| CN111935316A (en) | Method and device for acquiring front-end equipment catalog | |
| CN111241194A (en) | Data synchronization method and device | |
| EP1720285B1 (en) | Apparatus and method for processing messages in a network management system | |
| CN111787043A (en) | Data request method and device | |
| CN111314457B (en) | Method and device for setting virtual private cloud | |
| CN108337151A (en) | Realize that the more equipment of user message synchronize the method and system roamed in instant messaging |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |