CN115221242A - Data synchronization method and device and computer readable storage medium - Google Patents
Data synchronization method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN115221242A CN115221242A CN202210763888.9A CN202210763888A CN115221242A CN 115221242 A CN115221242 A CN 115221242A CN 202210763888 A CN202210763888 A CN 202210763888A CN 115221242 A CN115221242 A CN 115221242A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time change
- data synchronization
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The application discloses a data synchronization method, a data synchronization device and a computer readable storage medium, and relates to server technology. The data synchronization method provided by the application stores real-time change data in a message queue by acquiring the real-time change data in a source database; and acquiring real-time change data in the message queue, and writing the real-time change data into a target database. Therefore, the scheme realizes asynchronous processing of a data acquisition process and a data writing process in the data synchronization process of different databases by storing data in the message queue, and the data acquisition process and the data writing process do not interfere with each other. Therefore, data synchronization of a plurality of different databases can be carried out, the horizontal extension of the data synchronization is realized, the data synchronization among heterogeneous databases can be supported, and the synchronization efficiency is higher.
Description
Technical Field
The present application relates to the field of server technologies, and in particular, to a data synchronization method and apparatus, and a computer-readable storage medium.
Background
With the splitting of the business system, different databases are used by different business systems when processing the business. This difference involves two scenarios, the first is to use the same database, but the databases are not on the same server. The second is that different types of databases are used, for example service a uses MySQL database, but service B uses Oracle or PostgreSQL database. When data interaction exists between different services, API call can be used for realizing data interaction between the services, or data can be directly synchronized between databases in real time for realizing data transmission between different business systems.
However, the data synchronization schemes of the current databases, such as MySQL master-slave replication, postgreSQL stream replication, and Oracle, adopt the OGG tool to realize real-time synchronization. The synchronization tools directly realize data synchronization of the same database by using a bottom API of the database, and cannot support data synchronization between heterogeneous databases because the master-slave limitation cannot realize horizontal expansion.
In view of the above problems, it is an urgent need to solve the problems in the art to design a data synchronization method.
Disclosure of Invention
The application aims to provide a data synchronization method, a data synchronization device and a computer readable storage medium, which are used for realizing horizontal extension in a database data synchronization process and supporting data synchronization among heterogeneous databases.
In order to solve the above technical problem, the present application provides a data synchronization method, including:
acquiring real-time change data in a source database;
storing the real-time change data in a message queue;
acquiring the real-time change data in the message queue;
and writing the real-time change data into a destination database.
Preferably, said storing said real-time change data in a message queue comprises:
acquiring a first configuration center; the first configuration center is used for configuring the corresponding relation between table data in the real-time change data and topic of the message queue;
filtering the table data according to the first configuration center;
and writing the filtered table data into the topic of the corresponding message queue according to the first configuration center.
Preferably, the writing the real-time change data into the destination database includes:
acquiring a second configuration center; the second configuration center is configured to configure a correspondence between the topic of the message queue and a destination table required by the destination database;
filtering the table data according to the second configuration center;
and writing the filtered table data into the target table corresponding to the target database according to the second configuration center.
Preferably, the method further comprises the following steps:
and docking the real-time change data in the message queue into the Flink for processing the real-time change data through the Flink.
Preferably, before the acquiring the real-time change data in the source database, the method further includes:
monitoring a data synchronization request sent by the target database;
judging whether the data synchronization request has a data synchronization authority or not;
and if so, entering the step of acquiring real-time change data in the source database.
Preferably, after the writing the real-time change data into the destination database, the method further includes:
and generating prompt information for completing the synchronization of the real-time change data.
Preferably, if the data synchronization request does not have the data synchronization right, the method further includes:
and outputting the information of data synchronization failure.
In order to solve the above technical problem, the present application further provides a data synchronization apparatus, including:
the first acquisition module is used for acquiring real-time change data in the source database;
the storage module is used for storing the real-time change data in a message queue;
the second obtaining module is used for obtaining the real-time change data in the message queue;
and the writing module is used for writing the real-time change data into a target database.
Preferably, the storage module includes:
the third acquisition module is used for acquiring the first configuration center; the first configuration center is used for configuring the corresponding relation between table data in the real-time change data and topic of the message queue;
the first data filtering module is used for filtering the table data according to the first configuration center;
and the first table data writing module is used for writing the filtered table data into the topic of the corresponding message queue according to the first configuration center.
Preferably, the writing module includes:
the fourth acquisition module is used for acquiring a second configuration center; the second configuration center is configured to configure a correspondence between the topic of the message queue and a destination table required by the destination database;
the second data filtering module is used for filtering the table data according to the second configuration center;
and the second table data writing module is used for writing the filtered table data into the target table corresponding to the target database according to the second configuration center.
Preferably, the method further comprises the following steps:
and the data processing module is used for butting the real-time change data in the message queue into the Flink so as to process the real-time change data through the Flink.
Preferably, the method further comprises the following steps:
the monitoring module is used for monitoring a data synchronization request sent by the target database;
the judging module is used for judging whether the data synchronization request has data synchronization authority; if yes, the first acquisition module is triggered.
Preferably, the method further comprises the following steps:
and the generating module is used for generating prompt information for completing the synchronization of the real-time change data.
Preferably, the method further comprises the following steps:
and the output module is used for outputting the information of data synchronization failure when the data synchronization request does not have the data synchronization right.
In order to solve the above technical problem, the present application further provides a data synchronization apparatus, including:
a memory for storing a computer program;
a processor for implementing the steps of the data synchronization method when executing the computer program.
In order to solve the above technical problem, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the data synchronization method described above.
The data synchronization method provided by the application stores real-time change data in a message queue by acquiring the real-time change data in a source database; and acquiring real-time change data in the message queue, and writing the real-time change data into a target database. Therefore, the scheme realizes asynchronous processing of a data acquisition process and a data writing process in the data synchronization process of different databases by storing data in the message queue, and the data acquisition process and the data writing process do not interfere with each other. Therefore, data synchronization of a plurality of different databases can be carried out, the horizontal extension of the data synchronization is realized, the data synchronization among heterogeneous databases can be supported, and the synchronization efficiency is higher.
In addition, the embodiment of the application also provides a data synchronization device and a computer readable storage medium, and the effects are the same as above.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings needed for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present application;
fig. 2 is a flowchart of another data synchronization method provided in an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of another data synchronization apparatus according to an embodiment of the present application;
fig. 5 is a flowchart of a data synchronization method provided in an application scenario according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
The core of the application is to provide a data synchronization method, a data synchronization device and a computer readable storage medium.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings.
Currently, partial synchronization schemes exist between databases of the same kind, such as MySQL master-slave replication, postgreSQL stream replication, and Oracle uses an OGG tool to implement real-time synchronization. The synchronization tools directly realize data synchronization of the same database by using the bottom API of the database, and cannot realize horizontal extension due to the limitation of master and slave. Therefore, the embodiment of the present application provides a data synchronization method, which extracts data from a source database in real time and then writes the data into a destination database, supports one-to-many and supports synchronization between heterogeneous databases. Fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present application. As shown in fig. 1, the method comprises:
s10: and acquiring real-time change data in the source database.
S11: real-time change data is stored in a message queue.
S12: and acquiring real-time change data in the message queue.
S13: and writing the real-time change data into the destination database.
It is understood that during the data synchronization process of the databases, the source database and the destination database are two databases performing data synchronization. Some data in the source database needs to be synchronized to the destination database. The source database and the destination database may exist on the same server, or may exist on different servers, which is determined according to specific implementation situations and is not limited in this embodiment. In order to synchronize data, real-time changed data in a source database needs to be acquired first. As a preferred embodiment, the data acquisition process may be implemented by a data extraction device; because logs are generated when data in the database are changed, and the types of the logs generated by different databases are different, the data extraction modes of different types of source databases are also different. The specific manner of data capture is not limited in this embodiment, and depends on the specific implementation. In addition, in order to improve the robustness of the whole data synchronization process, as a preferred embodiment, the data extraction device can be constructed by adopting a master-slave structure.
Further, after the real-time change data of the source database is acquired, the real-time change data is stored in a message queue. A message queue is a container for holding messages during their transmission, and can be understood as a data storage device during database data synchronization. As a preferred embodiment, the message queue may be implemented by Kafka. Kafka is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java languages. Being a high throughput distributed publish-subscribe messaging system, it can handle all the action flow data of consumers in a website. This action (web browsing, searching and other user actions) is a key factor in many social functions on modern networks. These data are typically addressed by handling logs and log aggregations due to throughput requirements. This is a viable solution to the limitations of Hadoop-like log data and offline analysis systems, but which require real-time processing. The purpose of Kafka is to unify online and offline message processing through the parallel loading mechanism of Hadoop, and also to provide real-time messages through clustering.
Further, each message issued to the Kafka cluster has a category, which is referred to as topic. For table data in real-time change data obtained from a source database, the table data can be stored into different topics in Kafka, so that the corresponding storage of the data in a message queue is realized. In this embodiment, there is no limitation on the specific storage process of the real-time change data in the message queue, which is determined according to the specific implementation situation.
After the real-time change data is stored in the message queue, in order to achieve data synchronization between the source database and the destination database, the real-time change data in the message queue needs to be further acquired by the data output device, and written into the destination database. It should be noted that, since the data written into the destination database at this time comes from the message queue, data output asynchronous to the data extraction process, that is, data output into the destination database, is realized. In the whole process, the data extraction process only concerns the accuracy of the data written into the data storage device, the data output process only concerns the accuracy of the data output to the target database, and the data extraction process and the data output process are not interfered with each other. Therefore, in the data synchronization process of the databases, the data extraction process corresponding to one data extraction device can correspond to the data output processes of a plurality of data output devices, that is, the data in one source database can be synchronized to a plurality of target databases at the same time, and the horizontal extension of the data synchronization of the databases is realized. In this embodiment, the specific process of writing the real-time change data into the destination database is not limited, and is determined according to the specific implementation situation.
In this embodiment, the real-time change data in the source database is acquired and stored in the message queue; and acquiring real-time change data in the message queue, and writing the real-time change data into a target database. Therefore, the scheme realizes asynchronous processing of a data acquisition process and a data writing process in the data synchronization process of different databases by storing data in the message queue, and the data acquisition process and the data writing process do not interfere with each other. Therefore, data synchronization of a plurality of different databases can be carried out, the horizontal extension of the data synchronization is realized, the data synchronization among heterogeneous databases can be supported, and the synchronization efficiency is higher.
On the basis of the above-described embodiment:
as a preferred embodiment, storing the real-time change data in a message queue comprises:
acquiring a first configuration center; the first configuration center is used for configuring the corresponding relation between table data in the real-time change data and topic of the message queue;
filtering the table data according to the first configuration center;
and writing the filtered table data into the topic of the corresponding message queue according to the first configuration center.
In the above embodiments, the specific process of writing the real-time change data into the destination database is not limited, and is determined according to the specific implementation. As a preferred embodiment, a first configuration center is first acquired.
It is understood that, since the real-time change data in the source database is obtained to obtain the table data, there are usually many tables in the data synchronization process. In order to clarify that each table data is stored in a specific position of the message queue, the corresponding relationship between the table data in the real-time change data and the topic of the message queue needs to be configured through the first configuration center, so that the table data is stored in the message queue.
Furthermore, since the extracted real-time change data is not necessarily required by the destination database in the data synchronization process, the table data needs to be filtered according to the first configuration center, so as to filter out the required table data. And then writing the filtered table data into the topic of the corresponding message queue according to the first configuration center, and finally realizing the data storage of the message queue.
In the embodiment, table data is filtered according to a first configuration center by obtaining the first configuration center for configuring the corresponding relation between the table data in the real-time change data and the topic of the message queue; and writing the filtered table data into the topic of the corresponding message queue according to the first configuration center. The table data is filtered and stored, so that the table data can be acquired from the message queue subsequently.
On the basis of the above-described embodiment:
as a preferred embodiment, writing the real-time change data to the destination database includes:
acquiring a second configuration center; the second configuration center is used for configuring a corresponding relation between topic of the message queue and a target table required by a target database;
filtering the table data according to a second configuration center;
and writing the filtered table data into a target table corresponding to the target database according to the second configuration center.
In the above embodiments, the specific process of writing the real-time change data into the destination database is not limited, and is determined according to the specific implementation. As a preferred embodiment, a second configuration center is first acquired.
Since some of the data in the message queue is not needed by the destination database in synchronizing the data to the destination database, the data needs to be filtered. Meanwhile, in order to determine the position of the data synchronization in the destination database, a second configuration center for acquiring the correspondence between the topic for configuring the message queue and the destination table required by the destination database is required.
Specifically, in the process of writing the real-time change data into the destination database, the table data of the real-time change data in the message queue is read first, and the table data is filtered according to the table configuration in the second configuration center. The filtered table data are divided into different lists, and then the data of the different lists are written into different tables of the target database, so that the data synchronization of the target database is finally realized.
In this embodiment, a second configuration center for obtaining a correspondence between topic for configuring a message queue and a destination table required by a destination database is obtained, and table data is filtered according to the second configuration center; and writing the filtered table data into a target table corresponding to the target database according to the second configuration center. And the specified table data is read and the table fields are filtered through the table configuration of the second configuration center, so that the data is reassembled, the data can be adapted to the write-in requests of different purpose databases, and the data synchronization of heterogeneous databases is realized.
On the basis of the above-described embodiment:
as a preferred embodiment, further comprising:
and interfacing the real-time change data in the message queue into the Flink for processing the real-time change data through the Flink.
It will be appreciated that during data synchronization of the database, real-time change data is stored in the message list. As a preferred embodiment, the real-time change data in the message queue is interfaced into the Flink for processing the real-time change data through the Flink.
The Flink, apache Flink, is a framework and distributed processing engine for performing stateful computations on unbounded and bounded data streams. Flink can operate in all common clustered environments and can be calculated at memory speed and at any scale. The Flink has strong functions and supports the development and operation of various different kinds of application programs. Its main characteristics include: batch-flow integration, precise state management, event time support, precise one-time state consistency guarantee and the like. The Flink not only can run on a variety of resource management frameworks including YARN, messes, kubernets, but also supports independent deployment on bare metal clusters. With the highly available option enabled, it does not have a single point of failure problem. It has been demonstrated that Flink can already be extended to thousands of cores, with states that can reach the TB level, and still maintain the high throughput, low latency characteristics. Many demanding stream processing applications run on Flink around the world. Therefore, in this embodiment, the processing of the real-time change data in the message queue by the Flink has a more significant advantage.
In this embodiment, the extracted real-time change data is written into the message queue and is butted to the Flink for calculation, so that the processing capability of the stream data is greatly improved.
Fig. 2 is a flowchart of another data synchronization method according to an embodiment of the present application. In order to ensure that the data synchronization between different databases meets the requirement, as shown in fig. 2, before acquiring the real-time changed data in the source database, the method further includes:
s14: and monitoring a data synchronization request sent by the destination database.
S15: judging whether the data synchronization request has a data synchronization authority or not; if yes, the process proceeds to step S10.
It is understood that, during the operation of the server, a request for continuously listening to the data synchronization sent by the destination database may be set, so as to determine whether the destination database needs to perform data synchronization. When a data synchronization request sent by a destination database is received, judging whether the data synchronization request has data synchronization authority or not so as to determine whether the destination database has data synchronization qualification or not. And when the data synchronization request is confirmed to have the data synchronization authority, acquiring real-time changed data in the source database, and performing a subsequent data synchronization process. It should be noted that, in this embodiment, the data synchronization authority is not limited, and is determined according to a specific implementation situation. The determination of whether the destination database is qualified for data synchronization is realized through the steps.
As shown in fig. 2, in order to prompt the completion of the data synchronization of the database, after the real-time change data is written into the destination database, the method further includes:
s16: and generating prompt information for completing the synchronization of the real-time change data.
Further, in order to prompt the completion of the data synchronization of the database, after the real-time change data is written into the destination database, the server generates a prompt message indicating the completion of the data synchronization of the database, so as to prompt the completion of the data synchronization of the database, and subsequent related operations can be performed. It should be noted that the prompt information may be a text prompt information, an audio prompt information, or other types of prompt information, which is determined according to a specific implementation and is not limited in this embodiment.
As shown in fig. 2, if the data synchronization request does not have the data synchronization right, the method further includes:
s17: and outputting the information of data synchronization failure.
It can be understood that, in the process of determining whether the destination database is qualified for data synchronization, when the data synchronization request does not have the data synchronization permission, that is, the destination database cannot perform data synchronization, a relevant adjustment needs to be performed on the database in order to prompt that the data synchronization of the user is abnormal. It should be noted that, like the prompt information for completing the real-time data synchronization, the prompt information for failing to synchronize the data may also be a text prompt information, an audio prompt information, or other types of prompt information, which is determined according to the specific implementation and is not limited in this embodiment.
In the foregoing embodiments, detailed descriptions are given to the data synchronization method, and the present application also provides embodiments corresponding to the data synchronization apparatus. It should be noted that the present application describes the embodiments of the apparatus portion from two perspectives, one is based on the functional module, and the other is based on the hardware structure.
Fig. 3 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application. As shown in fig. 3, the data synchronization apparatus includes:
the first obtaining module 10 is configured to obtain real-time change data in a source database.
And the storage module 11 is used for storing the real-time change data in the message queue.
And a second obtaining module 12, configured to obtain real-time change data in the message queue.
And a writing module 13, configured to write the real-time change data into a destination database.
As a preferred embodiment, the memory module includes:
the third acquisition module is used for acquiring the first configuration center; the first configuration center is used for configuring the corresponding relation between the table data in the real-time change data and the topic of the message queue;
the first data filtering module is used for filtering the table data according to the first configuration center;
and the first table data writing module is used for writing the filtered table data into the topic of the corresponding message queue according to the first configuration center.
As a preferred embodiment, the writing module includes:
the fourth acquisition module is used for acquiring a second configuration center; the second configuration center is used for configuring a corresponding relation between topic of the message queue and a target table required by a target database;
the second data filtering module is used for filtering the table data according to a second configuration center;
and the second table data writing module is used for writing the filtered table data into a target table corresponding to the target database according to the second configuration center.
As a preferred embodiment, the method further comprises the following steps:
and the data processing module is used for butting the real-time change data in the message queue to the Flink so as to process the real-time change data through the Flink.
As a preferred embodiment, the method further comprises the following steps:
the monitoring module is used for monitoring a data synchronization request sent by a target database;
the judging module is used for judging whether the data synchronization request has the data synchronization authority; if yes, the first acquisition module is triggered.
As a preferred embodiment, the method further comprises the following steps:
and the generating module is used for generating prompt information for completing the synchronization of the real-time change data.
As a preferred embodiment, further comprising:
and the output module is used for outputting the information of data synchronization failure when the data synchronization request does not have the data synchronization right.
In this embodiment, the data synchronization apparatus includes a first obtaining module, a storage module, a second obtaining module, and a writing module. The real-time change data in the source database is acquired and stored in a message queue; and acquiring real-time change data in the message queue, and writing the real-time change data into a target database. Therefore, the scheme realizes asynchronous processing of a data acquisition process and a data writing process in the data synchronization process of different databases by storing data in the message queue, and the data acquisition process and the data writing process do not interfere with each other. Therefore, data synchronization of a plurality of different databases can be performed, the transverse expansion of the data synchronization is realized, the data synchronization between heterogeneous databases can be supported, and the synchronization efficiency is higher.
Fig. 4 is a schematic structural diagram of another data synchronization apparatus according to an embodiment of the present application. As shown in fig. 4, the data synchronization apparatus includes:
a memory 20 for storing a computer program.
A processor 21 for implementing the steps of the data synchronization method as mentioned in the above embodiments when executing the computer program.
The data synchronization device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer.
The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The Processor 21 may be implemented in hardware using at least one of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 21 may further include an Artificial Intelligence (AI) processor for processing computing operations related to machine learning.
The memory 20 may include one or more computer-readable storage media, which may be non-transitory. Memory 20 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used for storing the following computer program 201, wherein after being loaded and executed by the processor 21, the computer program can implement the relevant steps of the data synchronization method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202, data 203, and the like, and the storage manner may be a transient storage manner or a permanent storage manner. Operating system 202 may include, among other things, windows, unix, linux, etc. Data 203 may include, but is not limited to, data involved in a data synchronization method.
In some embodiments, the data synchronization device may further include a display 22, an input/output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
Those skilled in the art will appreciate that the configuration shown in fig. 4 does not constitute a limitation of the data synchronization apparatus and may include more or fewer components than those shown.
In this embodiment, the data synchronization apparatus includes a memory and a processor. Wherein the processor is adapted to carry out the steps of the data synchronization method as mentioned in the above embodiments when executing the computer program. The real-time change data in the source database is acquired and stored in a message queue; and acquiring real-time change data in the message queue, and writing the real-time change data into a target database. Therefore, the scheme realizes asynchronous processing of a data acquisition process and a data writing process in the data synchronization process of different databases by storing data in the message queue, and the data acquisition process and the data writing process do not interfere with each other. Therefore, data synchronization of a plurality of different databases can be performed, the transverse expansion of the data synchronization is realized, the data synchronization between heterogeneous databases can be supported, and the synchronization efficiency is higher.
Finally, the application also provides a corresponding embodiment of the computer readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps as set forth in the above-mentioned method embodiments.
It is understood that, if the method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, which are essential or part of the prior art, or all or part of the technical solutions may be embodied in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In this embodiment, a computer program is stored on a computer-readable storage medium, and when executed by a processor, the computer program implements the steps described in the above method embodiments. The real-time change data in the source database is acquired and stored in a message queue; and acquiring real-time change data in the message queue, and writing the real-time change data into a target database. Therefore, the scheme stores the data in the message queue, realizes asynchronous processing of a data acquisition flow and a data writing flow in the data synchronization process of different databases, and the two processes do not interfere with each other. Therefore, data synchronization of a plurality of different databases can be carried out, the horizontal extension of the data synchronization is realized, the data synchronization among heterogeneous databases can be supported, and the synchronization efficiency is higher.
In order to make those skilled in the art better understand the technical solution of the present application, the present application is further described in detail below with reference to fig. 5. Fig. 5 is a flowchart of a data synchronization method provided in an application scenario according to an embodiment of the present application. As shown in fig. 5, the method includes:
s18: extracting data of a source database;
s19: storing the extracted data to a message queue;
s20: reading data from the message queue;
s21: and writing the data into the destination database.
The application scene is specifically as follows: when the database data synchronization starts, extracting data to be synchronized from a source database and storing the data to be synchronized into a message queue; at this time, the destination database which needs to perform data synchronization reads data from the message queue and writes the data.
If the real-time change data of the source database is acquired through a corresponding interface of the database, then table data needing to be synchronized is filtered according to the configuration information of the first configuration center, and the table data is written into a message queue; and the target database acquires real-time change data in the message queue, filters required table data according to the configuration information of the second configuration center, and writes the table data into the second configuration center.
The data synchronization method, apparatus, and computer-readable storage medium provided in the present application are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method of data synchronization, comprising:
acquiring real-time change data in a source database;
storing the real-time change data in a message queue;
acquiring the real-time change data in the message queue;
and writing the real-time change data into a destination database.
2. The data synchronization method of claim 1, wherein the storing the real-time change data in a message queue comprises:
acquiring a first configuration center; the first configuration center is used for configuring the corresponding relation between table data in the real-time change data and topic of the message queue;
filtering the table data according to the first configuration center;
and writing the filtered table data into the topic of the corresponding message queue according to the first configuration center.
3. The data synchronization method of claim 2, wherein the writing the real-time change data to a destination database comprises:
acquiring a second configuration center; the second configuration center is used for configuring a corresponding relation between the topic of the message queue and a destination table required by the destination database;
filtering the table data according to the second configuration center;
and writing the filtered table data into the target table corresponding to the target database according to the second configuration center.
4. The data synchronization method according to claim 1, further comprising:
and docking the real-time change data in the message queue into the Flink for processing the real-time change data through the Flink.
5. The data synchronization method according to any one of claims 1 to 4, further comprising, before the acquiring the real-time change data in the source database:
monitoring a data synchronization request sent by the target database;
judging whether the data synchronization request has a data synchronization authority or not;
and if so, entering the step of acquiring real-time change data in the source database.
6. The method of claim 5, further comprising, after said writing said real-time change data to a destination database:
and generating prompt information for completing the synchronization of the real-time change data.
7. The data synchronization method of claim 5, wherein if the data synchronization request does not have the data synchronization right, further comprising:
and outputting the information of data synchronization failure.
8. A data synchronization apparatus, comprising:
the first acquisition module is used for acquiring real-time change data in the source database;
the storage module is used for storing the real-time change data in a message queue;
the second obtaining module is used for obtaining the real-time change data in the message queue;
and the writing module is used for writing the real-time change data into a target database.
9. A data synchronization apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data synchronization method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data synchronization method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210763888.9A CN115221242A (en) | 2022-06-30 | 2022-06-30 | Data synchronization method and device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210763888.9A CN115221242A (en) | 2022-06-30 | 2022-06-30 | Data synchronization method and device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115221242A true CN115221242A (en) | 2022-10-21 |
Family
ID=83610808
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210763888.9A Pending CN115221242A (en) | 2022-06-30 | 2022-06-30 | Data synchronization method and device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115221242A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116401117A (en) * | 2023-03-09 | 2023-07-07 | 北京海致星图科技有限公司 | Data processing method combining stream computing system and traditional software application system |
-
2022
- 2022-06-30 CN CN202210763888.9A patent/CN115221242A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116401117A (en) * | 2023-03-09 | 2023-07-07 | 北京海致星图科技有限公司 | Data processing method combining stream computing system and traditional software application system |
| CN116401117B (en) * | 2023-03-09 | 2024-04-09 | 北京海致星图科技有限公司 | Data processing method combining stream computing system and traditional software application system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9418241B2 (en) | Unified platform for big data processing | |
| CN106375360B (en) | Graph data updating method, device and system | |
| CN112636992B (en) | Dynamic routing method, device, equipment and storage medium | |
| CN112347192A (en) | Data synchronization method, device, platform and readable medium | |
| CN113835692A (en) | Dictionary data processing method and device, electronic equipment and computer storage medium | |
| CN114416868B (en) | Data synchronization method, device, equipment and storage medium | |
| CN112416991A (en) | Data processing method and device and storage medium | |
| CN115221242A (en) | Data synchronization method and device and computer readable storage medium | |
| CN115408391A (en) | Database table changing method, device, equipment and storage medium | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN112181724A (en) | Big data disaster tolerance method and device and electronic equipment | |
| CN109614271A (en) | Control method, device, equipment and the storage medium of multiple company-data consistency | |
| CN114077518A (en) | Data snapshot method, device, equipment and storage medium | |
| CN112491943A (en) | Data request method, device, storage medium and electronic equipment | |
| CN115730012A (en) | Database desensitization method and system | |
| CN115328898A (en) | Data processing method and device, electronic equipment and medium | |
| CN110580212B (en) | Data export method and device of application program, electronic equipment and storage medium | |
| CN114201508A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
| CN114218313A (en) | Data management method, device, electronic equipment, storage medium and product | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN111930841A (en) | Data synchronization method and device | |
| CN113626869A (en) | Data processing method, system, electronic device and storage medium | |
| CN112488462A (en) | Unified pushing method, device and medium for workflow data | |
| CN111147554A (en) | Data storage method and device and computer system | |
| CN111800519A (en) | Communication system, method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |