CN112417050A - Data synchronization method and device, system, storage medium and electronic device - Google Patents
Data synchronization method and device, system, storage medium and electronic device Download PDFInfo
- Publication number
- CN112417050A CN112417050A CN202011339972.5A CN202011339972A CN112417050A CN 112417050 A CN112417050 A CN 112417050A CN 202011339972 A CN202011339972 A CN 202011339972A CN 112417050 A CN112417050 A CN 112417050A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- sending
- component
- subscription system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000008859 change Effects 0.000 claims abstract description 105
- 238000013481 data capture Methods 0.000 claims abstract description 43
- 238000012544 monitoring process Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims description 132
- 238000004590 computer program Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The invention provides a data synchronization method, a device, a system, a storage medium and an electronic device, wherein the method comprises the following steps: monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data; and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system. The invention solves the technical problem of longer time required by data synchronization in the related technology, reduces the time required by data synchronization and improves the real-time property of data synchronization.
Description
Technical Field
The present invention relates to the field of communications, and in particular, to a data synchronization method, apparatus, system, storage medium, and electronic apparatus.
Background
Along with the popularization of intelligent household appliances, a user can use an intelligent control Application program (APP) installed in a terminal to control the intelligent household appliance, can share information of the aspect of scientific and technological life through the intelligent control APP, and can search the information through the intelligent control APP. In the related technology, the information shared and uploaded by the user through the intelligent control APP is stored in the database, the stored data is synchronized to the search system through the database, and the user can further search the information uploaded by the user or other users in the search system through the intelligent control APP. However, in the related art, taking the database as the TiDB database supporting distributed data storage as an example, when data synchronization is performed between the TiDB database and the search system, it is common to collect Binlog logs generated by each TiDB database instance through the TiDB Binlog component, submit the collected binlogs to the search system after sorting according to the time of transaction submission, and the search system performs data synchronization after parsing the received binlogs. However, since the data uploaded by the user is massive, a massive binlog exists, a certain time is consumed in each link of generation, collection, submission and the like of the massive binlog, and a risk of loss exists, so that the time required by the data synchronization process is long, and the data synchronization efficiency is low.
Aiming at the technical problem that the time required by data synchronization is long in the related technology, an effective technical scheme is not provided yet.
Disclosure of Invention
The embodiment of the invention provides a data synchronization method, a data synchronization device, a data synchronization system, a storage medium and an electronic device, and at least solves the technical problem that the time required for data synchronization is long in the related technology.
According to an embodiment of the present invention, there is provided a data synchronization method including: monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data; and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
Optionally, the database includes a TiDB database, and the monitoring the data change event in the database by a change data capture component in the database includes: monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database; and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
Optionally, the sending the data to the target search system through the subscription system includes: sending the data to a target processing component through the subscription system, wherein the target processing component is used for processing the data to obtain processed data and sending the processed data to the target search system; wherein the method further comprises: and storing the processed data through the target search system.
Optionally, the target processing component comprises an extract transform loading component, wherein after the sending of the data to the target processing component by the subscription system, the method further comprises: analyzing the data through the extraction conversion loading assembly to obtain first data, and removing the duplicate of the first data to obtain second data; determining the second data as the processed data.
Optionally, the target processing component includes a plurality of processing components, wherein the sending the data to the target processing component through the subscription system includes: and sending the processed data to a first processing component in the target processing components through the subscription system when the processed data belongs to a target type, wherein the plurality of processing components include the first processing component, the first processing component is used for acquiring the data belonging to the target type from the subscription system, and each processing component in the plurality of processing components is used for acquiring the data of one of a plurality of types from the subscription system, and the plurality of types include the target type.
Optionally, the sending, by the change data capture component, the first notification message to the subscription system includes: sending, by the change data capture component, the first notification message to the subscription system based on an open data protocol.
According to another embodiment of the present invention, there is provided a data synchronization apparatus including: the monitoring module is used for monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; a first sending module, configured to send a first notification message to a subscription system through the change data capture component, where the first notification message is used to instruct the subscription system to store the data; and the second sending module is used for sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
Optionally, the database includes a TiDB database, and the listening module is further configured to: monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database; and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
Optionally, the second sending module is further configured to: sending the data to a target processing component through the subscription system, processing the data through the target processing component to obtain processed data, and sending the processed data to the target search system through the target processing component; wherein the apparatus further comprises a storage module configured to: and storing the processed data through the target search system.
Optionally, the target processing component includes an extraction conversion loading component, where the apparatus further includes a processing module, configured to, after the data is sent to the target processing component through the subscription system, parse the data through the extraction conversion loading component to obtain first data, and perform deduplication on the first data to obtain second data; determining the second data as the processed data.
Optionally, the target processing component includes a plurality of processing components, wherein the second sending module is further configured to: and sending the processed data to a first processing component in the target processing components through the subscription system when the processed data belongs to a target type, wherein the plurality of processing components include the first processing component, the first processing component is used for acquiring the data belonging to the target type from the subscription system, and each processing component in the plurality of processing components is used for acquiring the data of one of a plurality of types from the subscription system, and the plurality of types include the target type.
Optionally, the first sending module is further configured to: sending, by the change data capture component, the first notification message to the subscription system based on an open data protocol.
According to another embodiment of the present invention, there is provided a data synchronization system including: the system comprises a database, a subscription system and a target search system, wherein the database is used for monitoring a data change event in the database through a change data capture component, and the data change event is used for indicating data to be written into the database; the database is further used for sending a first notification message to the subscription system through the changed data capturing component, wherein the first notification message is used for instructing the subscription system to store the data; the subscription system is used for receiving the first notification message and storing the data; and the subscription system is also used for sending the data to the target search system so as to synchronize the data of the target search system.
Optionally, the database comprises a TiDB database for: monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database; and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
Optionally, the system further includes a target processing component, wherein the subscription system is further configured to: sending the data to a target processing component; the target processing component is used for processing the data to obtain processed data and sending the processed data to the target search system; wherein the target search system is further configured to: and storing the processed data.
Optionally, the target processing component includes an extract transform load component, wherein the target processing component is further configured to: after the data are sent to a target processing component through the subscription system, analyzing the data to obtain first data, and removing the duplication of the first data to obtain second data; determining the second data as the processed data.
Optionally, the target processing component comprises a plurality of processing components, wherein the subscription system is further configured to: and sending the processed data to a first processing component in the target processing components through the subscription system when the processed data belongs to a target type, wherein the plurality of processing components include the first processing component, the first processing component is used for acquiring the data belonging to the target type from the subscription system, and each processing component in the plurality of processing components is used for acquiring the data of one of a plurality of types from the subscription system, and the plurality of types include the target type.
Optionally, the database is further configured to: sending, by the change data capture component, the first notification message to the subscription system based on an open data protocol.
Alternatively, according to another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is arranged to perform the above-mentioned method when executed.
Alternatively, according to another embodiment of the present invention, there is provided an electronic apparatus, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the above method.
According to the invention, a data change event in a database is monitored through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data; and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system. The data synchronization does not need to go through the processes of generation, collection and submission of the binlog any more, so that the time required by the data synchronization is saved, the technical problem of long time required by the data synchronization in the related technology is solved, the time required by the data synchronization is reduced, and the real-time performance of the data synchronization is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow chart of a data synchronization method according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating interception of a TiDB database in a data synchronization method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an application environment of a data synchronization method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a target processing component according to an embodiment of the invention;
FIG. 5 is a flow chart of a data synchronization method according to another embodiment of the present invention;
FIG. 6 is a block diagram of a data synchronization apparatus according to an embodiment of the present invention;
FIG. 7 is a block diagram of a data synchronization apparatus according to another embodiment of the present invention;
FIG. 8 is a block diagram of a data synchronization apparatus according to another embodiment of the present invention;
FIG. 9 is a schematic diagram of a data synchronization system according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
An embodiment of the present invention provides a data synchronization method, and fig. 1 is a flowchart of the data synchronization method according to the embodiment of the present invention, as shown in fig. 1, including:
step S102, monitoring a data change event in a database through a changed data capture component in the database, wherein the data change event is used for indicating to write data into the database;
step S104, sending a first notification message to a subscription system through the changed data capturing component, wherein the first notification message is used for instructing the subscription system to store the data;
and step S106, sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
According to the invention, a data change event in a database is monitored through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data; and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system. The data synchronization does not need to go through the processes of generation, collection and submission of the binlog any more, so that the time required by the data synchronization is saved, the technical problem of long time required by the data synchronization in the related technology is solved, the time required by the data synchronization is reduced, and the real-time performance of the data synchronization is improved.
In the above embodiment, the target search system may store the data, and return the data to the user after receiving a search request for the data initiated by the user in the following.
In an optional embodiment, the database comprises a TiDB database, wherein the monitoring for data change events in the database by a change data capture component in the database comprises: monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database; and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
In the above embodiment, the storage layer (also referred to as a KV (Key-Value) layer for storing Data) of the TiDB database is monitored by the monitoring component in the TiDB database, and the monitored Data change event is notified to the Change Data Capture (CDC) component, so that real-time Capture of Data change is realized, and the time for Data synchronization is saved. As an alternative, as shown in fig. 2, the data change event of the KV layer in the TiDB database may be directly monitored by the change data capture component, so that the efficiency of data synchronization is further improved as a whole.
In the above embodiment, the sending the data to the target search system through the subscription system includes: sending the data to a target processing component through the subscription system, wherein the target processing component is used for processing the data to obtain processed data and sending the processed data to the target search system; wherein the method further comprises: and storing the processed data through the target search system.
Optionally, the subscription system is a Message Queue Kafka system (MQ Kafka system for short), where the first notification Message includes the data, and a subscription request is sent to the subscription system by the target processing component, where the subscription request is used to instruct the target processing component to subscribe to the data of the TiDB database, so that after the subscription system receives the first notification Message and stores (including but not limited to caching), the stored data may be sent to the target processing component.
The above embodiments can be applied to the application environment shown in fig. 3, wherein, as shown in fig. 3, the system of the application environment includes a user terminal, a TiDB database, a subscription system, a target processing component, and a search system.
As an optional implementation, the target processing component includes an extraction conversion loading component, wherein after the sending of the data to the target processing component through the subscription system, the method further includes: analyzing the data through the extraction conversion loading assembly to obtain first data, and removing the duplicate of the first data to obtain second data; determining the second data as the processed data.
Based on the above embodiment, an Extract-Transform-Load (ETL) component is located between the subscription system and the target search system, the subscription system sends data to the ETL component, the ETL component processes the data (including but not limited to analysis, deduplication, and other processing operations) to obtain processed data, and sends the processed data to the target search system, thereby implementing data synchronization on the target search system.
Wherein the target processing component comprises a plurality of processing components, wherein the sending the data to the target processing component through the subscription system comprises: and sending the processed data to a first processing component in the target processing components through the subscription system when the processed data belongs to a target type, wherein the plurality of processing components include the first processing component, the first processing component is used for acquiring the data belonging to the target type from the subscription system, and each processing component in the plurality of processing components is used for acquiring the data of one of a plurality of types from the subscription system, and the plurality of types include the target type.
It should be noted that the plurality of types include, but are not limited to, a plurality of topics (or called topics) in the subscription system, and the target type may be a target Topic (or called Topic). Thus, in the above embodiment, as shown in fig. 4, the target processing component between the subscription system and the target search system includes a plurality of processing components (as an example, fig. 4 shows that the target processing component includes processing components 1 to n, where n is a natural number), and the sending of the data in the subscription system to the target search system is implemented by the plurality of processing components, and optionally, each processing component is responsible for sending the data of one topic.
In the above embodiment, the sending, by the change data capture component, the first notification message to the subscription system includes: sending, by the change data capture component, the first notification message to the subscription system based on an open data protocol.
As an alternative embodiment, the open Data protocol includes, but is not limited to, OData (open Data protocol) or Jdbopen protocol.
The data synchronization method in the above embodiment is explained with reference to an example, but is not limited to the technical solution of the embodiment of the present invention.
In the related technology, when data synchronization is carried out, the bin log generated by each TiDB instance is collected based on a TiDB Binlog component, so that data synchronization is achieved to the downstream, wherein a Drainer component in the TiDB Binlog component adopts single-node merging and sorting, the Drainer component cannot cope with a very large-scale TiDB cluster, and meanwhile, the bin log may be used in an extreme case, so that the transaction state of a TiKV node (wherein a storage layer KV layer is located in the TiKV node) in the TiDB needs to be checked reversely, further, the data synchronization delay can reach 10 minutes, and the problems of data safety and high service availability need to be solved independently by the TiDB Binlog component, so that the implementation cost is high.
In an alternative embodiment, there is provided a data synchronization method, as shown in fig. 5, comprising the steps of:
step 501, a user issues content information through an intelligent control APP in a user terminal;
for example, content information is sent through a publishing portal in an intelligent control APP
502, a background server of the intelligent control APP stores content information issued by a user in a TiDB database;
step 503, the TiDB database returns a message to the user terminal, so as to notify the user terminal that the content is successfully published;
step 504, when the content information (i.e. the data in the above embodiment) is stored in the TiDB database, a data change event occurs in the TiDB database;
it should be noted that, in this embodiment, the execution order of step 503 and step 504 may be interchanged, that is, step 504 is executed first, and then step 503 is executed.
505, capturing a data change event occurring in the TiDB database by a change data capturing component, namely capturing change data;
wherein, the data change event includes but is not limited to a data write event;
step 506, the change data capturing component sends the captured data to a subscription system;
optionally, the subscription system is an MQ Kafka system. Wherein the change data is delivered to the MQ Kafka system through a standard open data protocol through the change data capture component;
step 507, the subscription system caches the received data;
step 508, the subscription system sends the data to the target processing component;
in the above embodiment, the target processing component subscribes to the data written in the TiDB database from the subscription system;
step 509, the target processing component performs data processing on the received data to obtain processed data;
among these, data processing includes but is not limited to: analyzing, removing duplicate and other processing operations;
step 510, the target processing component sends the processed data to a target search system;
step 511, the target search system receives the processed data, stores the received processed data, and sends a stored data response to the target processing component, wherein the stored data response is used for notifying the target processing component that the processed data are stored;
step 512, the target processing component sends a consumption data response to the subscription system, where the consumption data response is used to notify the subscription system that the target search system has completed consuming the data (optionally, the target search system has completed consuming the data including, but not limited to, the target search system having completed acquiring the data);
step 513, the user searches the content information (including but not limited to the content information (such as the data in the above embodiment) previously sent by the user himself, the content information published by other users) in the target search system through the user terminal;
and 514, the target searching system sends the searched content information to the target terminal.
In the embodiment, the whole process of the user publishing the content information to the user searching the content information is involved, and a non-invasive and high-performance near real-time data synchronization method applied to a mass data connection storage system and a search system is provided, wherein a CDC component based on a distributed relational database TiDB and a Kafka subscription system are applied. Optionally, the CDC component may be implemented based on a KV Change log (KV Change Logs) of the TiKV nodes in the TiDB database.
In the above embodiments, where the CDC component is a stateless node, high availability is achieved through a registry (etcd) inside a management Driver (PD) component in the TiDB, providing higher reliability. Optionally, a plurality of CDC components may be employed to form a CDC cluster in the above embodiments, where the CDC cluster supports creation of a plurality of synchronization tasks, thereby implementing data synchronization to a plurality of different downstream (including but not limited to target search systems and other systems). Optionally, during the CDC cluster failure, the same data definition statement/data operation statement (DDL/DML) may be repeated; the ETL component supports clusters, the size of which can be extended to the number of partitions of TOPICs in Kafka (i.e., the total number of TOPICs), which do not affect the consumption of the entire data even if a partial node fails, and which can continue to consume after restarting even if the entire cluster hangs.
Through the embodiment, the method for synchronizing the near-real-time data to the search system is realized, wherein the sequential performance and consistency of data partition consumption are guaranteed by the ETL component on the basis of millisecond delay for data consumption, the ETL component is written into the target search system in real time, so that the real-time performance of data synchronization is guaranteed, the ETL component is expandable, highly reliable and flexible, the complete processing of the business data and the logical decoupling of data change capture are realized, and the business development is more convenient.
The synchronization delay of the data synchronization method in the embodiment is usually in the millisecond level, the data synchronization method has better expansibility, can cope with the use scene of a very large-scale TiDB cluster, has no single-point bottleneck when a plurality of nodes write downstream, and can ensure the data security; each node is stateless, high availability of data and high availability of service are conveniently realized, data conversion is completed from the change of captured data to the data change of subscription, and the whole data is synchronized to realize the near real-time retrieval of the data.
As described above, the present invention is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and when the drag editing operation is performed, the selection and drag of the content in the circle are not limited to operations such as right double click, etc., because the corresponding editing tool is opened and recognized for various contents in various content formats when editing the content, among circles of various content formats. The protection scope of the present invention shall be subject to the protection scope of the claims.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
According to another embodiment of the present invention, a data synchronization apparatus is provided, which is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted for brevity. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 6 is a block diagram of a data synchronization apparatus according to an embodiment of the present invention, the apparatus including:
a monitoring module 62, configured to monitor a data change event in a database through a changed data capture component in the database, where the data change event is used to instruct to write data into the database;
a first sending module 64, configured to send a first notification message to a subscription system through the change data capture component, where the first notification message is used to instruct the subscription system to store the data;
a second sending module 66, configured to send the data to the target search system through the subscription system to perform data synchronization on the target search system.
According to the invention, a data change event in a database is monitored through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data; and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system. The data synchronization does not need to go through the processes of generation, collection and submission of the binlog any more, so that the time required by the data synchronization is saved, the technical problem of long time required by the data synchronization in the related technology is solved, the time required by the data synchronization is reduced, and the real-time performance of the data synchronization is improved.
Wherein the database includes a TiDB database, and the listening module 62 is further configured to: monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database; and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
It should be noted that, the second sending module 66 is further configured to: sending the data to a target processing component through the subscription system, processing the data through the target processing component to obtain processed data, and sending the processed data to the target search system through the target processing component; as shown in fig. 7, the apparatus further includes a storage module 72, configured to: and storing the processed data through the target search system.
As an optional implementation manner, the target processing component includes an extraction conversion loading component, where as shown in fig. 8, the apparatus further includes a processing module 82, configured to, after the data is sent to the target processing component through the subscription system, parse the data through the extraction conversion loading component to obtain first data, and perform deduplication on the first data to obtain second data; determining the second data as the processed data.
The storage module 72 is further configured to store the processed data.
In the above embodiment, the target processing component includes a plurality of processing components, wherein the second sending module 66 is further configured to: and sending the processed data to a first processing component in the target processing components through the subscription system when the processed data belongs to a target type, wherein the plurality of processing components include the first processing component, the first processing component is used for acquiring the data belonging to the target type from the subscription system, and each processing component in the plurality of processing components is used for acquiring the data of one of a plurality of types from the subscription system, and the plurality of types include the target type.
Optionally, the first sending module 64 is further configured to: sending, by the change data capture component, the first notification message to the subscription system based on an open data protocol.
An embodiment of the present invention further provides a data synchronization system, as shown in fig. 9, the system includes: a database 92, a subscription system 94, and a target search system 96, wherein,
the database 92 is configured to listen to a data change event in the database 92 through a change data capture component in the database 92, where the data change event is used to indicate that data is written into the database 92;
the database 92 is further configured to send a first notification message to the subscription system 94 through the change data capture component, wherein the first notification message is used to instruct the subscription system 94 to store the data;
the subscription system 94 is configured to receive the first notification message and store the data;
the subscription system 94 is further configured to send the data to the target search system 96 to perform data synchronization on the target search system 96.
According to the invention, a data change event in a database is monitored through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database; sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data; and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system. The data synchronization does not need to go through the processes of generation, collection and submission of the binlog any more, so that the time required by the data synchronization is saved, the technical problem of long time required by the data synchronization in the related technology is solved, the time required by the data synchronization is reduced, and the real-time performance of the data synchronization is improved.
Wherein the database 92 comprises a TiDB database configured to: monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database; and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
Based on the above embodiment, the data synchronization system further includes a target processing component, wherein the subscription system 96 is further configured to: sending the data to a target processing component; the target processing component is configured to process the data to obtain processed data, and send the processed data to the target search system 96; wherein the target search system 96 is further configured to: and storing the processed data.
As an optional implementation, the target processing component includes a decimation conversion loading component, wherein the target processing component is further configured to: after the data is sent to the target processing component by the subscription system 96, analyzing the data to obtain first data, and performing deduplication on the first data to obtain second data; determining the second data as the processed data.
Wherein the target processing component comprises a plurality of processing components, wherein the subscription system 96 is further configured to: and in the case that the target processed data belongs to a target type, sending the processed data to a first processing component of the target processing components through the subscription system 96, wherein the plurality of processing components include the first processing component, the first processing component is configured to obtain the data belonging to the target type from the subscription system 96, and each processing component of the plurality of processing components is configured to obtain data of one of a plurality of types from the subscription system 96, and the plurality of types include the target type.
It should be noted that, the TiDB database is further configured to: the first notification message is sent to the subscription system 96 based on an open data protocol through the change data capture component.
An embodiment of the present invention further provides a storage medium including a stored program, wherein the program executes any one of the methods described above.
Alternatively, in the present embodiment, the storage medium may be configured to store program codes for performing the following steps:
s1, monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database;
s2, sending a first notification message to a subscription system through the changed data capturing component, wherein the first notification message is used for instructing the subscription system to store the data;
and S3, sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database;
s2, sending a first notification message to a subscription system through the changed data capturing component, wherein the first notification message is used for instructing the subscription system to store the data;
and S3, sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
Fig. 10 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention. Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 10 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 10 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 10, or have a different configuration than shown in FIG. 10.
The memory 1002 may be used to store software programs and modules, such as program instructions/modules corresponding to the data synchronization method and the data synchronization apparatus in the embodiment of the present invention, and the processor 1004 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002, that is, implementing the data synchronization method. The memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1002 may further include memory located remotely from the processor 1004, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. As an example, the memory 1002 may include, but is not limited to, the listening module 62, the first sending module 64, and the second sending module 66 of the data synchronization apparatus. In addition, the data synchronization device may further include, but is not limited to, other module units in the data synchronization device, which is not described in this example again.
Optionally, the transmission device 1006 is used for receiving or transmitting data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transport device 1006 includes a Network adapter (NIC) that can be connected to a router via a Network cable to communicate with the internet or a local area Network. In one example, the transmission device 1006 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In addition, the electronic device further includes: a display 1008 for displaying a screen; and a connection bus 1010 for connecting the respective module parts in the above-described electronic apparatus.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data synchronization, comprising:
monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database;
sending a first notification message to a subscription system through the change data capture component, wherein the first notification message is used for instructing the subscription system to store the data;
and sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
2. The method of claim 1, wherein the database comprises a TiDB database, and wherein listening for data change events in the database via a change data capture component in the database comprises:
monitoring the data change event executed in a storage layer of the TiDB database through a monitoring component in the TiDB database;
and sending a second notification message to the changed data capturing component when the monitoring component monitors the data change event, wherein the second notification message is used for indicating that the data is written in the storage layer.
3. The method of claim 1, wherein the sending the data to a target search system via the subscription system comprises:
sending the data to a target processing component through the subscription system, wherein the target processing component is used for processing the data to obtain processed data and sending the processed data to the target search system;
wherein the method further comprises:
and storing the processed data through the target search system.
4. The method of claim 3, wherein the target processing component comprises a decimating transform loader component, and wherein after the sending the data to the target processing component via the subscription system, the method further comprises:
analyzing the data through the extraction conversion loading assembly to obtain first data, and removing the duplicate of the first data to obtain second data;
determining the second data as the processed data.
5. The method of claim 4, wherein the target processing component comprises a plurality of processing components, and wherein the sending the data to the target processing component via the subscription system comprises:
and sending the processed data to a first processing component in the target processing components through the subscription system when the processed data belongs to a target type, wherein the plurality of processing components include the first processing component, the first processing component is used for acquiring the data belonging to the target type from the subscription system, and each processing component in the plurality of processing components is used for acquiring the data of one of a plurality of types from the subscription system, and the plurality of types include the target type.
6. The method of claim 1, wherein sending a first notification message to a subscription system via the change data capture component comprises:
sending, by the change data capture component, the first notification message to the subscription system based on an open data protocol.
7. A data synchronization apparatus, comprising:
the monitoring module is used for monitoring a data change event in a database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database;
a first sending module, configured to send a first notification message to a subscription system through the change data capture component, where the first notification message is used to instruct the subscription system to store the data;
and the second sending module is used for sending the data to a target search system through the subscription system so as to synchronize the data of the target search system.
8. A data synchronization system, comprising: a database, a subscription system, and a target search system, wherein,
the database is used for monitoring a data change event in the database through a change data capture component in the database, wherein the data change event is used for indicating that data is written into the database;
the database is further used for sending a first notification message to the subscription system through the changed data capturing component, wherein the first notification message is used for instructing the subscription system to store the data;
the subscription system is used for receiving the first notification message and storing the data;
and the subscription system is also used for sending the data to the target search system so as to synchronize the data of the target search system.
9. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 6 when executed.
10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 6 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011339972.5A CN112417050A (en) | 2020-11-25 | 2020-11-25 | Data synchronization method and device, system, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011339972.5A CN112417050A (en) | 2020-11-25 | 2020-11-25 | Data synchronization method and device, system, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112417050A true CN112417050A (en) | 2021-02-26 |
Family
ID=74842794
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011339972.5A Pending CN112417050A (en) | 2020-11-25 | 2020-11-25 | Data synchronization method and device, system, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112417050A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114048241A (en) * | 2022-01-12 | 2022-02-15 | 树根互联股份有限公司 | Industrial Internet data synchronization method and device and electronic equipment |
| WO2023123755A1 (en) * | 2021-12-30 | 2023-07-06 | 中国民航信息网络股份有限公司 | Information pushing method and apparatus, electronic device and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040078464A1 (en) * | 1999-09-16 | 2004-04-22 | Rajan Sreeranga P. | Method and apparatus for enabling real time monitoring and notification of data updates for WEB-based data synchronization services |
| CN1953420A (en) * | 2006-09-26 | 2007-04-25 | 杭州华为三康技术有限公司 | A method to forward the channel message and network device |
| CN102882973A (en) * | 2012-10-11 | 2013-01-16 | 北京邮电大学 | Distributed load balancing system and distributed load balancing method based on peer to peer (P2P) technology |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN108197263A (en) * | 2017-12-30 | 2018-06-22 | 苏州精易会信息技术有限公司 | Method of data synchronization |
| CN109582731A (en) * | 2018-10-18 | 2019-04-05 | 恒峰信息技术有限公司 | A kind of real time data synchronization method and system |
| CN110046137A (en) * | 2017-12-20 | 2019-07-23 | 宏碁云端技术公司 | By data stream packet and the system and method that store into cloud storage file |
| CN110532113A (en) * | 2019-08-30 | 2019-12-03 | 北京地平线机器人技术研发有限公司 | Information processing method, device, computer readable storage medium and electronic equipment |
| CN111274257A (en) * | 2020-01-20 | 2020-06-12 | 山东汇贸电子口岸有限公司 | Real-time synchronization method and system based on data |
| CN111382206A (en) * | 2020-03-20 | 2020-07-07 | 北京奇艺世纪科技有限公司 | Data storage method and device |
-
2020
- 2020-11-25 CN CN202011339972.5A patent/CN112417050A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040078464A1 (en) * | 1999-09-16 | 2004-04-22 | Rajan Sreeranga P. | Method and apparatus for enabling real time monitoring and notification of data updates for WEB-based data synchronization services |
| CN1953420A (en) * | 2006-09-26 | 2007-04-25 | 杭州华为三康技术有限公司 | A method to forward the channel message and network device |
| CN102882973A (en) * | 2012-10-11 | 2013-01-16 | 北京邮电大学 | Distributed load balancing system and distributed load balancing method based on peer to peer (P2P) technology |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN110046137A (en) * | 2017-12-20 | 2019-07-23 | 宏碁云端技术公司 | By data stream packet and the system and method that store into cloud storage file |
| CN108197263A (en) * | 2017-12-30 | 2018-06-22 | 苏州精易会信息技术有限公司 | Method of data synchronization |
| CN109582731A (en) * | 2018-10-18 | 2019-04-05 | 恒峰信息技术有限公司 | A kind of real time data synchronization method and system |
| CN110532113A (en) * | 2019-08-30 | 2019-12-03 | 北京地平线机器人技术研发有限公司 | Information processing method, device, computer readable storage medium and electronic equipment |
| CN111274257A (en) * | 2020-01-20 | 2020-06-12 | 山东汇贸电子口岸有限公司 | Real-time synchronization method and system based on data |
| CN111382206A (en) * | 2020-03-20 | 2020-07-07 | 北京奇艺世纪科技有限公司 | Data storage method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023123755A1 (en) * | 2021-12-30 | 2023-07-06 | 中国民航信息网络股份有限公司 | Information pushing method and apparatus, electronic device and storage medium |
| CN114048241A (en) * | 2022-01-12 | 2022-02-15 | 树根互联股份有限公司 | Industrial Internet data synchronization method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107018042B (en) | Tracking method and tracking system for online service system | |
| CN110908879A (en) | Method, device, terminal and storage medium for reporting buried point data | |
| CN109918349B (en) | Log processing method, log processing device, storage medium and electronic device | |
| CN111090699A (en) | Service data synchronization method and device, storage medium and electronic device | |
| CN111046011B (en) | Log collection method, system, device, electronic equipment and readable storage medium | |
| CN111800443B (en) | Data processing system and method, device and electronic equipment | |
| CN105653425A (en) | Complicated event processing engine based monitoring system | |
| CN102594598A (en) | Log management system and implementation method thereof | |
| CA2896865A1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN110837423A (en) | Method and device for automatically acquiring data of guided transport vehicle | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| CN113138900B (en) | Data acquisition processing method and device and electronic equipment | |
| CN112445861B (en) | Information processing method, device, system and storage medium | |
| CN112579406B (en) | Log call chain generation method and device | |
| CN112732663A (en) | Log information processing method and device | |
| CN109684279B (en) | Data processing method and system | |
| CN108805741B (en) | Fusion method, device and system of power quality data | |
| CN113312174A (en) | Information query method and device, electronic equipment and container management system | |
| CN112685486A (en) | Data management method and device for database cluster, electronic equipment and storage medium | |
| CN112417015A (en) | Data distribution method and device, storage medium and electronic device | |
| CN113872814A (en) | Information processing method, device and system for content distribution network | |
| CN113157475A (en) | Log processing method and device, storage medium and electronic equipment | |
| CN107330089B (en) | Cross-network structured data collection system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210226 |