CN111400407A - Data synchronization method and device, storage medium and electronic device - Google Patents
Data synchronization method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN111400407A CN111400407A CN202010281618.5A CN202010281618A CN111400407A CN 111400407 A CN111400407 A CN 111400407A CN 202010281618 A CN202010281618 A CN 202010281618A CN 111400407 A CN111400407 A CN 111400407A
- Authority
- CN
- China
- Prior art keywords
- data
- topic
- target
- source table
- predetermined
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Abstract
The invention provides a data synchronization method and device, a storage medium and an electronic device, wherein the method comprises the following steps: querying a latest update record of a predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording data in a source table; screening incremental data in the source table based on the latest update record, and writing the incremental data into the predetermined topic to obtain a target topic; synchronizing the delta data recorded in the target topic into a target table. The invention solves the problem that the data in the source table needs to be read for many times when the data are synchronized in the related technology, and the pressure is caused to the source table, thereby achieving the purposes of reducing the pressure of the source table and improving the data synchronization efficiency.
Description
Technical Field
The present invention relates to the field of communications, and in particular, to a method and an apparatus for synchronizing data, a storage medium, and an electronic apparatus.
Background
In the related art, data in the same source table is synchronized into a plurality of target tables, and the data in the source table needs to be read for many times, so that a certain pressure is applied to the source table. For example, when incremental data extraction is performed, a temporary table or snapshot table needs to be created for recording the flag bits of the current extraction. When multi-table to single-table incremental data synchronization is performed, a flag bit record is created for each source table during each extraction, so that data in the source table needs to be read for many times, and a large storage resource is occupied. When a plurality of source data tables exist, the maximum value of the number of the data entry in the target data table may be from other source data tables, which causes the synchronization abnormality of the incremental data, thereby causing a certain pressure on the source tables.
Therefore, in the related art, when data synchronization is performed, data in the source table needs to be read for many times, and stress is caused on the source table.
In view of the above problems in the related art, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a data synchronization method and device, a storage medium and an electronic device, which are used for at least solving the problem that the data in a source table needs to be read for multiple times when the data are synchronized in the related technology, so that the source table is stressed.
According to an embodiment of the present invention, there is provided a data synchronization method including: querying a latest update record of a predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording data in a source table; screening incremental data in the source table based on the latest update record, and writing the incremental data into the predetermined topic to obtain a target topic; synchronizing the delta data recorded in the target topic into a target table.
According to another embodiment of the present invention, there is provided a data synchronization apparatus including: the query module is used for querying the latest update record of a predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording data in a source table; the processing module is used for screening the incremental data in the source table based on the latest update record and writing the incremental data into the preset topic to obtain a target topic; and the synchronization module is used for synchronizing the incremental data recorded in the target topic into a target table.
According to a further embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
According to the invention, the latest update record of the scheduled topic is inquired in the scheduled message system, the incremental data in the source table is screened according to the latest update record, the incremental data is written into the scheduled topic to obtain the target topic, and then the incremental data in the target topic is synchronized into the target table.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a block diagram of a hardware configuration of a mobile terminal of a data synchronization method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of synchronization of data according to an embodiment of the present invention;
FIG. 3 is a flow diagram of an incremental data production team according to a specific embodiment of the present invention;
FIG. 4 is a flow diagram of an incremental data consumption queue according to a specific embodiment of the present invention;
fig. 5 is a block diagram of a data synchronization apparatus according to an embodiment of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the present application may be executed in a mobile terminal, a computer terminal or a similar computing device. Taking the example of the method running on the mobile terminal, fig. 1 is a hardware structure block diagram of the mobile terminal of the data synchronization method according to the embodiment of the present invention. As shown in fig. 1, the mobile terminal 10 may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and optionally may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used for storing computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data synchronization method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In the present embodiment, a data synchronization method is provided, and fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, inquiring a latest update record of a predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording data in a source table;
step S204, screening the incremental data in the source table based on the latest update record, and writing the incremental data into the preset topic to obtain a target topic;
step S206, synchronizing the incremental data recorded in the target topic into a target table.
Alternatively, the executing subject of the above steps may be a processing terminal, or other devices with similar processing capabilities, such as a server, but not limited thereto.
In this embodiment, the source table includes an increment sequence field or a timestamp field, and the target table and source table may be identical in structure.
According to the invention, the latest update record of the scheduled topic is inquired in the scheduled message system, the incremental data in the source table is screened according to the latest update record, the incremental data is written into the scheduled topic to obtain the target topic, and then the incremental data in the target topic is synchronized into the target table.
In an optional embodiment, screening the incremental data in the source table based on the latest update record, and writing the incremental data into the predetermined topic to obtain the target topic includes: when the latest update record is determined to be empty, determining the full data of the source table as the incremental data, and writing the incremental data into the preset topic to obtain the target topic; and screening incremental data in the source table based on a target incremental sequence field value recorded in the latest update record when the latest update record is determined not to be empty, and writing the incremental data into the predetermined topic to obtain the target topic. In this embodiment, when synchronizing data in a predetermined message system, all data in the source table needs to be synchronized into the predetermined message system first, and subsequently, when a data update occurs in the source table, the data updated in the source table is written into the predetermined message system again (that is, the content before the update and the content after the update are recorded in the predetermined message system, and are updated several times and recorded several times). Therefore, during the first synchronization, the predetermined topic record is empty, the full data of the source table needs to be written into the predetermined topic, and in the subsequent synchronization, the incremental data in the source table needs to be determined, and the incremental data in the source table is written into the predetermined topic of the synchronization plan sequence. The source table may be a message queue, a java memory, or other queues.
In an alternative embodiment, filtering the delta data in the source table based on the target incremented sequence field value recorded in the most recent update record comprises: determining data recorded in the source table after the target incremented sequence field value as delta data in the source table. In this embodiment, data in the source table is queried, and according to the latest queried record, an incremental sequence field value in the record is obtained, that is, an end marker of the last synchronization and a start marker for the current extraction, and the record after the start marker of the source table is determined as incremental data. Wherein, the target incremented sequence field value may be identification information, field information, timestamp, etc.
In an optional embodiment, synchronizing the delta data recorded in the target topic into a target table comprises: inserting the data recorded in the target topic into the target table; under the condition that the conflict of the primary key exists, deleting the record corresponding to the conflicting primary key in the target table; reinserting data recorded in the target topic corresponding to the conflicting primary key in the target table. In this embodiment, if there is a primary key conflict exception, the records in the target table, which are the same as the exception record, are deleted, and then the exception record is inserted, so as to achieve the purpose of inserting and updating the incremental data. When data is synchronized in a target table, all data in a table corresponding to a source table stored in a predetermined message system are sequentially inserted into the target table, and a plurality of records with the same identification (namely, records before and after updating of the source table) in the predetermined message system are also sequentially inserted, but because the primary key identifications of the records before and after updating are the same, a problem of primary key collision can occur when the records are sequentially inserted into the target table.
In an optional embodiment, before querying a pre-created pre-defined message system for the latest update record of the pre-defined topic, the method further comprises: creating a synchronization task for the source table; allocating a unique sequence for the synchronization task; creating the predetermined topic in the predetermined message system with the same name as the sequence. In this embodiment, when incremental data synchronization is performed, there are scenes in which the source tables are multiple tables and are located in different databases, and a synchronization task may be created for each source table, and a unique sequence is generated, and a sequence homonym topic is created in a predetermined message system by using the sequence, so that synchronization efficiency is improved. Furthermore, the predetermined topic created in the predetermined messaging system may also be named differently from the sequence, as long as the consumer is aware of the synchronization from a particular table. For example, a plurality of topics recorded by a plurality of tables are stored in a predetermined message system, and the contents of different topics need to be synchronized to different target tables, so that a consumer needs to know the specific name of the topic to be synchronized when performing data synchronization, which is convenient for synchronization.
In an alternative embodiment, the predetermined message system includes kafka. In this embodiment, all data in the source table may be recorded in kafka, and then the data recorded in kafka is used for synchronization of the target table, thereby reducing the pressure on the source table. It should be noted that kafka is only an alternative embodiment, and the present invention is not limited to the predetermined message system.
How to synchronize data is described below with reference to specific embodiments:
taking the incremental sequence field as an example, the sync task needs to configure a timing schedule, including an incremental data production queue (writing data in the source table into kafka) and an incremental data consumption queue (inserting data in kafka, i.e., data written into kafka from the source table, into the target table). Fig. 3 is a flowchart of an incremental data production team according to an embodiment of the present invention, and as shown in fig. 3, the incremental data production team in the embodiment of the present invention includes the following steps:
in step S302, writing of data in the source table into kafka is started.
In step S304, the latest record of topic (corresponding to the above-mentioned predetermined topic) which is the same name as the synchronization plan sequence is queried in kafka. The latest record may be identification information, field information, etc., and then specific content corresponding to the identification information and the field information needs to be found from the source table and written into kafka.
Step S306, judging whether the latest record exists, if the judgment result is no, executing step S308, and if the judgment result is yes, executing step S310.
Step S308, acquiring the full data of the source table, writing the full data into the kafka queue, namely, when the data is synchronized for the first time, the topic record is empty, and inquiring the topic of the same name of the full data writing synchronization plan sequence of the source table. It should be noted that topic may also be named differently from the synchronization plan sequence, as long as the consumer can know to synchronize from a specific table (kafka stores a plurality of tables, and the contents in different tables may need to be synchronized to different target tables, so the consumer needs to know the specific table to be synchronized when synchronizing data).
Step S310, obtaining an incremental sequence field value in the record, screening the incremental data of the source table, and writing the incremental sequence field value into the kafka queue, that is, in the subsequent synchronization, according to the latest record queried in step S304, obtaining an incremental sequence field value in the record, that is, an end mark of the last synchronization, a start mark for the current extraction, and writing the record after the start mark of the source table as the incremental data into the synchronization plan sequence homonym topic.
In step S312, writing of data in the source table is completed in kafka.
Fig. 4 is a flowchart of an incremental data consumption queue according to an embodiment of the present invention, and as shown in fig. 4, the incremental data consumption queue in the embodiment of the present invention includes the following steps:
step S402, begins to insert the data in kafka (i.e., the data written to kafka from the source table) into the destination table.
Step S404, consuming the homonym topic record of the synchronization plan sequence and inserting the homonym topic record into the target table.
Step S406, judging whether the insertion process has a main key conflict abnormality, if so, executing step S408, and if not, indicating that the incremental data has no updated data, executing step S412.
In step S408, the record of the same primary key as the abnormal record in the target table is deleted.
And step S410, inserting an exception record to realize the synchronization of inserting and updating the incremental data.
In step S412, the insertion of the data in kafka into the target table is completed.
In the foregoing embodiment, when the source table has an increment sequence field or a timestamp field, and a target table with the same structure as that of the source table exists in the target database, the timestamp or the increment sequence of the latest record (corresponding to the latest update record) of the data queue is queried, which is used as the starting position of the increment extraction of the source table, where the data queue may be a message queue, or another queue such as a java memory. When a plurality of source tables exist, a synchronization plan sequence is generated for each source table, a sequence homonymy theme (corresponding to the preset topic) is created in a message queue, and incremental data synchronization from the plurality of source tables to the same destination table (corresponding to the target table) is realized. When the incremental data is updated, after the data is inserted, if the main key conflict abnormality exists, deleting the record of the target table and the abnormal record which are the same as the main key record, and then inserting the abnormal record, thereby realizing the purpose of inserting and updating the incremental data.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
In this embodiment, a data synchronization apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and the description already made is omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of a data synchronization apparatus according to an embodiment of the present invention, as shown in fig. 5, the apparatus including:
a query module 52, configured to query a latest update record of a predetermined topic in a predetermined message system, where the predetermined topic is used to record data in a source table;
a processing module 54, configured to filter incremental data in the source table based on the latest update record, and write the incremental data into the predetermined topic to obtain a target topic;
a synchronization module 56, configured to synchronize the incremental data recorded in the target topic into a target table.
In an alternative embodiment, the processing module 54 includes: a first processing unit, configured to determine, when it is determined that the latest update record is empty, full data of the source table as the incremental data, and write the incremental data into the predetermined topic to obtain the target topic; and a second processing unit, configured to, when it is determined that the latest update record is not empty, filter incremental data in the source table based on a target incremented sequence field value recorded in the latest update record, and write the incremental data into the predetermined topic to obtain the target topic.
In an alternative embodiment, the second processing unit may implement the screening of the delta data in the source table based on the target incremented sequence field value recorded in the latest update record by: determining data recorded in the source table after the target incremented sequence field value as delta data in the source table.
In an alternative embodiment, the synchronization module 56 may implement the synchronization of the incremental data recorded in the target topic into a target table by: inserting the data recorded in the target topic into the target table; under the condition that the conflict of the primary key exists, deleting the record corresponding to the conflicting primary key in the target table; reinserting data recorded in the target topic corresponding to the conflicting primary key in the target table.
In an alternative embodiment, the apparatus may be configured to create a synchronization task for the source table before querying a pre-created pre-defined message system for a latest update record of a pre-defined topic; allocating a unique sequence for the synchronization task; creating the predetermined topic in the predetermined message system with the same name as the sequence.
In an alternative embodiment, the predetermined message system includes kafka.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above-mentioned method embodiments when executed.
Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, inquiring the latest update record of the predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording the data in the source table;
s2, screening the incremental data in the source table based on the latest update record, and writing the incremental data into the predetermined topic to obtain a target topic;
s3, synchronizing the incremental data recorded in the target topic into a target table.
Optionally, in this embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, inquiring the latest update record of the predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording the data in the source table;
s2, screening the incremental data in the source table based on the latest update record, and writing the incremental data into the predetermined topic to obtain a target topic;
s3, synchronizing the incremental data recorded in the target topic into a target table.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for synchronizing data, comprising:
querying a latest update record of a predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording data in a source table;
screening incremental data in the source table based on the latest update record, and writing the incremental data into the predetermined topic to obtain a target topic;
synchronizing the delta data recorded in the target topic into a target table.
2. The method of claim 1, wherein filtering incremental data in the source table based on the latest update record, and writing incremental data into the predetermined topic to obtain a target topic comprises:
when the latest update record is determined to be empty, determining the full data of the source table as the incremental data, and writing the incremental data into the preset topic to obtain the target topic;
and screening incremental data in the source table based on a target incremental sequence field value recorded in the latest update record when the latest update record is determined not to be empty, and writing the incremental data into the predetermined topic to obtain the target topic.
3. The method of claim 2, wherein filtering delta data in the source table based on a target incremented sequence field value recorded in the most recent update record comprises:
determining data recorded in the source table after the target incremented sequence field value as delta data in the source table.
4. The method of claim 1, wherein synchronizing the delta data recorded in the target topic into a target table comprises:
inserting the data recorded in the target topic into the target table;
under the condition that the conflict of the primary key exists, deleting the record corresponding to the conflicting primary key in the target table;
reinserting data recorded in the target topic corresponding to the conflicting primary key in the target table.
5. The method of claim 1, wherein prior to querying a pre-established pre-defined messaging system for a most recent update record for a pre-defined topic, the method further comprises:
creating a synchronization task for the source table;
allocating a unique sequence for the synchronization task;
creating the predetermined topic in the predetermined message system with the same name as the sequence.
6. The method according to any one of claims 1 to 5, wherein the predetermined message system comprises kafka.
7. An apparatus for synchronizing data, comprising:
the query module is used for querying the latest update record of a predetermined topic in a predetermined message system, wherein the predetermined topic is used for recording data in a source table;
the processing module is used for screening the incremental data in the source table based on the latest update record and writing the incremental data into the preset topic to obtain a target topic;
and the synchronization module is used for synchronizing the incremental data recorded in the target topic into a target table.
8. The apparatus of claim 7, wherein the processing module comprises:
a first processing unit, configured to determine, when it is determined that the latest update record is empty, full data of the source table as the incremental data, and write the incremental data into the predetermined topic to obtain the target topic;
and a second processing unit, configured to, when it is determined that the latest update record is not empty, filter incremental data in the source table based on a target incremented sequence field value recorded in the latest update record, and write the incremental data into the predetermined topic to obtain the target topic.
9. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 6 when executed.
10. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010281618.5A CN111400407B (en) | 2020-04-10 | 2020-04-10 | Data synchronization method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010281618.5A CN111400407B (en) | 2020-04-10 | 2020-04-10 | Data synchronization method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400407A true CN111400407A (en) | 2020-07-10 |
| CN111400407B CN111400407B (en) | 2023-09-26 |
Family
ID=71433195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010281618.5A Active CN111400407B (en) | 2020-04-10 | 2020-04-10 | Data synchronization method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400407B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112182104A (en) * | 2020-09-25 | 2021-01-05 | 中国建设银行股份有限公司 | Data synchronization method, device, equipment and storage medium |
| CN112347103A (en) * | 2020-11-05 | 2021-02-09 | 深圳市极致科技股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112486986A (en) * | 2020-11-26 | 2021-03-12 | 清创网御(合肥)科技有限公司 | Automatic persistence method for consumption data of topic newly added in Kafka |
| CN112527879A (en) * | 2020-12-15 | 2021-03-19 | 中国人寿保险股份有限公司 | Kafka-based real-time data extraction method and related equipment |
| CN113742420A (en) * | 2021-08-09 | 2021-12-03 | 广州市易工品科技有限公司 | Data synchronization method and device |
| CN113961572A (en) * | 2021-12-23 | 2022-01-21 | 中电云数智科技有限公司 | Database synchronization method and synchronization device based on increment field |
| CN116089537A (en) * | 2023-04-07 | 2023-05-09 | 江西省智能产业技术创新研究院 | Incremental data synchronization method, system, computer and storage medium |
Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050091290A1 (en) * | 2003-10-23 | 2005-04-28 | Kim Cameron | Method and system for synchronizing identity information |
| US20130006950A1 (en) * | 2011-07-01 | 2013-01-03 | International Business Machines Corporation | Method for Attaching Partition ONLINE to Range Partitioned Table |
| US20150066942A1 (en) * | 2013-08-29 | 2015-03-05 | Oracle International Corporation | Unit of work based incremental data processing |
| CN104967639A (en) * | 2014-07-28 | 2015-10-07 | 浙江大华技术股份有限公司 | Information reporting method, information reporting apparatus and data node server |
| CN106339262A (en) * | 2016-08-26 | 2017-01-18 | 南京车易淘网络信息技术有限公司 | Data synchronization method in parallel calculation |
| WO2017084410A1 (en) * | 2015-11-17 | 2017-05-26 | 中兴通讯股份有限公司 | Network management data synchronization method and apparatus |
| CN107346314A (en) * | 2016-05-06 | 2017-11-14 | 银联国际有限公司 | A kind of database unidirectional synchronization method |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN107958082A (en) * | 2017-12-15 | 2018-04-24 | 杭州有赞科技有限公司 | Offline increment synchronization method and system of the database to data warehouse |
| CN109101599A (en) * | 2018-08-01 | 2018-12-28 | 北京奇艺世纪科技有限公司 | A kind of increment index update method and system |
| CN109739929A (en) * | 2018-12-18 | 2019-05-10 | 中国人民财产保险股份有限公司 | Method of data synchronization, apparatus and system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110196884A (en) * | 2019-05-31 | 2019-09-03 | 北京大米科技有限公司 | Method for writing data, storage medium and electronic equipment based on distributed data base |
| WO2019178979A1 (en) * | 2018-03-21 | 2019-09-26 | 平安科技（深圳）有限公司 | Method for querying report data, apparatus, storage medium and server |
| CN110321387A (en) * | 2019-07-10 | 2019-10-11 | 中国联合网络通信集团有限公司 | Method of data synchronization, equipment and terminal device |
| CN110334152A (en) * | 2019-06-28 | 2019-10-15 | 浙江吉利控股集团有限公司 | A kind of method of data synchronization, device and server |
| CN110674213A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data synchronization method and device |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| US20200073978A1 (en) * | 2018-08-31 | 2020-03-05 | International Business Machines Corporation | Processing event messages for changed data objects to determine changed data objects to backup |
-
2020
- 2020-04-10 CN CN202010281618.5A patent/CN111400407B/en active Active
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050091290A1 (en) * | 2003-10-23 | 2005-04-28 | Kim Cameron | Method and system for synchronizing identity information |
| US20130006950A1 (en) * | 2011-07-01 | 2013-01-03 | International Business Machines Corporation | Method for Attaching Partition ONLINE to Range Partitioned Table |
| US20150066942A1 (en) * | 2013-08-29 | 2015-03-05 | Oracle International Corporation | Unit of work based incremental data processing |
| CN104967639A (en) * | 2014-07-28 | 2015-10-07 | 浙江大华技术股份有限公司 | Information reporting method, information reporting apparatus and data node server |
| WO2017084410A1 (en) * | 2015-11-17 | 2017-05-26 | 中兴通讯股份有限公司 | Network management data synchronization method and apparatus |
| CN107346314A (en) * | 2016-05-06 | 2017-11-14 | 银联国际有限公司 | A kind of database unidirectional synchronization method |
| CN106339262A (en) * | 2016-08-26 | 2017-01-18 | 南京车易淘网络信息技术有限公司 | Data synchronization method in parallel calculation |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN107958082A (en) * | 2017-12-15 | 2018-04-24 | 杭州有赞科技有限公司 | Offline increment synchronization method and system of the database to data warehouse |
| WO2019178979A1 (en) * | 2018-03-21 | 2019-09-26 | 平安科技（深圳）有限公司 | Method for querying report data, apparatus, storage medium and server |
| CN109101599A (en) * | 2018-08-01 | 2018-12-28 | 北京奇艺世纪科技有限公司 | A kind of increment index update method and system |
| US20200073978A1 (en) * | 2018-08-31 | 2020-03-05 | International Business Machines Corporation | Processing event messages for changed data objects to determine changed data objects to backup |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN109739929A (en) * | 2018-12-18 | 2019-05-10 | 中国人民财产保险股份有限公司 | Method of data synchronization, apparatus and system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
| CN110196884A (en) * | 2019-05-31 | 2019-09-03 | 北京大米科技有限公司 | Method for writing data, storage medium and electronic equipment based on distributed data base |
| CN110334152A (en) * | 2019-06-28 | 2019-10-15 | 浙江吉利控股集团有限公司 | A kind of method of data synchronization, device and server |
| CN110321387A (en) * | 2019-07-10 | 2019-10-11 | 中国联合网络通信集团有限公司 | Method of data synchronization, equipment and terminal device |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN110674213A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data synchronization method and device |
Non-Patent Citations (2)
| Title |
|---|
| 李苒;段同乐;刘伯华;: "中间件技术在异构数据库同步中的应用", 计算机与网络, no. 09 * |
| 胡金龙;许卫;房福龙;李刚;曹晓宁;: "异构数据库同步复制技术研究与实现", 长江科学院院报, no. 06 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112182104A (en) * | 2020-09-25 | 2021-01-05 | 中国建设银行股份有限公司 | Data synchronization method, device, equipment and storage medium |
| CN112347103A (en) * | 2020-11-05 | 2021-02-09 | 深圳市极致科技股份有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112347103B (en) * | 2020-11-05 | 2024-04-12 | 深圳市极致科技股份有限公司 | Data synchronization method, device, electronic equipment and storage medium |
| CN112486986A (en) * | 2020-11-26 | 2021-03-12 | 清创网御(合肥)科技有限公司 | Automatic persistence method for consumption data of topic newly added in Kafka |
| CN112527879A (en) * | 2020-12-15 | 2021-03-19 | 中国人寿保险股份有限公司 | Kafka-based real-time data extraction method and related equipment |
| CN112527879B (en) * | 2020-12-15 | 2024-04-16 | 中国人寿保险股份有限公司 | Kafka-based real-time data extraction method and related equipment |
| CN113742420A (en) * | 2021-08-09 | 2021-12-03 | 广州市易工品科技有限公司 | Data synchronization method and device |
| CN113742420B (en) * | 2021-08-09 | 2024-02-02 | 广州市易工品科技有限公司 | Data synchronization method and device |
| CN113961572A (en) * | 2021-12-23 | 2022-01-21 | 中电云数智科技有限公司 | Database synchronization method and synchronization device based on increment field |
| CN116089537A (en) * | 2023-04-07 | 2023-05-09 | 江西省智能产业技术创新研究院 | Incremental data synchronization method, system, computer and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400407B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111400407B (en) | Data synchronization method and device, storage medium and electronic device | |
| CN108460115B (en) | Message pushing method and device, computer equipment and storage medium | |
| CN108460041B (en) | Data processing method and device | |
| EP3125501B1 (en) | File synchronization method, server, and terminal | |
| JP2007074724A (en) | Method and apparatus for synchronizing epg information between server and client in digital broadcasting network | |
| CN110532493B (en) | Data processing method and device, storage medium and electronic device | |
| CN108833584B (en) | Message pushing method, terminal, server and computer storage medium | |
| CN111367916B (en) | Data storage method and device | |
| CN110990381B (en) | Processing method and device of server, storage medium and electronic device | |
| CN110471896B (en) | Data processing method, system and server | |
| CN109165259B (en) | Index table updating method based on network attached storage, processor and storage device | |
| CN110502574B (en) | Cross-system information synchronization method, user equipment, storage medium and device | |
| CN110032586B (en) | Storage method, query method and acquisition and storage system for energy storage cell data | |
| CN115495527A (en) | Data synchronization management system and method | |
| CN107422991B (en) | Storage strategy management system | |
| CN113934742B (en) | Data updating method, node information storage method, electronic device and medium | |
| CN111209332B (en) | Database synchronization method based on homemade serialization algorithm | |
| CN110968646B (en) | Embedded system database synchronization method, device and storage medium | |
| CN111694887B (en) | Data adaptive storage scheduling system and method | |
| CN110647546A (en) | Third-party rule engine generation method and device | |
| CN113852919B (en) | Method and device for generating early warning message, storage medium and electronic device | |
| CN113434598B (en) | Method, device and electronic device for realizing data double writing | |
| CN117909414A (en) | Method, device, equipment and medium for realizing informix and MySQL database synchronization based on XXL-JOB | |
| CN117874033A (en) | Database sub-table method and device, electronic equipment and storage medium | |
| CN115550360A (en) | Database-based operation node management method, device, system and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |