CN111400407B - Data synchronization method and device, storage medium and electronic device - Google Patents
Data synchronization method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN111400407B CN111400407B CN202010281618.5A CN202010281618A CN111400407B CN 111400407 B CN111400407 B CN 111400407B CN 202010281618 A CN202010281618 A CN 202010281618A CN 111400407 B CN111400407 B CN 111400407B
- Authority
- CN
- China
- Prior art keywords
- data
- topic
- target
- source table
- increment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data synchronization method and device, a storage medium and an electronic device, wherein the method comprises the following steps: inquiring a latest updated record of a preset topic in a preset message system, wherein the preset topic is used for recording data in a source table; screening the increment data in the source table based on the latest updated record, and writing the increment data into the preset topic to obtain a target topic; and synchronizing the increment data recorded in the target topic into a target table. The application solves the problem that the data in the source table needs to be read for many times when the data in the related technology is synchronized, and causes pressure on the source table, thereby achieving the purposes of reducing the pressure of the source table and improving the data synchronization efficiency.
Description
Technical Field
The present application relates to the field of communications, and in particular, to a data synchronization method and apparatus, a storage medium, and an electronic device.
Background
In the related art, data in the same source table is synchronized to a plurality of target tables, and the data in the source table needs to be read many times, so that a certain pressure is caused on the source table. For example, when incremental data extraction is performed, a temporary table or snapshot table needs to be created for recording the flag bit of the current extraction. In performing incremental data synchronization from multiple tables to a single table, a flag bit record is created for each source table for each extraction, and therefore, the data in the source table needs to be read multiple times, and large storage resources are occupied. When a plurality of source data tables exist, the maximum value of the number of the data entry in the target data table may be derived from other source data tables, so that the incremental data is abnormal in synchronization, and a certain pressure is caused to the source tables.
As can be seen from the above, the related art has a problem that the data in the source table needs to be read multiple times during data synchronization, which causes stress to the source table.
In view of the above problems in the related art, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the application provides a data synchronization method and device, a storage medium and an electronic device, which at least solve the problem that the data in a source table needs to be read for a plurality of times when the data are synchronized in the related technology, and the pressure is caused to the source table.
According to an embodiment of the present application, there is provided a data synchronization method including: inquiring a latest updated record of a preset topic in a preset message system, wherein the preset topic is used for recording data in a source table; screening the increment data in the source table based on the latest updated record, and writing the increment data into the preset topic to obtain a target topic; and synchronizing the increment data recorded in the target topic into a target table.
According to another embodiment of the present application, there is provided a data synchronizing apparatus including: the query module is used for querying the latest updated record of a preset theme topic in a preset message system, wherein the preset topic is used for recording data in a source table; the processing module is used for screening the increment data in the source table based on the latest updated record, and writing the increment data into the preset topic to obtain a target topic; and the synchronization module is used for synchronizing the increment data recorded in the target topic into a target table.
According to a further embodiment of the application, there is also provided a computer readable storage medium having stored therein a computer program, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
According to a further embodiment of the application, there is also provided an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
According to the application, the latest update record of the scheduled topic is queried in the scheduled message system, the incremental data in the source table is screened according to the latest update record, the incremental data is written into the scheduled topic to obtain the target topic, and then the incremental data in the target topic is synchronized into the target table, so that the data in the source table and the target table are synchronized by utilizing the scheduled message system when the data is synchronized, and the data in the source table is not required to be read for multiple times, and therefore, the problem that the pressure is caused to the source table due to the fact that the data in the source table is required to be read for multiple times when the data is synchronized in the related technology can be solved, and the purposes of reducing the pressure of the source table and improving the data synchronization efficiency are achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a block diagram of a hardware structure of a mobile terminal of a data synchronization method according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of synchronizing data according to an embodiment of the application;
FIG. 3 is a flow chart of an incremental data production team according to an embodiment of the present application;
FIG. 4 is a flow chart of an incremental data consuming queue in accordance with a specific embodiment of the present application;
fig. 5 is a block diagram of a data synchronizing device according to an embodiment of the present application.
Detailed Description
The application will be described in detail hereinafter with reference to the drawings in conjunction with embodiments. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided by the application can be executed in a mobile terminal, a computer terminal or similar computing device. Taking the mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of a mobile terminal according to a data synchronization method according to an embodiment of the present application. As shown in fig. 1, the mobile terminal 10 may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, and optionally a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal 10 may also include more or fewer components than shown in FIG. 1 or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a method for synchronizing data in an embodiment of the present application, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, to implement the above-mentioned method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission means 106 is arranged to receive or transmit data via a network. The specific examples of networks described above may include wireless networks provided by the communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.
In this embodiment, a method for synchronizing data is provided, and fig. 2 is a flowchart of a method for synchronizing data according to an embodiment of the present application, as shown in fig. 2, where the flowchart includes the following steps:
step S202, inquiring a latest updated record of a preset theme topic in a preset message system, wherein the preset topic is used for recording data in a source table;
step S204, the incremental data in the source table is filtered based on the latest updated record, and the incremental data is written into the preset topic to obtain a target topic;
step S206, synchronizing the increment data recorded in the target topic into a target table.
Alternatively, the main body of execution of the above steps may be a processing terminal, or other devices with similar processing capabilities, for example, but not limited to, a server, etc.
In this embodiment, the source table includes an increment sequence field or a time stamp field, and the destination table and the source table may be identical in structure.
According to the application, the latest update record of the scheduled topic is queried in the scheduled message system, the incremental data in the source table is screened according to the latest update record, the incremental data is written into the scheduled topic to obtain the target topic, and then the incremental data in the target topic is synchronized into the target table, so that the data in the source table and the target table are synchronized by utilizing the scheduled message system when the data is synchronized, and the data in the source table is not required to be read for multiple times, and therefore, the problem that the pressure is caused to the source table due to the fact that the data in the source table is required to be read for multiple times when the data is synchronized in the related technology can be solved, and the purposes of reducing the pressure of the source table and improving the data synchronization efficiency are achieved.
In an alternative embodiment, filtering the incremental data in the source table based on the latest updated record, writing the incremental data into the predetermined topic to obtain a target topic includes: when the latest update record is determined to be empty, determining the full data of the source table as the increment data, and writing the increment data into the preset topic to obtain the target topic; and when the latest updated record is not empty, screening the increment data in the source table based on the target increment sequence field value recorded in the latest updated record, and writing the increment data into the preset topic to obtain the target topic. In this embodiment, when synchronizing data into a predetermined message system, all data in a source table needs to be synchronized into the predetermined message system first, and then when data update occurs in the source table, the data updated in the source table is written into the predetermined message system again (i.e. the content before update and the content after update are recorded in the predetermined message system, updated several times, and recorded several times). Therefore, when the synchronization is performed for the first time, the predetermined topic record is empty, the total data of the source table needs to be written into the predetermined topic, and in the subsequent synchronization, the incremental data in the source table needs to be determined, and the incremental data in the source table needs to be written into the predetermined topic of the synchronization plan sequence. The source table may be a message queue, a java memory, or other queues.
In an alternative embodiment, filtering the delta data in the source table based on the target delta sequence field value recorded in the latest update record includes: and determining the data recorded in the source table and positioned behind the target increment sequence field value as increment data in the source table. In this embodiment, the data in the source table is queried, and according to the latest queried record, the incremental sequence field value in the record is obtained, that is, the last synchronized end mark is used as the start mark for this extraction, and the record after the start mark of the source table is determined as incremental data. Wherein the target increment sequence field value may be identification information, field information, a time stamp, etc.
In an alternative embodiment, synchronizing the incremental data recorded in the target topic into a target table includes: inserting the data recorded in the target topic into the target table; deleting a record corresponding to the main key with conflict in the target table under the condition that the main key conflict exists; reinsertion of data corresponding to the primary key of the conflict recorded in the target topic in the target table. In this embodiment, if there is a primary key collision exception, the primary key record in the target table, which is the same as the exception record, is deleted, and then the exception record is inserted, so as to achieve the purpose of inserting and updating the incremental data. When synchronizing data in the target table, all data in the table corresponding to the source table stored in the predetermined message system are sequentially inserted into the target table, and a plurality of records with the same identifier in the predetermined message system (namely, records before and after updating when the source table is updated) are also sequentially inserted, however, since the primary key identifiers of the records before and after updating are the same, a problem of primary key conflict can occur when the records are sequentially inserted into the target table, in this case, contents under the same identifier inserted in the target table before need to be deleted, so that contents under the same identifier of the record at the next time can be successfully inserted into the target table, and incremental synchronization of the updated data is realized.
In an alternative embodiment, before querying the pre-created pre-defined message system for the latest update record of the pre-defined topic, the method further comprises: creating a synchronous task for the source table; assigning a unique sequence to the synchronization task; creating the predetermined topic having the same name as the sequence in the predetermined message system. In this embodiment, when incremental data synchronization is performed, there are multiple source tables, and scenes located in different databases, a synchronization task may be created for each source table, and a unique sequence may be generated, and a sequence common-name topic may be created in a predetermined message system by using the sequence, so as to improve synchronization efficiency. In addition, the predetermined topic created in the predetermined messaging system may also be named differently from the sequence, as long as the consumer is aware of synchronizing from a particular table. For example, a plurality of topics recorded by a plurality of tables are stored in a predetermined message system, and the contents of different topics need to be synchronized into different target tables, so that a consumer needs to know the specific topic name to be synchronized when performing data synchronization, and the synchronization is convenient, and therefore, only needs to know the name of the topic to be synchronized, namely, the synchronization to any one table.
In an alternative embodiment, the predetermined messaging system includes kafka. In this embodiment, all data in the source table may be recorded in kafka, and then synchronization may be performed using the data recorded in kafka as the target table, thereby reducing the pressure of the source table. It should be noted that kafka is only an alternative embodiment, and the present application is not limited to the predetermined messaging system.
How the synchronization of data is performed is described below in connection with the detailed description:
taking the incremental sequence field as an example for illustration, the synchronization task needs to configure a timing schedule including an incremental data production queue (writing data in the source table into kafka) and an incremental data consumption queue (inserting data in kafka, i.e., writing data of kafka from the source table into the target table). FIG. 3 is a flow chart of an incremental data production team according to an embodiment of the present application, as shown in FIG. 3, the flow chart of the incremental data production team in an embodiment of the present application includes the steps of:
step S302, writing of data in the source table into kafka is started.
Step S304, the latest record of topic (corresponding to the above-mentioned predetermined topic) named as the synchronization plan sequence is queried in kafka. The latest record may be identification information, field information, etc., and then specific content corresponding to the identification information, the field information needs to be found from the source table and written into kafka.
Step S306, judging whether the latest record exists, if not, executing step S308, and if yes, executing step S310.
In step S308, the full source table data is obtained and written into the kafka queue, i.e. when synchronizing for the first time, the topic record is empty, and the full source table data is searched and written into the topic with the same name as the synchronization planning sequence. Note that topic may be named differently from the synchronization plan sequence, as long as the consumer can learn to synchronize from a specific table (kafka stores multiple tables, and the contents of different tables may need to be synchronized to different target tables, so the consumer needs to know the specific table to be synchronized when synchronizing data).
Step S310, the incremental sequence field value in the record is obtained, the incremental data of the source table is screened and written into the kafka queue, namely, in the subsequent synchronization, the incremental sequence field value in the record is obtained according to the latest record inquired in step S304, namely, the last synchronous ending mark is used for the initial mark of the current extraction, the record after the initial mark of the source table is used as the incremental data and is written into the synchronous planning sequence identical-name topic.
In step S312, the data writing in the source table is completed in kafka.
FIG. 4 is a flow chart of an incremental data consuming queue according to an embodiment of the application, as shown in FIG. 4, wherein the flow chart of the incremental data consuming queue in an embodiment of the application comprises the steps of:
step S402, inserting the data in kafka (i.e., the data written in kafka from the source table) into the target table is started.
Step S404, consuming synchronous planning sequence homologous topic record and inserting the record into the target table.
Step S406, judging whether the insertion process has the main key conflict abnormality, if yes, executing step S408, if no, executing step S412.
In step S408, the primary key record in the target table is deleted together with the abnormal record.
Step S410, inserting abnormal records to realize synchronization of inserting and updating incremental data.
The data insertion in the kafka is ended in the target table at step S412.
In the foregoing embodiment, when the source table has an increment sequence field or a timestamp field, and the target database has a target table with the same structure as the source table, the timestamp or the increment sequence of the latest record of the data queue (corresponding to the latest record) is queried for the starting position of source table increment extraction, where the data queue may be a message queue or other queues such as java memory. When a plurality of source tables exist, a synchronization plan sequence is generated for each source table, a sequence homonymy theme (corresponding to the preset topic) is created in a message queue, and incremental data synchronization from the plurality of source tables to the same destination table (corresponding to the target table) is realized. When the incremental data is updated, after the data is inserted, if the main key conflict abnormality exists, deleting the main key record which is the same as the abnormal record in the target table, and then inserting the abnormal record, thereby realizing the purposes of inserting and updating the incremental data.
From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present application.
The embodiment also provides a data synchronization device, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 5 is a block diagram of a data synchronization apparatus according to an embodiment of the present application, as shown in fig. 5, including:
a query module 52, configured to query a predetermined message system for a latest update record of a predetermined topic, where the predetermined topic is used for recording data in a source table;
a processing module 54, configured to filter the incremental data in the source table based on the latest update record, and write the incremental data into the predetermined topic to obtain a target topic;
a synchronization module 56, configured to synchronize the incremental data recorded in the target topic into a target table.
In an alternative embodiment, the processing module 54 includes: the first processing unit is used for determining the full data of the source table as the increment data when the latest update record is determined to be empty, and writing the increment data into the preset topic to obtain the target topic; and the second processing unit is used for screening the increment data in the source table based on the target increment sequence field value recorded in the latest updated record when the latest updated record is not empty, and writing the increment data into the preset topic to obtain the target topic.
In an alternative embodiment, the second processing unit may implement filtering the delta data in the source table based on the target incremental sequence field value recorded in the latest update record by: and determining the data recorded in the source table and positioned behind the target increment sequence field value as increment data in the source table.
In an alternative embodiment, the synchronization module 56 may implement the synchronization of the incremental data recorded in the target topic into a target table by: inserting the data recorded in the target topic into the target table; deleting a record corresponding to the main key with conflict in the target table under the condition that the main key conflict exists; reinsertion of data corresponding to the primary key of the conflict recorded in the target topic in the target table.
In an alternative embodiment, the apparatus may be configured to create a synchronization task for the source table before querying a pre-created pre-determined messaging system for a newly updated record of a pre-determined topic; assigning a unique sequence to the synchronization task; creating the predetermined topic having the same name as the sequence in the predetermined message system.
In an alternative embodiment, the predetermined messaging system includes kafka.
It should be noted that each of the above modules may be implemented by software or hardware, and for the latter, it may be implemented by, but not limited to: the modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.
Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
Alternatively, in the present embodiment, the above-described computer-readable storage medium may be configured to store a computer program for performing the steps of:
s1, inquiring a latest updated record of a preset theme topic in a preset message system, wherein the preset topic is used for recording data in a source table;
s2, screening the increment data in the source table based on the latest updated record, and writing the increment data into the preset topic to obtain a target topic;
and S3, synchronizing the increment data recorded in the target topic into a target table.
Alternatively, in the present embodiment, the above-described computer-readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.
An embodiment of the application also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input/output device is connected to the processor.
Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:
s1, inquiring a latest updated record of a preset theme topic in a preset message system, wherein the preset topic is used for recording data in a source table;
s2, screening the increment data in the source table based on the latest updated record, and writing the increment data into the preset topic to obtain a target topic;
and S3, synchronizing the increment data recorded in the target topic into a target table.
Alternatively, specific examples in this embodiment may refer to examples described in the foregoing embodiments and optional implementations, and this embodiment is not described herein.
It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a memory device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps within them may be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application should be included in the protection scope of the present application.
Claims (8)
1. A method for synchronizing data, comprising:
inquiring a latest updated record of a preset topic in a preset message system, wherein the preset topic is used for recording data in a source table;
screening the increment data in the source table based on the latest updated record, and writing the increment data into the preset theme topic to obtain a target topic;
synchronizing the incremental data recorded in the target topic into a target table;
wherein filtering the incremental data in the source table based on the latest updated record, and writing the incremental data into the predetermined topic to obtain a target topic includes:
when the latest update record is determined to be empty, determining the full data of the source table as the increment data, and writing the increment data into the preset theme topic to obtain the target topic;
and when the latest updated record is not empty, screening the increment data in the source table based on the target increment sequence field value recorded in the latest updated record, and writing the increment data into the preset theme topic to obtain the target topic.
2. The method of claim 1, wherein filtering delta data in the source table based on a target delta sequence field value recorded in the most recent update record comprises:
and determining the data recorded in the source table and positioned behind the target increment sequence field value as increment data in the source table.
3. The method of claim 1, wherein synchronizing the delta data recorded in the target topic into a target table comprises:
inserting the data recorded in the target topic into the target table;
deleting a record corresponding to the main key with conflict in the target table under the condition that the main key conflict exists;
reinsertion of data corresponding to the primary key of the conflict recorded in the target topic in the target table.
4. The method of claim 1, wherein prior to querying a pre-created pre-determined messaging system for a newly updated record of a pre-determined topic, the method further comprises:
creating a synchronous task for the source table;
assigning a unique sequence to the synchronization task;
creating the predetermined topic having the same name as the sequence in the predetermined message system.
5. The method according to any one of claims 1 to 4, wherein the predetermined messaging system comprises kafka.
6. A data synchronizing device, comprising:
the query module is used for querying the latest updated record of a preset theme topic in a preset message system, wherein the preset theme topic is used for recording data in a source table;
the processing module is used for screening the increment data in the source table based on the latest updating record, and writing the increment data into the preset theme topic to obtain a target topic;
the synchronization module is used for synchronizing the increment data recorded in the target topic into a target table;
wherein the processing module comprises:
the first processing unit is used for determining the full data of the source table as the increment data when the latest update record is determined to be empty, and writing the increment data into the preset theme topic to obtain the target topic;
and the second processing unit is used for screening the increment data in the source table based on the target increment sequence field value recorded in the latest update record when the latest update record is not empty, and writing the increment data into the preset theme topic to obtain the target topic.
7. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program, wherein the computer program is arranged to execute the method of any of the claims 1 to 5 when run.
8. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the method of any of the claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010281618.5A CN111400407B (en) | 2020-04-10 | 2020-04-10 | Data synchronization method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010281618.5A CN111400407B (en) | 2020-04-10 | 2020-04-10 | Data synchronization method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400407A CN111400407A (en) | 2020-07-10 |
| CN111400407B true CN111400407B (en) | 2023-09-26 |
Family
ID=71433195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010281618.5A Active CN111400407B (en) | 2020-04-10 | 2020-04-10 | Data synchronization method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400407B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112182104A (en) * | 2020-09-25 | 2021-01-05 | 中国建设银行股份有限公司 | Data synchronization method, device, equipment and storage medium |
| CN112347103B (en) * | 2020-11-05 | 2024-04-12 | 深圳市极致科技股份有限公司 | Data synchronization method, device, electronic equipment and storage medium |
| CN112486986A (en) * | 2020-11-26 | 2021-03-12 | 清创网御(合肥)科技有限公司 | Automatic persistence method for consumption data of topic newly added in Kafka |
| CN112527879B (en) * | 2020-12-15 | 2024-04-16 | 中国人寿保险股份有限公司 | Kafka-based real-time data extraction method and related equipment |
| CN113742420B (en) * | 2021-08-09 | 2024-02-02 | 广州市易工品科技有限公司 | Data synchronization method and device |
| CN113961572A (en) * | 2021-12-23 | 2022-01-21 | 中电云数智科技有限公司 | Database synchronization method and synchronization device based on increment field |
| CN116089537B (en) * | 2023-04-07 | 2023-08-04 | 江西省智能产业技术创新研究院 | Incremental data synchronization method, system, computer and storage medium |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967639A (en) * | 2014-07-28 | 2015-10-07 | 浙江大华技术股份有限公司 | Information reporting method, information reporting apparatus and data node server |
| CN106339262A (en) * | 2016-08-26 | 2017-01-18 | 南京车易淘网络信息技术有限公司 | Data synchronization method in parallel calculation |
| WO2017084410A1 (en) * | 2015-11-17 | 2017-05-26 | 中兴通讯股份有限公司 | Network management data synchronization method and apparatus |
| CN107346314A (en) * | 2016-05-06 | 2017-11-14 | 银联国际有限公司 | A kind of database unidirectional synchronization method |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN107958082A (en) * | 2017-12-15 | 2018-04-24 | 杭州有赞科技有限公司 | Offline increment synchronization method and system of the database to data warehouse |
| CN109101599A (en) * | 2018-08-01 | 2018-12-28 | 北京奇艺世纪科技有限公司 | A kind of increment index update method and system |
| CN109739929A (en) * | 2018-12-18 | 2019-05-10 | 中国人民财产保险股份有限公司 | Method of data synchronization, apparatus and system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110196884A (en) * | 2019-05-31 | 2019-09-03 | 北京大米科技有限公司 | Method for writing data, storage medium and electronic equipment based on distributed data base |
| WO2019178979A1 (en) * | 2018-03-21 | 2019-09-26 | 平安科技（深圳）有限公司 | Method for querying report data, apparatus, storage medium and server |
| CN110321387A (en) * | 2019-07-10 | 2019-10-11 | 中国联合网络通信集团有限公司 | Method of data synchronization, equipment and terminal device |
| CN110334152A (en) * | 2019-06-28 | 2019-10-15 | 浙江吉利控股集团有限公司 | A kind of method of data synchronization, device and server |
| CN110674213A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data synchronization method and device |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7181472B2 (en) * | 2003-10-23 | 2007-02-20 | Microsoft Corporation | Method and system for synchronizing identity information |
| US8615503B2 (en) * | 2011-07-01 | 2013-12-24 | International Business Machines Corporation | Method for attaching partition online to range partitioned table |
| US9454557B2 (en) * | 2013-08-29 | 2016-09-27 | Oracle International Corporation | Unit of work based incremental data processing |
| US11042532B2 (en) * | 2018-08-31 | 2021-06-22 | International Business Machines Corporation | Processing event messages for changed data objects to determine changed data objects to backup |
-
2020
- 2020-04-10 CN CN202010281618.5A patent/CN111400407B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967639A (en) * | 2014-07-28 | 2015-10-07 | 浙江大华技术股份有限公司 | Information reporting method, information reporting apparatus and data node server |
| WO2017084410A1 (en) * | 2015-11-17 | 2017-05-26 | 中兴通讯股份有限公司 | Network management data synchronization method and apparatus |
| CN107346314A (en) * | 2016-05-06 | 2017-11-14 | 银联国际有限公司 | A kind of database unidirectional synchronization method |
| CN106339262A (en) * | 2016-08-26 | 2017-01-18 | 南京车易淘网络信息技术有限公司 | Data synchronization method in parallel calculation |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN107958082A (en) * | 2017-12-15 | 2018-04-24 | 杭州有赞科技有限公司 | Offline increment synchronization method and system of the database to data warehouse |
| WO2019178979A1 (en) * | 2018-03-21 | 2019-09-26 | 平安科技（深圳）有限公司 | Method for querying report data, apparatus, storage medium and server |
| CN109101599A (en) * | 2018-08-01 | 2018-12-28 | 北京奇艺世纪科技有限公司 | A kind of increment index update method and system |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN109739929A (en) * | 2018-12-18 | 2019-05-10 | 中国人民财产保险股份有限公司 | Method of data synchronization, apparatus and system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
| CN110196884A (en) * | 2019-05-31 | 2019-09-03 | 北京大米科技有限公司 | Method for writing data, storage medium and electronic equipment based on distributed data base |
| CN110334152A (en) * | 2019-06-28 | 2019-10-15 | 浙江吉利控股集团有限公司 | A kind of method of data synchronization, device and server |
| CN110321387A (en) * | 2019-07-10 | 2019-10-11 | 中国联合网络通信集团有限公司 | Method of data synchronization, equipment and terminal device |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN110674213A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data synchronization method and device |
Non-Patent Citations (2)
| Title |
|---|
| 中间件技术在异构数据库同步中的应用;李苒;段同乐;刘伯华;;计算机与网络(第09期);全文 * |
| 异构数据库同步复制技术研究与实现;胡金龙;许卫;房福龙;李刚;曹晓宁;;长江科学院院报(第06期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400407A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111400407B (en) | Data synchronization method and device, storage medium and electronic device | |
| CN107332876B (en) | Method and device for synchronizing block chain state | |
| EP2328302B1 (en) | A method and device for maintaining a changelog in data synchronization | |
| EP3125501A1 (en) | File synchronization method, server, and terminal | |
| JP2007074724A (en) | Method and apparatus for synchronizing epg information between server and client in digital broadcasting network | |
| EP2741217A1 (en) | Database synchronization | |
| CN106709066B (en) | Data synchronization method and device | |
| CN102202087A (en) | Method for identifying storage equipment and system thereof | |
| CN102024040A (en) | Database synchronization method, device and system | |
| CN112069195A (en) | Database-based message transmission method and device, electronic equipment and storage medium | |
| CN111367916B (en) | Data storage method and device | |
| CN111209332B (en) | Database synchronization method based on homemade serialization algorithm | |
| CN110968646B (en) | Embedded system database synchronization method, device and storage medium | |
| CN109165259B (en) | Index table updating method based on network attached storage, processor and storage device | |
| CN110032586B (en) | Storage method, query method and acquisition and storage system for energy storage cell data | |
| CN115495527A (en) | Data synchronization management system and method | |
| CN113934742B (en) | Data updating method, node information storage method, electronic device and medium | |
| CN111143899A (en) | Method and device for storing data, storage medium and electronic device | |
| CN113918436A (en) | Log processing method and device | |
| CN111694887B (en) | Data adaptive storage scheduling system and method | |
| CN110647546A (en) | Third-party rule engine generation method and device | |
| CN110362706B (en) | Data searching method and device, storage medium and electronic device | |
| CN112035169A (en) | Skip processing method and device, computer equipment and computer readable storage medium | |
| CN111506646A (en) | Data synchronization method, device, system, storage medium and processor | |
| CN113434598B (en) | Method, device and electronic device for realizing data double writing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |