CN111651519B - Data synchronization method, data synchronization device, electronic equipment and storage medium - Google Patents
Data synchronization method, data synchronization device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111651519B CN111651519B CN202010382046.XA CN202010382046A CN111651519B CN 111651519 B CN111651519 B CN 111651519B CN 202010382046 A CN202010382046 A CN 202010382046A CN 111651519 B CN111651519 B CN 111651519B
- Authority
- CN
- China
- Prior art keywords
- file
- data
- full
- primary key
- changed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Abstract
The invention discloses a data synchronization method, a data synchronization device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring changed and newly added data from a data table of a source database, and acquiring full-quantity primary key field data from an index block of the data table; synchronizing the full-quantity main key field data and the changed and newly added data to a target database to respectively obtain a main key file and a first increment file; deleting a second increment file in the first history file to obtain a second history file; combining the first increment file and the second history file to obtain a temporary file; and combining the primary key file and the temporary file to obtain the full-quantity file. The invention synchronizes the data of the full-quantity main key field in the index block of one data table by the full-quantity method, ensures the consistent data quantity in the upstream and downstream system databases, simultaneously synchronizes the changed and newly added data in the data table of the source database by the increment method, reduces the influence on the performance of the upstream system and improves the timeliness of the data.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data synchronization method, a data synchronization device, an electronic device, and a storage medium.
Background
At present, when data is deleted at the upstream of a database, a full-volume synchronous data method is generally used for keeping the data volume of the upstream and the downstream consistent, but if the data volume becomes large, the performance of an upstream system is affected, and meanwhile, the timeliness of the data cannot be guaranteed; the other method is an incremental data synchronization method, the incremental data and the total data of the previous day are combined to obtain the data of the current day, the influence on an online system is reduced, the timeliness is improved, but the phenomenon that the downstream data is more than the upstream data can occur, and the consistency of the upstream data and the downstream data cannot be ensured.
Disclosure of Invention
The invention aims to overcome the defects that the performance of an upstream system is affected, the timeliness of data is reduced and the data volume of the upstream and downstream is inconsistent when a full-volume synchronization or incremental synchronization method is used in the prior art, and provides a data synchronization method, a data synchronization device, electronic equipment and a storage medium.
The invention solves the technical problems by the following technical scheme:
the first aspect of the present invention provides a data synchronization method, comprising the steps of:
acquiring changed and newly added data from a data table of a source database, and acquiring full-quantity primary key field data from an index block of the data table;
synchronizing the full-quantity primary key field data to a target database to obtain a primary key file, and synchronizing the changed and newly added data to the target database to obtain a first incremental file;
deleting a second increment file in a first history file to obtain a second history file, wherein the first history file is a history data file closest to the current time, and the second increment file is a data file changed and added in the first history file;
combining the first increment file and the second history file to obtain a temporary file;
and combining the primary key file and the temporary file in an equivalent way to obtain a full-quantity file.
Preferably, the step of obtaining the changed and newly added data from a data table of the source database and obtaining the full-scale primary key field data from an index block of the data table includes:
and extracting the changed and newly added data from a data table of the source database according to the timestamp field, and directly reading the full-quantity primary key field data from an index block of the data table.
Preferably, the step of synchronizing the full-size primary key field data to a target database to obtain a primary key file, and synchronizing the changed and newly added data to the target database to obtain a first incremental file includes:
and synchronizing the data of the primary key field by using a full-scale method to obtain the primary key file, and synchronizing the changed and newly added data by using an increment method to obtain the first increment file.
Preferably, the operation of acquiring the changed and newly added data from one data table of the source database and acquiring the full-quantity primary key field data from the index block of the data table are simultaneously executed within a preset time period;
and/or the number of the groups of groups,
the first history file is a previous day file, and the full-quantity file is a current day full-quantity file.
The second aspect of the invention provides a data synchronization device, which comprises an acquisition module, a synchronization module, a deletion module, a combination module and an equivalent combination module;
the acquisition module is used for acquiring changed and newly added data from a data table of the source database and acquiring full-quantity primary key field data from an index block of the data table;
the synchronization module is used for synchronizing the full-quantity primary key field data to a target database to obtain a primary key file, and synchronizing the changed and newly added data to the target database to obtain a first incremental file;
the deleting module is used for deleting a second increment file in the first history file to obtain a second history file, the first history file is a history data file closest to the current time, and the second increment file is a data file changed and added in the first history file;
the merging module is used for merging the first increment file and the second history file to obtain a temporary file;
and the equivalent combination module is used for equivalent combination of the primary key file and the temporary file to obtain a full-quantity file.
Preferably, the acquisition module comprises an extraction unit and a reading unit;
the extraction unit is used for extracting changed and newly added data from a data table of the source database according to the timestamp field and calling the reading unit;
the reading unit is used for directly reading the full-quantity main key field data from the index block of the data table.
Preferably, the synchronization module comprises a full synchronization unit and an incremental synchronization unit;
the full synchronization unit is used for synchronizing the primary key field data by using a full method to obtain the primary key file and calling the increment synchronization unit;
the increment synchronization unit is used for synchronizing the change and the newly added data by an increment method to obtain the first increment file.
Preferably, the obtaining module simultaneously performs the operations of obtaining changed and newly added data from one data table of the source database and obtaining full-scale primary key field data from an index block of the data table in a preset time period;
and/or the number of the groups of groups,
the first history file is a previous day file, and the full-quantity file is a current day full-quantity file.
A third aspect of the invention provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the data synchronization method according to the first aspect when executing the computer program.
A fourth aspect of the invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data synchronization method according to the first aspect.
The invention has the positive progress effects that:
the data synchronization method, the data synchronization device, the electronic equipment and the storage medium disclosed by the invention ensure the consistent data volume in the upstream and downstream system databases by synchronizing the full-volume primary key field data in the index block of one data table by using the full-volume method, and simultaneously synchronize the changed and newly added data in the data table of the source database by using the increment method, thereby reducing the influence on the performance of the upstream system and improving the timeliness of the data.
Drawings
Fig. 1 is a flowchart of a data synchronization method according to embodiment 1 of the present invention.
Fig. 2 is a schematic block diagram of a data synchronization device according to embodiment 2 of the present invention.
Fig. 3 is a schematic diagram of an acquisition module of a data synchronization device according to embodiment 2 of the present invention.
Fig. 4 is a schematic diagram of a synchronization module of the data synchronization device in embodiment 2 of the present invention.
Fig. 5 is a schematic structural diagram of an electronic device according to embodiment 3 of the present invention.
Detailed Description
The invention is further illustrated by means of the following examples, which are not intended to limit the scope of the invention.
Example 1
As shown in fig. 1, the present embodiment provides a data synchronization method, which includes the following steps:
In this embodiment, the changed and newly added data is extracted from a data table of the source database according to the timestamp field, and the full-scale primary key field data is directly read from the index block of the data table.
In this embodiment, the method uses a full-scale method to synchronize full-scale primary key field data read directly from an index block of a data table with a target database to obtain a primary key file, and uses an incremental method to synchronize data extracted from the data table of a source database and added to the target database to obtain a first incremental file, that is, the primary key file is the full-scale primary key field data file read from the index block of the data table, and the first incremental file is the data file extracted from the data table of the source database and added.
In this embodiment, the second history file is a data file obtained by deleting a changed and newly added data file in the history data file closest to the current time.
And 105, equivalent joint primary key files and temporary files to obtain a full-volume file.
In this embodiment, the source database is a relational database, the relational database includes a data table and an index block corresponding to the data table, and the target database is a distributed database system, where a plurality of data block files are stored in the distributed database system.
In this embodiment, the first history file is a previous day file, and the full-size file is a current day full-size file.
In this embodiment, the operations of acquiring the changed and newly added data from one data table of the source database and acquiring the full-size primary key field data from the index block of the data table are simultaneously performed every day within a preset period of time. The method does not need to access the data block, has good performance advantages, does not influence the performance of the source system, and simultaneously avoids the performance problem of the source system caused by the whole data extraction.
For example, after deleting the data of the second incremental file in the first historical file (i.e. the previous day file), a second historical file is obtained, and then the second historical file is combined with the first incremental file to obtain a temporary file, where the step from the data synchronization method to the data combination in the prior art is finished, but there is a disadvantage that if the data in the upstream system database is deleted, the deleted data will exist in the downstream system database all the time, so that the data amount in the upstream system database and the downstream system database is inconsistent. In this embodiment, the temporary file obtained by merging is equivalent combined with the primary key file to obtain the full-quantity file of the same day, so that the deleted second incremental file data in the first historical file is removed, and the consistency of the data quantity in the upstream and downstream system databases is ensured.
For another example, when the amount of data in the database is relatively large, the time to acquire the data becomes long, and the upstream system database is occupied for a long time, and the amount of calculation of the downstream system data also increases. In this embodiment, the temporary file and the primary key file are equivalently combined, so that the deleted second incremental file data in the first history file is removed. Although the data volume of the primary key file is larger, only one field exists, so that the performance bottleneck of sql (structured query language) cannot occur when equivalent combination is performed, the influence on the performance of an upstream system is reduced, and the timeliness of the data is improved.
According to the embodiment, the full-quantity main key field data in the index block of one data table is synchronized by using the full-quantity method, so that the data quantity in the upstream and downstream system databases is consistent, meanwhile, the change and newly added data in the data table of the source database are synchronized by using the increment method, the influence on the performance of the upstream system is reduced, and the timeliness of the data is improved.
Example 2
As shown in fig. 2, the embodiment provides a data synchronization device, which includes an acquisition module 1, a synchronization module 2, a deletion module 3, a merging module 4, and an equivalent joint module 5.
The acquisition module 1 is used for acquiring changed and newly added data from a data table of the source database, and acquiring the full-quantity primary key field data from an index block of the data table.
As shown in fig. 3, the acquisition module 1 includes an extraction unit 11 and a reading unit 12.
The extracting unit 11 is configured to extract changed and newly added data from a data table in the source database according to the timestamp field, and call the reading unit 12;
the reading unit 12 is configured to read the full amount of primary key field data directly from the index block of the data table.
The synchronization module 2 is configured to synchronize the full amount of primary key field data to the target database to obtain a primary key file, and synchronize the changed and newly added data to the target database to obtain a first incremental file.
As shown in fig. 4, the synchronization module 2 includes a full synchronization unit 21 and an incremental synchronization unit 22.
The full-size synchronization unit 21 is configured to synchronize full-size primary key field data read directly from an index block of a data table with a full-size method to a target database, obtain a primary key file, and call the delta synchronization unit 22.
The delta synchronizing unit 22 is configured to synchronize, in a delta manner, the data extracted from the data table of the source database and the new data to the target database, to obtain a first delta file.
In this embodiment, the primary key file is a full primary key field data file read from an index block of a data table, and the first delta file is a changed and newly added data file extracted from the data table of the source database.
The deleting module 3 is configured to delete a second incremental file in the first history file to obtain a second history file, where the first history file is a history data file closest to the current time, and the second incremental file is a changed and newly added data file in the first history file.
In this embodiment, the second history file is a data file obtained by deleting a changed and newly added data file in the history data file closest to the current time.
The merging module 4 is configured to merge the first incremental file and the second historical file to obtain a temporary file.
The equivalent joint module 5 is used for equivalent joint of the primary key file and the temporary file to obtain the full-volume file.
In this embodiment, the source database is a relational database, the relational database includes a data table and an index block corresponding to the data table, and the target database is a distributed database system, where a plurality of data block files are stored in the distributed database system.
In this embodiment, the first history file is a previous day file, and the full-size file is a current day full-size file.
In this embodiment, the operations of acquiring the changed and newly added data from one data table of the source database and acquiring the full-size primary key field data from the index block of the data table are simultaneously performed every day within a preset period of time. The method does not need to access the data block, has good performance advantages, does not influence the performance of the source system, and simultaneously avoids the performance problem of the source system caused by the whole data extraction.
For example, after deleting the data of the second incremental file in the first historical file (i.e. the previous day file), a second historical file is obtained, and then the second historical file is combined with the first incremental file to obtain a temporary file, where the step from the data synchronization method to the data combination in the prior art is finished, but there is a disadvantage that if the data in the upstream system database is deleted, the deleted data will exist in the downstream system database all the time, so that the data amount in the upstream system database and the downstream system database is inconsistent. In this embodiment, the temporary file obtained by merging is equivalent combined with the primary key file to obtain the full-quantity file of the same day, so that the deleted second incremental file data in the first historical file is removed, and the consistency of the data quantity in the upstream and downstream system databases is ensured.
For another example, when the amount of data in the database is relatively large, the time to acquire the data becomes long, and the upstream system database is occupied for a long time, and the amount of calculation of the downstream system data also increases. In this embodiment, the temporary file and the primary key file are equivalently combined, so that the deleted second incremental file data in the first history file is removed. Although the data volume of the primary key file is larger, only one field exists, so that the performance bottleneck of sql (structured query language) cannot occur when equivalent combination is performed, the influence on the performance of an upstream system is reduced, and the timeliness of the data is improved.
According to the embodiment, the full-quantity main key field data in the index block of one data table is synchronized by using the full-quantity method, so that the data quantity in the upstream and downstream system databases is consistent, meanwhile, the change and newly added data in the data table of the source database are synchronized by using the increment method, the influence on the performance of the upstream system is reduced, and the timeliness of the data is improved.
Example 3
Fig. 5 is a schematic structural diagram of an electronic device according to embodiment 3 of the present invention. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the data synchronization method of embodiment 1 when executing the computer program. The electronic device 30 shown in fig. 5 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 5, the electronic device 30 may be embodied in the form of a general purpose computing device, which may be a server device, for example. Components of electronic device 30 may include, but are not limited to: the at least one processor 31, the at least one memory 32, a bus 33 connecting the different system components, including the memory 32 and the processor 31.
The bus 33 includes a data bus, an address bus, and a control bus.
The processor 31 executes various functional applications and data processing such as the data synchronization method provided in embodiment 1 of the present invention by running a computer program stored in the memory 32.
The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). Such communication may be through an input/output (I/O) interface 35. Also, the resulting device 30 may also communicate with one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through a network adapter 36. As shown in fig. 5, network adapter 36 communicates with the other modules of model-generating device 30 via bus 33. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in connection with the model-generating device 30, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, data backup storage systems, and the like.
It should be noted that although several units/modules or sub-units/modules of an electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present invention. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Example 4
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data synchronization method provided by embodiment 1.
More specifically, among others, readable storage media may be employed including, but not limited to: portable disk, hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible embodiment, the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps of implementing the data synchronization method as described in embodiment 1, when said program product is run on the terminal device.
Wherein the program code for carrying out the invention may be written in any combination of one or more programming languages, which program code may execute entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on the remote device or entirely on the remote device.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the principles and spirit of the invention, but such changes and modifications fall within the scope of the invention.
Claims (10)
1. A method of data synchronization comprising the steps of:
acquiring changed and newly added data from a data table of a source database, and acquiring full-quantity primary key field data from an index block of the data table;
synchronizing the full-quantity primary key field data to a target database to obtain a primary key file, and synchronizing the changed and newly added data to the target database to obtain a first incremental file;
deleting a second increment file in a first history file to obtain a second history file, wherein the first history file is a history data file closest to the current time, and the second increment file is a data file changed and added in the first history file;
combining the first increment file and the second history file to obtain a temporary file;
and combining the primary key file and the temporary file in an equivalent way to obtain a full-quantity file.
2. The data synchronization method of claim 1, wherein the steps of obtaining changed and newly added data from a data table of the source database and obtaining full-size primary key field data from an index block of the data table include:
and extracting the changed and newly added data from a data table of the source database according to the timestamp field, and directly reading the full-quantity primary key field data from an index block of the data table.
3. The method of claim 1, wherein the step of synchronizing the full primary key field data to a target database to obtain a primary key file, and synchronizing the changed and newly added data to the target database to obtain a first delta file comprises:
and synchronizing the data of the primary key field by using a full-scale method to obtain the primary key file, and synchronizing the changed and newly added data by using an increment method to obtain the first increment file.
4. The data synchronization method of claim 1, wherein the operations of acquiring changed and newly added data from one data table of the source database and acquiring full amount of primary key field data from an index block of the data table are simultaneously performed within a preset period of time;
and/or the number of the groups of groups,
the first history file is a previous day file, and the full-quantity file is a current day full-quantity file.
5. The data synchronization device is characterized by comprising an acquisition module, a synchronization module, a deletion module, a combination module and an equivalent combination module;
the acquisition module is used for acquiring changed and newly added data from a data table of the source database and acquiring full-quantity primary key field data from an index block of the data table;
the synchronization module is used for synchronizing the full-quantity primary key field data to a target database to obtain a primary key file, and synchronizing the changed and newly added data to the target database to obtain a first incremental file;
the deleting module is used for deleting a second increment file in the first history file to obtain a second history file, the first history file is a history data file closest to the current time, and the second increment file is a data file changed and added in the first history file;
the merging module is used for merging the first increment file and the second history file to obtain a temporary file;
and the equivalent combination module is used for equivalent combination of the primary key file and the temporary file to obtain a full-quantity file.
6. The data synchronization device of claim 5, wherein the acquisition module comprises an extraction unit and a reading unit;
the extraction unit is used for extracting changed and newly added data from a data table of the source database according to the timestamp field and calling the reading unit;
the reading unit is used for directly reading the full-quantity main key field data from the index block of the data table.
7. The data synchronization device of claim 5, wherein the synchronization module comprises a full synchronization unit and an incremental synchronization unit;
the full synchronization unit is used for synchronizing the primary key field data by using a full method to obtain the primary key file and calling the increment synchronization unit;
the increment synchronization unit is used for synchronizing the change and the newly added data by an increment method to obtain the first increment file.
8. The data synchronization device according to claim 5, wherein the acquisition module simultaneously performs the operations of acquiring changed and newly added data from one data table of the source database and acquiring full-size primary key field data from an index block of the data table within a preset period of time;
and/or the number of the groups of groups,
the first history file is a previous day file, and the full-quantity file is a current day full-quantity file.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data synchronization method of any of claims 1-4 when the computer program is executed by the processor.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, carries out the steps of the data synchronization method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010382046.XA CN111651519B (en) | 2020-05-08 | 2020-05-08 | Data synchronization method, data synchronization device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010382046.XA CN111651519B (en) | 2020-05-08 | 2020-05-08 | Data synchronization method, data synchronization device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111651519A CN111651519A (en) | 2020-09-11 |
| CN111651519B true CN111651519B (en) | 2023-04-25 |
Family
ID=72348031
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010382046.XA Active CN111651519B (en) | 2020-05-08 | 2020-05-08 | Data synchronization method, data synchronization device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111651519B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112434108A (en) * | 2020-11-10 | 2021-03-02 | 深圳市明源云科技有限公司 | Database synchronization method, device and equipment |
| CN112632191A (en) * | 2020-12-29 | 2021-04-09 | 中国农业银行股份有限公司 | Data processing method and system |
| CN112732702B (en) * | 2021-03-31 | 2021-06-25 | 腾讯科技（深圳）有限公司 | Database engine file processing method and device |
| CN113052543B (en) * | 2021-05-31 | 2022-05-03 | 广州宸祺出行科技有限公司 | Statistical storage method, device and application of regional orders and driver heating power |
| CN113452519B (en) * | 2021-06-25 | 2022-07-19 | 深圳市电子商务安全证书管理有限公司 | Key synchronization method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956123A (en) * | 2016-05-03 | 2016-09-21 | 无锡雅座在线科技发展有限公司 | Local updating software-based data processing method and apparatus |
| CN107229721A (en) * | 2017-06-02 | 2017-10-03 | 泰华智慧产业集团股份有限公司 | A kind of method and device for changing data pick-up |
| WO2018113580A1 (en) * | 2016-12-19 | 2018-06-28 | 腾讯科技（深圳）有限公司 | Data management method and server |
| CN110321383A (en) * | 2019-05-20 | 2019-10-11 | 平安普惠企业管理有限公司 | Big data platform method of data synchronization, device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8918380B2 (en) * | 2009-07-09 | 2014-12-23 | Norsync Technology As | Methods, systems and devices for performing incremental updates of partial databases |
-
2020
- 2020-05-08 CN CN202010382046.XA patent/CN111651519B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956123A (en) * | 2016-05-03 | 2016-09-21 | 无锡雅座在线科技发展有限公司 | Local updating software-based data processing method and apparatus |
| WO2018113580A1 (en) * | 2016-12-19 | 2018-06-28 | 腾讯科技（深圳）有限公司 | Data management method and server |
| CN107229721A (en) * | 2017-06-02 | 2017-10-03 | 泰华智慧产业集团股份有限公司 | A kind of method and device for changing data pick-up |
| CN110321383A (en) * | 2019-05-20 | 2019-10-11 | 平安普惠企业管理有限公司 | Big data platform method of data synchronization, device, computer equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| Zhang Xiao-jie et al..Operation log based synchronization algorithm for cloud storage service with multiple clients.Computer Engineering and Design.2013,第34卷(第11期),第3894-3899页. * |
| 姚远湘 ; .不规范业务数据增量抽取策略的研究.价值工程.2010,(36),第176-177页. * |
| 戴浩 ; 杨波 ; .ETL中的数据增量抽取机制研究.计算机工程与设计.2009,(23),第5552-5555页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111651519A (en) | 2020-09-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111651519B (en) | Data synchronization method, data synchronization device, electronic equipment and storage medium | |
| CN107544984B (en) | Data processing method and device | |
| JP2022141957A (en) | Metadata snapshot method and snap device thereof | |
| CN111324610A (en) | Data synchronization method and device | |
| EP2380090B1 (en) | Data integrity in a database environment through background synchronization | |
| CN106934048A (en) | Online data moving method, agent node | |
| WO2022126974A1 (en) | Kafka-based incremental data synchronization method and apparatus, device, and medium | |
| CN105900093A (en) | Keyvalue database data table updating method and data table updating device | |
| CN102946410A (en) | Method and device for network synchronization | |
| CN105608126A (en) | Method and apparatus for establishing secondary indexes for massive databases | |
| CN102946411A (en) | Network synchronization system | |
| US20230012642A1 (en) | Method and device for snapshotting metadata, and storage medium | |
| CN110704442A (en) | Real-time acquisition method and device for big data | |
| CN112131214A (en) | Method, system, equipment and storage medium for data writing and data query | |
| US20170270149A1 (en) | Database systems with re-ordered replicas and methods of accessing and backing up databases | |
| CN103809915B (en) | The reading/writing method of a kind of disk file and device | |
| CN113760600B (en) | Database backup method, database restoration method and related devices | |
| CN115080666A (en) | Data synchronization method, system, electronic device and storage medium | |
| CN113672556A (en) | Batch file migration method and device | |
| CN108595552B (en) | Data cube publishing method and device, electronic equipment and storage medium | |
| CN111966650A (en) | Operation and maintenance big data sharing data table processing method and device and storage medium | |
| CN111694801A (en) | Data deduplication method and device applied to fault recovery | |
| CN110795499B (en) | Cluster data synchronization method, device, equipment and storage medium based on big data | |
| CN111813779A (en) | Data query method, system, device and medium based on data interface configuration | |
| CN117235179A (en) | SQOOP-based data synchronization method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |