CN113760910A - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN113760910A CN113760910A CN202111009935.2A CN202111009935A CN113760910A CN 113760910 A CN113760910 A CN 113760910A CN 202111009935 A CN202111009935 A CN 202111009935A CN 113760910 A CN113760910 A CN 113760910A
- Authority
- CN
- China
- Prior art keywords
- data
- key
- data table
- increment
- self
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for data synchronization, which comprises the following steps: the data synchronization end acquires a first autonomous key sequence serving as a unique identifier of each row of data in a first data table from a database; the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode; determining an autonomous key which is not recorded in the first autonomous key sequence in the second autonomous key sequence of the second data table as a difference primary key; the second data table is a data table which is locally stored by the data synchronization end and has a synchronization relation with the first data table; and deleting the line data corresponding to the difference primary key from the second data table to obtain the synchronized second data table. Therefore, the data consistency between the database and the data synchronization end is ensured, the data synchronization efficiency is improved, and the database security is improved.
Description
Technical Field
The invention relates to the field of financial technology (Fintech), in particular to a data synchronization method and device.
Background
In the prior art, with the continuous enrichment of service scenes and the continuous increase of the number of advertisements, a data synchronization end needs to acquire data in a database and then cache the data in the local of the data synchronization end, so that the speed of recommending advertisements on line is increased.
Currently, data synchronization between a database and a data synchronization end is generally achieved through the following several ways:
1. the data synchronization end periodically acquires the full data of the database and caches the full data to the local of the data synchronization end.
2. The data synchronization end periodically acquires the incremental data of the database and caches the incremental data to the local of the data synchronization end.
3. And using a preset frame, and acquiring and analyzing the data log in the database by the data synchronization end through the preset frame, so as to synchronously update the local data of the data synchronization end according to the data log.
However, the updating in the mode 1 consumes time and has low performance, so that the normal access of the cache is influenced; in the mode 2, the synchronization operation cannot be performed according to the data deletion condition in the database, so that the data consistency between the database and the data synchronization end cannot be ensured; in the mode 3, the preset framework can directly acquire the data log of the database, so that potential safety hazards exist in the database.
Therefore, a data synchronization method between a database and a data synchronization end is needed to improve the security of the database and improve the data synchronization efficiency between the database and the data synchronization end.
Disclosure of Invention
The embodiment of the invention provides a data synchronization method and device, which are used for improving the security of a database and improving the data synchronization efficiency between the database and a data synchronization end.
In a first aspect, an embodiment of the present invention provides a method for data synchronization, including:
the data synchronization end acquires a first autonomous key sequence serving as a unique identifier of each row of data in a first data table from a database; the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode;
the data synchronization end determines the self-increment key which is not recorded in the first self-increment key sequence in the second self-increment key sequence of the second data table as a difference main key; the second data table is a data table which is locally stored by the data synchronization end and has a synchronization relation with the first data table;
and the data synchronization end deletes the line data corresponding to the difference main key from the second data table to obtain a synchronized second data table.
In the technical scheme, the deleted data in the database is determined, so that the data consistency between the database and the local part of the data synchronization end is ensured, specifically, a difference main key between the database and the local part of the data synchronization end is determined through an auto-increment main key sequence, inconsistent data between the database and the local part of the data synchronization end is determined through the difference main key, then the inconsistent data is deleted, so that a local second data table of the data synchronization end is updated, and the data consistency between the database and the data synchronization end is ensured; when the data are synchronized, the full data do not need to be synchronized, so that the data synchronization efficiency is improved; the deleted data in the database is determined, so that the synchronization of the deleted data in the database is ensured between the database and the data synchronization end; because the invention uses the difference main key to determine the deleted data in the database, the database log is not needed to determine the deleted data in the database, thereby improving the safety of the database.
Optionally, the method further includes:
the data synchronization end acquires time stamps of data in each row in the first data table; the time stamp of each row of data is used for representing the modification and/or increase time of each row of data;
the data synchronization end determines incremental data according to the time stamp of each row of data;
and the data synchronization end updates the incremental data to the second data table.
In the technical scheme, when the added and/or modified data in the database are synchronized, the data to be synchronized, namely the incremental data, is determined through the timestamp of each row of data in the first data table, so that the data consistency between the database and the data synchronization end is ensured; and because the incremental data does not need to be determined through a database log, the safety of the database is improved.
Optionally, the determining, by the data synchronization end, an autonomic key that is not recorded in the first autonomic key sequence in the second autonomic key sequence of the second data table as a difference autonomic key includes:
the data synchronization end stores the first autonomous key corresponding to each row of data in the first autonomous key sequence in an array form; storing second self-increment keys of data in each row in the second self-increment key sequence in a key value pair mode;
for any second self-increment key, the data synchronization end traverses the first self-increment key sequence stored in an array form and judges whether a first self-increment key consistent with the second self-increment key is recorded in the first self-increment key sequence or not;
and if not, determining the second self-increment main key as a difference main key.
In the above technical solution, when determining the difference primary key, since the first autonomous key sequence to be traversed is stored in an array form, the traversal efficiency can be increased, and since the second autonomous key is stored in a key value pair form, the efficiency of selecting the second autonomous key to be determined can be improved, thereby improving the efficiency of data synchronization.
Optionally, after obtaining the synchronized second data table, the method further includes:
and the data synchronization end deletes each difference main key in the second self-increment main key sequence to obtain an updated second self-increment main key sequence.
In the above technical solution, the difference primary key is equivalent to being recorded in the second self-increment key sequence, but has been deleted in the first self-increment key sequence, that is, is not recorded in the first self-increment key sequence, so that each difference primary key in the second self-increment key sequence is deleted for determining the difference primary key in the next cycle, thereby increasing the real-time performance of data synchronization.
Optionally, the determining, by the data synchronization end, incremental data according to the timestamp of each row of data includes:
the data synchronization end determines data corresponding to the field to be synchronized, of which the time stamp meets preset conditions, in a database as the incremental data according to the field to be synchronized in the configuration file; the configuration file is used for indicating fields to be synchronized between the first data table and the second data table; the preset condition is that the timestamp is within the synchronization period.
In the technical scheme, the incremental data is determined through the field to be synchronized, so that the data of the database can be selectively acquired, and the flexibility of data synchronization is improved.
Optionally, a table name structure is configured in the configuration file;
the data synchronization end acquires a first autonomous key sequence serving as a unique identifier of each row of data in a first data table from a database, and the method comprises the following steps:
and the data synchronization end generates a query statement according to the table name structure in the configuration file, and acquires a first self-increment key sequence serving as a unique identifier of each row of data in the first data table from a database through the query statement.
In the technical scheme, the first autonomous key sequence is generated by taking the table name as a unit, so that data synchronization errors are prevented, and the accuracy of data synchronization is improved.
Optionally, the method further includes:
the data synchronization end carries out table structure deformation on the second data table according to a preset rule of a configuration file to obtain a third data table;
the data synchronization end determines an associated key corresponding to each difference main key according to the preset rule;
and the data synchronization end updates incremental data to the third data table through the preset rule, and deletes the data corresponding to the association key from the third data table to obtain an updated third data table.
In the technical scheme, when data synchronization is carried out, data synchronization can be carried out on the third data table which has an association relation with the second data table, and the flexibility of data synchronization is improved.
In a second aspect, an embodiment of the present invention provides an apparatus for data synchronization, including:
the acquisition module is used for acquiring a first autonomous key sequence which is used as a unique identifier of each row of data in a first data table from a database; the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode;
the processing module is used for determining an autonomous key which is not recorded in the first autonomous key sequence in the second autonomous key sequence of the second data table as a difference primary key; the second data table is a data table which is locally stored by the data synchronization end and has a synchronization relation with the first data table;
and deleting the line data corresponding to the difference primary key from the second data table to obtain a synchronized second data table.
Optionally, the processing module is further configured to:
the control acquisition module acquires the time stamp of each row of data in the first data table; the time stamp of each row of data is used for representing the modification and/or increase time of each row of data;
the processing module is further configured to:
determining incremental data according to the time stamps of the data of all rows;
updating the incremental data to the second data table.
Optionally, the processing module is specifically configured to:
storing the first autonomic key corresponding to each row of data in the first autonomic key sequence in an array form; storing second self-increment keys of data in each row in the second self-increment key sequence in a key value pair mode;
for any second autonomic key, traversing the first autonomic key sequence stored in an array form, and judging whether a first autonomic key consistent with the second autonomic key is recorded in the first autonomic key sequence;
and if not, determining the second self-increment main key as a difference main key.
Optionally, the processing module is further configured to:
and deleting the different main keys in the second self-increment main key sequence after the synchronized second data table is obtained, so as to obtain an updated second self-increment main key sequence.
Optionally, the processing module is specifically configured to:
determining data corresponding to the field to be synchronized, of which the time stamp meets preset conditions, in a database as the incremental data according to the field to be synchronized in the configuration file; the configuration file is used for indicating fields to be synchronized between the first data table and the second data table; the preset condition is that the timestamp is within the synchronization period.
Optionally, the processing module is specifically configured to:
and generating a query statement according to the table name structure in the configuration file, and acquiring a first autonomous key sequence serving as a unique identifier of each row of data in the first data table from a database through the query statement.
Optionally, the processing module is further configured to:
performing table structure deformation on the second data table according to a preset rule of the configuration file to obtain a third data table;
determining an associated key corresponding to each difference main key according to the preset rule;
updating incremental data to the third data table according to the preset rule, and deleting the data corresponding to the associated key from the third data table to obtain an updated third data table.
In a third aspect, an embodiment of the present invention further provides a computer device, including:
a memory for storing program instructions;
and the processor is used for calling the program instructions stored in the memory and executing the data synchronization method according to the obtained program.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, where computer-executable instructions are stored, and the computer-executable instructions are configured to enable a computer to execute the above data synchronization method.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of data synchronization according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a system architecture according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a method for data synchronization according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a second data table for determining a difference primary key update according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a set of primary keys according to an embodiment of the present invention;
fig. 6 is a flowchart illustrating a method for data synchronization according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating a second data table for determining incremental data updates according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the prior art, with the continuous enrichment of service scenes and the continuous increase of the number of advertisements, the data synchronization end needs to acquire data in the database to realize the data consistency between the data synchronization end and the database. For example, the data synchronization end is a cloud flash payment advertisement system, and the cloud flash payment advertisement system stores advertisement data in a database by adopting a Redis cache technology so as to increase the efficiency of advertisement pushing.
Fig. 1 is a schematic diagram illustrating data synchronization, and as shown in fig. 1, in order to ensure security of data synchronization, authority management is performed on a cloud flash payment advertisement system through a management application, and when data synchronization between the cloud flash payment advertisement system and a database is performed, the following 4 methods are generally included.
1. And (4) fishing the data in the database in full, and carrying out full replacement on the data stored in the cloud flash payment advertisement system.
2. And acquiring new data of the database, storing the new data to the local by the cloud flash advertisement system through a Redis cache technology, and accessing the local data of the data synchronization end when the data synchronization is realized.
3. The cloud flash payment advertisement system uses an open source framework databus, and the data of the database log is analyzed by crossing management applications to determine changed data in the database, so that data synchronization between the cloud flash payment advertisement system and the database is realized. The data source end of the open source framework databus supports oracle and mysql, and the target end supports extended database types such as Redis and memory database.
4. The cloud flash payment advertising system analyzes the database logs by using an open source framework otter across management applications, and determines changed data in the database, so that data synchronization between the cloud flash payment advertising system and the database is realized.
For the method 1, when the data volume in the database or the data volume in the cache is huge, the database which is fetched in a full amount occupies a large amount of computing resources, the time is long, and the cloud flash payment advertisement system consumes a large amount of time when being updated in a full amount, so that the data synchronization efficiency and the access efficiency are influenced.
Aiming at the method 2, when the incremental data is acquired, the deleted data in the database cannot be determined, so that the data consistency between the cloud flash payment advertisement system and the database cannot be realized.
For the above methods 3 and 4, the open source framework databus and the open source framework otter only support oracle and mysql, but do not support other commonly used databases, such as sql server and the like. And both the open source framework databus and the open source framework otter stride across the management application and directly read the database logs, so that the right management of the management application is invalid, and the database has great security risk, namely low security.
Therefore, a data synchronization method is needed to perform data synchronization without analyzing data logs, so that the security of the database is improved, and the data synchronization efficiency between the database and the data synchronization end is improved.
Fig. 2 illustrates an exemplary system architecture to which an embodiment of the present invention is applicable, which includes a database 210, a management application 220, and a data synchronization end 230.
The database 210 may be a relational database, a non-relational database, etc., and is not limited in detail herein.
And the management application 220 is used for performing authority management and ensuring the security of the database.
The data synchronization end 230, the data synchronization end 230 may be a cloud flash payment advertisement system or other advertisement systems, or may be a non-advertisement system, which is not specifically limited herein.
The data synchronization end 230 is configured to obtain a timestamp of each row of data in the first data table in the database, and determine incremental data according to the timestamp; acquiring a first self-increment key sequence of a first data table in a database, and determining a difference main key according to a second self-increment key sequence and the first self-increment key sequence stored by the database; and deleting the data corresponding to the difference main key in the second data table corresponding to the first data table according to the difference main key, and storing the incremental data to the second data table to obtain the second data table after data synchronization.
It should be noted that the structure shown in fig. 2 is only an example, and the embodiment of the present invention is not limited thereto.
Based on the above description, fig. 3 is a schematic flow chart illustrating a method for data synchronization according to an embodiment of the present invention, where the flow chart is executable by a device for data synchronization.
As shown in fig. 3, the process specifically includes:
in step 310, the data synchronization end obtains a first autonomy key sequence, which is used as a unique identifier of each row of data, from the database in the first data table.
In the embodiment of the invention, the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode. For example, 10 rows of data are shared in a certain data table, and a row of data is inserted into the data table, the increment key of the row of data is 11, the increment value in this example is 1, and the increment value may be other values, such as 2, according to a specific application scenario, which is not limited herein.
The method comprises the steps of constructing an auto-increment key by using a program statement (such as sequence) of a database (such as oracle), and creating the auto-increment key for each row of data of each first data table in the database through a Structured Query Language (such as sqlserver).
And 320, determining the self-increment key which is not recorded in the first self-increment key sequence in the second self-increment key sequence of the second data table as the difference main key by the data synchronization terminal.
In this embodiment of the present invention, the second data table is a data table that is locally stored by the data synchronization end and has a synchronization relationship with the first data table. For example, if the table name of the first data table is a1, the table name of the corresponding second data table is a1, or the table name corresponding to the a1 first data table is b 1.
And 330, deleting the line data corresponding to the difference primary key from the second data table by the data synchronization end to obtain a synchronized second data table.
For example, if the difference primary key is 7, the row of data in the second data table with the autonomy primary key of 7 is deleted, so that data synchronization is realized.
In step 320, after the first sequence of autonomic keys is obtained, for any second autonomic key, traversal is performed in the first sequence of autonomic keys to determine a second autonomic key that is not recorded in the first sequence of autonomic keys, thereby determining a difference primary key.
Further, the data synchronization end stores the first self-increment key corresponding to each row of data in the first self-increment key sequence in an array form; storing second self-increment keys of data in each row in the second self-increment key sequence in a key value pair mode; for any second autonomic key, traversing the first autonomic key sequence stored in an array form, and judging whether a first autonomic key consistent with the second autonomic key is recorded in the first autonomic key sequence; and if not, determining the second self-increment main key as a difference main key.
In the embodiment of the invention, a plurality of first data tables exist in the database, and the first autonomic key of each first data table is stored by taking the table name as a unit.
In one implementable manner, storage is made in the form of key value pairs, such as a key (key) as table name a1 and a value (value) as the first autonomic key present in the a1 first data table.
In another practical manner, the storage is performed in an array form, for example, [ a1] ═ 1,3,4, … …, n }, where a1 is a table name of the first data table, and "{ 1,3,4, … …, n1 }" is the first set of autonomous keys recorded in the a1 first data table, where n is a natural number and n1 is the largest first autonomous key in the first data table, and details of n are not described below.
In the embodiment of the invention, for the first data table, each first autonomic key is stored in an array form, so that the traversal efficiency is improved. Illustratively, for each first data table, each first autonomic key of each first data table is stored in an array, resulting in a first set. For example, the sequence of the autonomic keys of each first data table in the database is obtained, and the first set is [ a1] ═ {1,3,4, … …, n1}, [ a2] ═ 1,7,9, … …, n2} and the like.
And for a second data table locally stored at the data synchronization end, storing each second autonomous key in a key value pair mode. Illustratively, for each second data table, each second autonomous key of each second data table is stored as a key-value pair, resulting in a second set.
Where the key-value pair form may be in hashset form, for example, the second set is { k-b1, v-1,3,4, … …, n1}, { k-b2, v-1,7,9, … …, n2 }.
In the embodiment of the present invention, before determining the first set, it may be determined whether the database deletes data by determining the number of first autonomic keys, for example, determining that 10 first autonomic keys in the first set are included and 7 second autonomic keys in the second set are included, determining that the database deletes data.
After the first set is determined, any second autonomic key is traversed in the first set, and it is determined whether there is a first autonomic key corresponding to the second autonomic key in the first set, for example, the second autonomic key is { k-b1, v-1}, since "1" is recorded in the first autonomic key sequence [ a1] ═ 1,3,4, … …, n1} of the first data table a1, it is determined that the second autonomic key { k-b1, v-1} is not a difference primary key, and if "1" is not recorded in the first autonomic key sequence of the first data table a1, it is determined that the second autonomic key { k-b1, v-1} is a difference primary key. And according to the same method, traversing and judging all the second self-increment main keys to determine all the difference main keys.
To better illustrate the above technical solution, fig. 4 exemplarily shows a schematic diagram of determining a difference primary key to update the second data table, as shown in fig. 4, including the following steps.
In step 410, a first number of autonomics keys is obtained.
Assuming that two rows of data with the first autonomic key of 7 and 9 are deleted from the first data table a1 of the database in the current synchronization period, i-2 first autonomic keys are collectively acquired from the database when data synchronization is performed, wherein i is a natural number and represents the number of the autonomic keys.
Step 420, determining whether the number of the primary keys meets a number condition, if so, executing step 430, otherwise, canceling the synchronization process for deleting the data.
The quantity condition is that the first quantity of all the first autonomic keys acquired from the database in common is consistent with the second quantity of all the second autonomic keys stored locally at the data synchronization end. For example, if the second number is i, the first number is not consistent with the second number, and step 430 is performed.
At step 430, a first autonomy key sequence is determined.
And taking the table name as a unit, acquiring the first autonomic key from the database in full quantity, and determining the first autonomic sequence of each first data table to obtain a first set.
At step 440, a difference primary key is determined.
And traversing the first set aiming at any second self-increment primary key value locally stored at the data synchronization end to determine a difference primary key.
Taking any second self-increment key value of a second data table with the table name b1 as an example, the second self-increment key is { k-b1, v-1}, the first set is traversed, if the first self-increment key sequence of the first data table a1 in the first set records "1", the second self-increment key { k-b1, v-1} is determined not to be the difference primary key, and if the first self-increment key sequence of the first data table a1 does not record "1", the second self-increment key { k-b1, v-1} is determined to be the difference primary key. And traversing and judging all the second autonomic keys in the second set to determine all the difference primary keys, where fig. 5 exemplarily shows a schematic diagram of a difference primary key set, and as shown in fig. 5, a difference primary key is an autonomic key recorded in the second set but not recorded in the first set.
Step 450, updating the data synchronization end.
And determining a second data table to be updated according to the difference primary key, and deleting data corresponding to the difference primary key in the second data table to be updated to obtain the updated second data table.
After the deleted data of the database is synchronized, namely the updated second data table is obtained, the data synchronization end can also update the second set, namely the second self-increment key sequence, so that the difference main key is determined in the next synchronization period.
Further, the data synchronization end deletes each difference primary key in the second self-increment primary key sequence to obtain an updated second self-increment primary key sequence.
For example, if the second auto-increment key sequence is { k-b1, v-1,3,4, … …, n1}, and it is now determined that the second auto-increment key { k-b1, v-1} is the difference key, then the updated second auto-increment key sequence is { k-b1, v-,3,4, … …, n1 }.
For example, the data synchronization end may generate a query statement according to a pre-configured configuration file, so as to determine a first autonomic key sequence of each line of data of each first data table. Specifically, the data synchronization end generates a query statement according to a table name structure in the configuration file, and obtains a first self-increment key sequence serving as a unique identifier of each row of data in a first data table from a database through the query statement.
In addition, in order to prevent data errors when the data synchronization end synchronizes data, the accuracy of data synchronization can be ensured by setting a flag.
Fig. 6 is a flow chart illustrating an exemplary method of data synchronization, as shown in fig. 6, with the following steps.
At step 610, a clear flag is set.
After determining that the database deletes data through the first number of all the first autonomic keys in the database, the data synchronization end judges whether a clearing mark is set, if so, the data synchronization end is proved to be performing data synchronization, the data synchronization of this time is cancelled, and if not, the clearing mark is set, so that data errors caused by repeated data synchronization of the data synchronization end are prevented.
Taking the data table as a unit, judging the second autonomous key sequence in each second data table, for example, the data synchronization end has 3 data tables in total, and if it is determined that only the first data table is judged to be completed, the next data table is also required to be judged; and if the 3 data tables are determined to be completely judged, the data synchronization transaction is completed.
Taking any second data table as an example, for any second autonomic key in the second data table, the first autonomic key sequence is traversed, and whether the second autonomic key is a difference autonomic key is determined.
At step 640, a set of difference primary keys is determined.
And traversing each second self-increment key in the second data table through the first self-increment key sequence to determine one or more difference main keys to obtain a difference main key set.
In step 650, it is determined whether all the difference keys have been processed, if yes, step 620 is performed, otherwise step 660 is performed.
And judging whether to delete the data corresponding to the difference primary keys in the second data table aiming at all the difference primary keys in the difference primary key set. For example, the difference primary key set includes difference primary keys z1 and z2, and when it is determined that only the data corresponding to the difference primary key z1 or z2 in the second data table is deleted, the processing on other primary keys in the difference primary key set needs to be continued; if the data corresponding to the difference primary keys z1 and z2 in the second data table are determined to be deleted, all the processing of all the difference primary keys in the difference primary key set is completed.
In step 660, the data of the difference primary key is deleted.
In step 670, the cleanup flag is deleted.
In step 330, a synchronized second data table is obtained, which further includes synchronization of the incremental data, i.e. synchronization of the data added/modified in the database.
Specifically, the data synchronization end acquires time stamps of data in each row in the first data table; the time stamp of each row of data is used for representing the modification and/or increase time of each row of data; determining incremental data according to the time stamps of the data of all rows; updating the incremental data to the second data table.
Taking any first data table as an example, the first data table a1 has 3 rows of data, each row of data has a corresponding timestamp, for example, the timestamp of the third row of data is 2021-08-15-18:01:37, that is, the third row of data is modified and/or added at the time of 2021-08-15-18:01:37 and recorded in the first data table a1, it should be noted that the unit of timestamp is preset, or may be a unit of minute, and is not limited herein.
In an implementation manner, when the timestamp meets a preset condition, the line data corresponding to the timestamp is determined as incremental data.
Wherein the preset condition is that the timestamp is within the synchronization period. For example, if the data synchronization end and the database are periodically synchronized in the synchronization period, for example, if data synchronization is performed in a period of 10 minutes, and the time of the previous synchronization period is 10:00 at 8 months and 16 days in 2021, the time of the current synchronization period is 10:10 at 8 months and 16 days in 2021, that is, the data with the timestamp between 10:00 at 8 months and 16 days in 2021 and 10:10 at 8 months and 16 days in 2021 is value-added data. It should be noted that the intervals between the synchronization cycles may be the same or different, such as the previous synchronization cycle is 10 minutes, and the next synchronization cycle is 20 minutes, which is not limited herein.
In the embodiment of the present invention, the incremental data may be further determined by using a configuration file, and specifically, the data synchronization end determines, according to a field to be synchronized in the configuration file, data corresponding to the field to be synchronized, where a timestamp in a database meets a preset condition, as the incremental data; the configuration file is used for indicating fields to be synchronized between the first data table and the second data table.
For example, the first data table in the database has data of two fields, c1 and c2, and the field to be synchronized in the configuration file is c1, and when the timestamp meets the preset condition, the data of the c1 field in the row of data corresponding to the timestamp is determined as incremental data; in other words, the second data table includes only a part of the fields in the first data table, but not all the fields, for example, the first data table has 5 fields in total, but only 3 fields are included in the second data table corresponding to the first data table stored at the data synchronization end.
To better illustrate the above-mentioned technical solution of the incremental data, fig. 7 exemplarily shows a schematic diagram of determining that the incremental data updates the second data table, as shown in fig. 7, including the following steps.
Step 710, reading the configuration file.
It is assumed that, in the synchronization cycle, the database records the modified data and inserts the newly added data, and updates the corresponding timestamp.
And the data synchronization end determines the fields to be synchronized in the configuration file.
Step 720, judging whether the first loading is carried out, if so, acquiring data corresponding to the field to be synchronized of each first data table in the database, otherwise, executing step 730.
At step 730, incremental data is determined.
Taking any first data table as an example, the timestamp of each row of data in the first data table is obtained, and the data corresponding to the field to be synchronized in the row of data with the timestamp meeting the preset condition is determined as incremental data.
Step 740, updating the second data table.
And after the incremental data are determined, storing the incremental data into the second data table to obtain an updated second data table.
For example, the configuration file may further configure a preset rule to perform table structure deformation on the second data table, when the first data table in the database is loaded for the first time, the first data table is stored to the local of the data synchronization end to obtain the second data table, then the third data table is obtained according to the preset rule, and when data synchronization is performed, data synchronization may also be performed on the third data table according to the preset rule of the configuration file.
Further, the data synchronization end performs table structure deformation on the second data table according to a preset rule of the configuration file to obtain a third data table; determining an associated key corresponding to each difference main key according to the preset rule; updating incremental data to the third data table according to the preset rule, and deleting the data corresponding to the associated key from the third data table to obtain an updated third data table.
For example, the second data table is shown in table 1 below, the predetermined rule is an association key, and the third data table is shown in table 2 below.
TABLE 1
| Advertisement number | Advertising information | User name of advertisement to be pushed |
| 01 | f1 | s1、s2 |
| 02 | f2 | s2、s3 |
TABLE 2
| User name | Number of advertisement to be pushed | Advertising information |
| s1 | 01 | f1 |
| s2 | 01、02 | f1、f2 |
| s3 | 02 | f2 |
To better illustrate the above technical solution, an exemplary procedure for implementing the above data synchronization will be described below.
For example, the database is a relational database in which an advertisement information table is created as shown in table 3 below.
TABLE 3
| Name of field | Attribute name | Description of values | Remarks for note |
| Advertisement number | ID | BIGINT | Physical key |
| Black and white list of users | ACT_LIMIT_FLAG | VARchar(200) | User ID |
From table 3, an advertisement information table as shown in table 1 above can be created.
The configuration file is determined using the table structure of Lua (scripting language), such as the configuration file shown in table 4 below.
TABLE 4
The procedure for implementing table 4 above is as follows:
the table name is used for representing a table name structure and generating a query statement with the table name as a unit; the data is used for embodying a data structure of a first data table in the database, so that the generated query statement can accurately access the data; "rule" is used to characterize the field to be synchronized; the "unique key" is used to characterize the primary key of the first data table, such as the "advertisement number" of table 1 above; the "associated key" is used to characterize the associated primary key corresponding to the "unique key", such as the "user name" in table 2 above.
Aiming at the deleted data in the database, an execution script generated by the Lua script analysis parameters needs to be loaded in advance at the data synchronization end, and the execution script is as follows:
res＝redis.call('SET','updating',1)；
the method comprises the steps of setting a cleaning mark to prevent repeated data synchronization and ensure the safety of the data synchronization, wherein the cleaning mark is set in a read-write separation mode, so that local data access to a data synchronization end is not influenced during the data synchronization;
local confTable＝loadString(KEYS[1])；
reading a configuration file, wherein 'KEYS [1 ]' is the configuration file;
local confTable＝loadString(KEYS[1])；
traversing the local data synchronization end according to the configuration file;
for k,v in pairs(confTable)do；
local redisKeys＝redis.call('smembers',v["name"])；
determining a second autonomy key sequence of the second data table, wherein 'redisKeys' is the second autonomy key sequence;
local dbKeys＝split(ARGV[k])；
determining first autogenic key sequences of the first data tables, wherein "ARGV [ ] is a first set of first autogenic key sequences of each first data table, separated by a" | "separator, e.g., ARGV [1] is the first data table;
in any second data table, selecting a second autonomic key "v 1", determining whether the second autonomic key "v 1" is recorded in the first set, if not, determining the second autonomic key "v 1" as a difference primary key, and adding the difference primary key to the difference primary key set, wherein "delKeys" is the difference primary key set;
for k2,v2 in pairs(delKeys)do；
selecting a difference primary key "v 2" in the set of difference primary keys;
redis.call('SREM',v["name"],v2)；
deleting data corresponding to the primary key "v 2" in the second data table, and deleting the primary key "v 2" in the second sequence of autonomic keys;
redis.call('DEL',v["name"]..':'..v2)；
for k3,v3 in pairs(v["column"])do；
if v3["property"]＝＝"UK"then；
determining that the differential primary key 'v 2' in the second data table corresponds to the primary key 'v 3' in the third data table according to the associated key 'UK';
local unionKey＝redis.call('smembers',v3["name"]..':ID:'..v2)；
deleting the data corresponding to the primary key 'v 3' in the third data table; taking the above tables 1 and 2 as an example, for example, when the difference primary key "v 2" is the advertisement number "02", it is determined that the difference primary key "v 2" corresponds to the user names "s 2" and "s 3" in table 3 according to the association key, and then the advertisement number "02" and the advertisement information "f 2" corresponding to the user names "s 2" and "s 3" are deleted in the third data table;
and determining that the primary key 'v 3' corresponds to the primary key 'v 4' in the data tables with other table structure forms according to the associated key 'UK', then carrying out data synchronization on all the data tables with corresponding relations, and then finishing the data synchronization.
res＝redis.call('DEL','updating')；
And clearing the cleaning mark.
In the embodiment of the invention, the data synchronization end periodically reads the configuration file, obtains the incremental data according to the table name structure in the configuration file, and synchronizes the incremental data to the local part (such as a Redis cache unit) of the data synchronization end.
After the incremental data synchronization is finished, the deleted data in the database is synchronized by means of the Lua script, specifically, inconsistent data between the database and the local data synchronization end is determined through a difference main key, and then the inconsistent data is deleted, so that a second data table of the local data synchronization end is updated, and the data consistency between the database and the local data synchronization end is further ensured; the efficiency of data synchronization is improved because the full data does not need to be synchronized; the deleted data in the database is determined, so that the synchronization of the deleted data in the database between the database and the data synchronization end is ensured; because the difference primary key is used for determining the deleted data in the database, the database log is not required to be used for determining the deleted data in the database, and therefore the safety of the database is improved.
Based on the same technical concept, fig. 8 exemplarily shows a schematic structural diagram of a data synchronization apparatus provided by an embodiment of the present invention, and the apparatus can execute a flow.
As shown in fig. 8, the apparatus specifically includes:
an obtaining module 810, configured to obtain, from a database, a first autonomic key sequence serving as a unique identifier of each row of data in a first data table; the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode;
a processing module 820, configured to determine an autonomic key in the second sequence of autonomic keys of the second data table that is not recorded in the first sequence of autonomic keys as a difference primary key; the second data table is a data table which is locally stored by the data synchronization end and has a synchronization relation with the first data table;
and deleting the line data corresponding to the difference primary key from the second data table to obtain a synchronized second data table.
Optionally, the processing module 820 is further configured to:
the control obtaining module 810 obtains the time stamp of each row of data in the first data table; the time stamp of each row of data is used for representing the modification and/or increase time of each row of data;
the processing module 820 is further configured to:
determining incremental data according to the time stamps of the data of all rows;
updating the incremental data to the second data table.
Optionally, the processing module 820 is specifically configured to:
storing the first autonomic key corresponding to each row of data in the first autonomic key sequence in an array form; storing second self-increment keys of data in each row in the second self-increment key sequence in a key value pair mode;
for any second autonomic key, traversing the first autonomic key sequence stored in an array form, and judging whether a first autonomic key consistent with the second autonomic key is recorded in the first autonomic key sequence;
and if not, determining the second self-increment main key as a difference main key.
Optionally, the processing module 820 is further configured to:
and deleting the different main keys in the second self-increment main key sequence after the synchronized second data table is obtained, so as to obtain an updated second self-increment main key sequence.
Optionally, the processing module 820 is specifically configured to:
determining data corresponding to the field to be synchronized, of which the time stamp meets preset conditions, in a database as the incremental data according to the field to be synchronized in the configuration file; the configuration file is used for indicating fields to be synchronized between the first data table and the second data table; the preset condition is that the timestamp is within the synchronization period.
Optionally, the processing module 820 is specifically configured to:
and generating a query statement according to the table name structure in the configuration file, and acquiring a first autonomous key sequence serving as a unique identifier of each row of data in the first data table from a database through the query statement.
Optionally, the processing module 820 is further configured to:
performing table structure deformation on the second data table according to a preset rule of the configuration file to obtain a third data table;
determining an associated key corresponding to each difference main key according to the preset rule;
updating incremental data to the third data table according to the preset rule, and deleting the data corresponding to the associated key from the third data table to obtain an updated third data table.
Based on the same technical concept, an embodiment of the present invention further provides a computer device, including:
a memory for storing program instructions;
and the processor is used for calling the program instructions stored in the memory and executing the data synchronization method according to the obtained program.
Based on the same technical concept, the embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions for causing a computer to execute the above data synchronization method.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A method of data synchronization, comprising:
the data synchronization end acquires a first autonomous key sequence serving as a unique identifier of each row of data in a first data table from a database; the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode;
the data synchronization end determines the self-increment key which is not recorded in the first self-increment key sequence in the second self-increment key sequence of the second data table as a difference main key; the second data table is a data table which is locally stored by the data synchronization end and has a synchronization relation with the first data table;
and the data synchronization end deletes the line data corresponding to the difference main key from the second data table to obtain a synchronized second data table.
2. The method of claim 1, wherein the method further comprises:
the data synchronization end acquires time stamps of data in each row in the first data table; the time stamp of each row of data is used for representing the modification and/or increase time of each row of data;
the data synchronization end determines incremental data according to the time stamp of each row of data;
and the data synchronization end updates the incremental data to the second data table.
3. The method of claim 1, wherein the data sync terminal determines an autonomic key, which is not recorded in the first autonomic key sequence, of the second autonomic key sequence of the second data table as a difference autonomic key, comprising:
the data synchronization end stores the first autonomous key corresponding to each row of data in the first autonomous key sequence in an array form; storing second self-increment keys of data in each row in the second self-increment key sequence in a key value pair mode;
for any second self-increment key, the data synchronization end traverses the first self-increment key sequence stored in an array form and judges whether a first self-increment key consistent with the second self-increment key is recorded in the first self-increment key sequence or not;
and if not, determining the second self-increment main key as a difference main key.
4. The method of claim 1, wherein after obtaining the synchronized second data table, further comprising:
and the data synchronization end deletes each difference main key in the second self-increment main key sequence to obtain an updated second self-increment main key sequence.
5. The method of claim 2, wherein the determining incremental data by the data synchronization end according to the time stamp of each row of data comprises:
the data synchronization end determines data corresponding to the field to be synchronized, of which the time stamp meets preset conditions, in a database as the incremental data according to the field to be synchronized in the configuration file; the configuration file is used for indicating fields to be synchronized between the first data table and the second data table; the preset condition is that the timestamp is within the synchronization period.
6. The method of claim 1, wherein a table name structure is configured in the configuration file;
the data synchronization end acquires a first autonomous key sequence serving as a unique identifier of each row of data in a first data table from a database, and the method comprises the following steps:
and the data synchronization end generates a query statement according to the table name structure in the configuration file, and acquires a first self-increment key sequence serving as a unique identifier of each row of data in the first data table from a database through the query statement.
7. The method of any of claims 1 to 6, further comprising:
the data synchronization end carries out table structure deformation on the second data table according to a preset rule of a configuration file to obtain a third data table;
the data synchronization end determines an associated key corresponding to each difference main key according to the preset rule;
and the data synchronization end updates incremental data to the third data table through the preset rule, and deletes the data corresponding to the association key from the third data table to obtain an updated third data table.
8. An apparatus for data synchronization, comprising:
the acquisition module is used for acquiring a first autonomous key sequence which is used as a unique identifier of each row of data in a first data table from a database; the respective increment keys in the self-increment key sequence are sequentially generated according to the sequence of writing data in each row into the data table in a self-increment mode;
the processing module is used for determining an autonomous key which is not recorded in the first autonomous key sequence in the second autonomous key sequence of the second data table as a difference primary key; the second data table is a data table which is locally stored by the data synchronization end and has a synchronization relation with the first data table;
and deleting the line data corresponding to the difference primary key from the second data table to obtain a synchronized second data table.
9. A computer device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory to perform the method of any of claims 1 to 7 in accordance with the obtained program.
10. A computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111009935.2A CN113760910A (en) | 2021-08-31 | 2021-08-31 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111009935.2A CN113760910A (en) | 2021-08-31 | 2021-08-31 | Data synchronization method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113760910A true CN113760910A (en) | 2021-12-07 |
Family
ID=78792062
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111009935.2A Pending CN113760910A (en) | 2021-08-31 | 2021-08-31 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760910A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114510534A (en) * | 2022-01-28 | 2022-05-17 | 广东航宇卫星科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN115361399A (en) * | 2022-10-24 | 2022-11-18 | 中国水利水电第七工程局有限公司 | Multi-terminal data synchronization method, device and system |
| CN117708239A (en) * | 2023-12-14 | 2024-03-15 | 阳光凯讯(北京)科技股份有限公司 | Method, system and electronic equipment for paging synchronous data |
| CN118368297A (en) * | 2024-06-20 | 2024-07-19 | 杭州宇信数字科技有限公司 | Heterogeneous data real-time synchronization method, device and system crossing data centers |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102375890A (en) * | 2011-11-02 | 2012-03-14 | 杭州合众信息工程有限公司 | Data synchronization method for source terminal table of database without major key |
| CN102915336A (en) * | 2012-09-18 | 2013-02-06 | 北京金和软件股份有限公司 | Incremental data capturing and extraction method based on timestamps and logs |
| CN104252481A (en) * | 2013-06-27 | 2014-12-31 | 阿里巴巴集团控股有限公司 | Dynamic check method and device for consistency of main and salve databases |
| CN108572996A (en) * | 2017-03-14 | 2018-09-25 | 北京京东尚科信息技术有限公司 | Synchronous method, device, electronic equipment and the storage medium of database table structure |
| CN109241188A (en) * | 2018-09-05 | 2019-01-18 | 上海汽车集团股份有限公司 | The trickle transmission method and device of data, storage medium, terminal |
| CN110674146A (en) * | 2019-08-22 | 2020-01-10 | 视联动力信息技术股份有限公司 | Data synchronization method, synchronization end, end to be synchronized, equipment and storage medium |
| CN111104445A (en) * | 2019-12-06 | 2020-05-05 | 杭州数梦工场科技有限公司 | Data synchronization method, device and equipment |
| CN113220785A (en) * | 2021-05-21 | 2021-08-06 | 上海申铁信息工程有限公司 | Database change record synchronization method and device based on primary key comparison |
-
2021
- 2021-08-31 CN CN202111009935.2A patent/CN113760910A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102375890A (en) * | 2011-11-02 | 2012-03-14 | 杭州合众信息工程有限公司 | Data synchronization method for source terminal table of database without major key |
| CN102915336A (en) * | 2012-09-18 | 2013-02-06 | 北京金和软件股份有限公司 | Incremental data capturing and extraction method based on timestamps and logs |
| CN104252481A (en) * | 2013-06-27 | 2014-12-31 | 阿里巴巴集团控股有限公司 | Dynamic check method and device for consistency of main and salve databases |
| CN108572996A (en) * | 2017-03-14 | 2018-09-25 | 北京京东尚科信息技术有限公司 | Synchronous method, device, electronic equipment and the storage medium of database table structure |
| CN109241188A (en) * | 2018-09-05 | 2019-01-18 | 上海汽车集团股份有限公司 | The trickle transmission method and device of data, storage medium, terminal |
| CN110674146A (en) * | 2019-08-22 | 2020-01-10 | 视联动力信息技术股份有限公司 | Data synchronization method, synchronization end, end to be synchronized, equipment and storage medium |
| CN111104445A (en) * | 2019-12-06 | 2020-05-05 | 杭州数梦工场科技有限公司 | Data synchronization method, device and equipment |
| CN113220785A (en) * | 2021-05-21 | 2021-08-06 | 上海申铁信息工程有限公司 | Database change record synchronization method and device based on primary key comparison |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114510534A (en) * | 2022-01-28 | 2022-05-17 | 广东航宇卫星科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN114510534B (en) * | 2022-01-28 | 2022-09-06 | 广东航宇卫星科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN115361399A (en) * | 2022-10-24 | 2022-11-18 | 中国水利水电第七工程局有限公司 | Multi-terminal data synchronization method, device and system |
| CN117708239A (en) * | 2023-12-14 | 2024-03-15 | 阳光凯讯(北京)科技股份有限公司 | Method, system and electronic equipment for paging synchronous data |
| CN118368297A (en) * | 2024-06-20 | 2024-07-19 | 杭州宇信数字科技有限公司 | Heterogeneous data real-time synchronization method, device and system crossing data centers |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113760910A (en) | Data synchronization method and device | |
| US9953051B2 (en) | Multi-version concurrency control method in database and database system | |
| JP7170638B2 (en) | Generating, Accessing, and Displaying Lineage Metadata | |
| US20200372373A1 (en) | System and method for generating organizational memory using semantic knowledge graphs | |
| CN106547811B (en) | Distributed merging of data sets | |
| US8799235B2 (en) | Data de-duplication system | |
| CN107657049B (en) | Data processing method based on data warehouse | |
| CN110134689B (en) | Target group screening method and system based on main body object label change and computer equipment | |
| CN113094442B (en) | Full data synchronization method, device, equipment and medium | |
| CN111949315A (en) | Management device and method for block chain account book data | |
| CN105900093A (en) | Keyvalue database data table updating method and data table updating device | |
| CN113704343A (en) | Data blood margin visualization implementation method and system in data processing | |
| CN115543402A (en) | Software knowledge graph increment updating method based on code submission | |
| CN114564500A (en) | Method and system for implementing structured data storage and query in block chain system | |
| CN109101368B (en) | Data processing method and device | |
| CN111984662B (en) | Method and device for updating databases in batches | |
| CN111460000B (en) | Backtracking data query method and system based on relational database | |
| CN110555065A (en) | Data processing method and device | |
| CN111427871B (en) | Data processing method, device and equipment | |
| US20190266163A1 (en) | System and method for behavior-on-read query processing | |
| CN110147396B (en) | Mapping relation generation method and device | |
| CN104317820A (en) | Statistical method and device of report | |
| CN117251448B (en) | Method and device for processing data of wide-table zipper table | |
| CN111563091B (en) | Method and system for batch updating MongoDB in non-round-trip mode | |
| CN113327167B (en) | Block chain data processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |