CN113392088A - Data synchronization method, device, terminal and storage medium - Google Patents
Data synchronization method, device, terminal and storage medium Download PDFInfo
- Publication number
- CN113392088A CN113392088A CN202110632999.1A CN202110632999A CN113392088A CN 113392088 A CN113392088 A CN 113392088A CN 202110632999 A CN202110632999 A CN 202110632999A CN 113392088 A CN113392088 A CN 113392088A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- target
- task
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000013467 fragmentation Methods 0.000 claims abstract description 119
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 119
- 230000005012 migration Effects 0.000 claims abstract description 84
- 238000013508 migration Methods 0.000 claims abstract description 84
- 238000012545 processing Methods 0.000 claims description 72
- 230000001360 synchronised effect Effects 0.000 claims description 37
- 230000008569 process Effects 0.000 abstract description 12
- 238000010586 diagram Methods 0.000 description 10
- 238000003672 processing method Methods 0.000 description 9
- 239000012634 fragment Substances 0.000 description 4
- 241000478345 Afer Species 0.000 description 2
- 238000013524 data verification Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- BASFCYQUMIYNBI-UHFFFAOYSA-N platinum Chemical compound [Pt] BASFCYQUMIYNBI-UHFFFAOYSA-N 0.000 description 2
- 238000004590 computer program Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 229910052697 platinum Inorganic materials 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method, a data synchronization device, a terminal and a storage medium, wherein the method comprises the following steps: acquiring a first synchronization task, and fragmenting the first synchronization task to obtain at least one fragmentation task, wherein the first synchronization task comprises a plurality of data IDs, and each fragmentation task comprises a plurality of continuous data IDs; generating a migration plan according to each fragmentation task, and when a target fragmentation task in the at least one fragmentation task is executed, respectively reading corresponding target data from a first data source according to a data ID in the migration plan corresponding to the target fragmentation task; and synchronizing each read target data to a second data source. The synchronization process of the invention can be realized by using a general language, and the data synchronization between heterogeneous data sources can be realized without depending on the structure of the first data source or the second data source.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a data synchronization method, an apparatus, a terminal, and a storage medium.
Background
The most common data synchronization scheme at present is the Canal project, and data synchronization is completed by subscribing and consuming the Mysql database binlog, so the oriented main body is limited to Mysql or Mysql derivative products, that is, both synchronization parties need to be available to Mysql or its derivative products and belong to synchronization of isomorphic data sources. In practice, due to the difference between the development time and the selection technology of the information system, there is a need for data synchronization between a plurality of heterogeneous data sources, and the current data synchronization scheme cannot be realized.
Thus, there is a need for improvements and enhancements in the art.
Disclosure of Invention
In view of the above defects in the prior art, the present invention provides a data synchronization method, apparatus, terminal and storage medium, and aims to solve the problem that a data synchronization scheme in the prior art cannot achieve data synchronization between heterogeneous data sources.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
in a first aspect of the present invention, a data synchronization method is provided, where the method includes:
acquiring a first synchronization task, and fragmenting the first synchronization task to obtain at least one fragmentation task, wherein the first synchronization task comprises a plurality of data IDs, and each fragmentation task comprises a plurality of continuous data IDs;
generating a migration plan according to each fragmentation task, and when a target fragmentation task in the at least one fragmentation task is executed, respectively reading corresponding target data from a first data source according to a data ID in the migration plan corresponding to the target fragmentation task;
and synchronizing each read target data to a second data source.
The data synchronization method, wherein the generating a migration plan according to each of the fragmentation tasks includes:
detecting a distributed lock when the number of the generated migration plans is less than the number of the fragmentation tasks;
and generating a migration plan according to the fragmentation tasks for obtaining the distributed locks, and repeatedly executing the step of detecting the distributed locks until the generated migration plan is equal to the number of the fragmentation tasks.
The data synchronization method, wherein the reading of the corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task includes:
reading a preset number of pieces of target data from the first data source according to the starting ID of the migration plan corresponding to the target fragmentation task;
wherein an initial value of the start ID of the migration plan corresponding to the target fragmentation task is a first data ID of a plurality of consecutive data IDs included in the target fragmentation task.
The data synchronization method, wherein the synchronizing the read target data to a second data source, includes:
performing first preset processing on each read target data and writing the target data into the second data source;
performing second preset processing on the target data written into the second data source, and updating the starting ID in the target fragmentation task by using the ID corresponding to the last piece of data in each target data;
and repeatedly executing the step of reading corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task until the data corresponding to the end ID in the migration plan corresponding to the target fragmentation task is written into the second data source.
The data synchronization method, after synchronizing each of the read target data into the second data source, further includes:
acquiring a second synchronous task, wherein the second synchronous task comprises a synchronous timestamp;
searching each first target data ID in the first data source according to the synchronization timestamp, wherein the update timestamp of the data corresponding to the first target data ID is not earlier than the synchronization timestamp;
respectively acquiring a first updating time and a second updating time corresponding to each first target data ID, wherein the first updating time is the updating time of data corresponding to the first target data ID in the first data source, and the second updating time is the updating time of data corresponding to the first target data ID in the second data source;
when the first updating time corresponding to the first target data ID is later than the second updating time, reading the data corresponding to the first target data ID from the first data source, performing first preset processing, and updating the data corresponding to the first target data ID in the second data source according to the data subjected to the first preset processing.
The data synchronization method, wherein after the updating of the data corresponding to the first target data ID in the second data source according to the data subjected to the first preset processing, includes:
when the update timestamp of the latest data in the first data source is the same as the synchronization timestamp, updating the synchronization timestamp, wherein the time corresponding to the updated synchronization timestamp is later than the time corresponding to the synchronization timestamp before updating;
and when the update time stamp of the data with the latest update time in the first data source is later than the synchronization time stamp, updating the synchronization time stamp into the update time stamp of the data with the latest update time in the first data source.
The data synchronization method, after the step of synchronizing each of the read target data into the second data source, further includes:
acquiring a third synchronous task, wherein the third synchronous task comprises a synchronous schedule;
when the data in the first data source is changed, recording the data ID of the changed data into the synchronous schedule and marking the state as successful processing;
reading a second target data ID from the synchronization schedule, wherein the second target data ID is a data ID in the synchronization schedule, the state of which is successful in processing, if the update time stamp of the data corresponding to the second target data ID in the first data source is later than that of the data corresponding to the second data source, reading the data corresponding to the second target data ID from the first data source, performing first preset processing, and updating the data corresponding to the second target data ID in the second data source according to the data subjected to the first preset processing;
modifying the state of the second target data ID in the synchronization schedule to be processed;
and clearing the data ID with the processed state in the synchronization schedule.
In a second aspect of the present invention, there is provided a data synchronization apparatus, comprising:
the task obtaining module is used for obtaining a first synchronization task and fragmenting the first synchronization task to obtain at least one fragmentation task, wherein the first synchronization task comprises a plurality of data IDs, and each fragmentation task comprises a plurality of continuous data IDs;
the data reading module is used for generating a migration plan according to each fragmentation task, and when the migration plan corresponding to a target fragmentation task in the at least one fragmentation task is executed, reading corresponding target data from a first data source according to a data ID in the target fragmentation task;
and the synchronization module is used for synchronizing each read target data to a second data source.
In a third aspect of the present invention, there is provided a terminal comprising a processor, and a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being adapted to store a plurality of instructions, and the processor being adapted to invoke the instructions in the computer-readable storage medium to perform the steps of implementing the data synchronization method according to any one of the above.
In a fourth aspect of the present invention, there is provided a computer readable storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the steps of the data synchronization method of any of the above.
Compared with the prior art, the data synchronization method, the device, the terminal and the storage medium provided by the invention have the advantages that the synchronization task which needs to synchronize a plurality of pieces of data from a first data source to a second data source is segmented, a migration plan is generated for each segmented task, when the migration plan is executed, corresponding data are read from the first data source and synchronized to the second data source according to a data ID in the segmented task, in the whole process, only the steps of searching, reading and writing are needed according to the data ID, the synchronization process can be realized by using a universal language, the structure of the first data source or the second data source is not depended on, and the data synchronization between heterogeneous data sources can be realized.
Drawings
FIG. 1 is a flow chart of an embodiment of a data synchronization method provided by the present invention;
FIG. 2 is a logic diagram of obtaining target data when processing a first synchronization task in an embodiment of a data synchronization method provided in the present invention;
FIG. 3 is a logic diagram of generating a migration plan while processing a first synchronization task in an embodiment of a data synchronization method provided by the present invention;
FIG. 4 is a logic diagram of a data synchronization method provided by the present invention for synchronizing target data to a second data source when processing a first synchronization task;
FIG. 5 is a logic diagram of querying data from a first data source while processing a second sync task in an embodiment of a data synchronization method provided by the present invention;
FIG. 6 is a logic diagram of updating a synchronization timestamp when processing a second synchronization task in an embodiment of the data synchronization method provided in the present invention;
FIG. 7 is a logic diagram of writing data to a second data source while processing a second sync task in an embodiment of the data synchronization method provided in the present invention;
FIG. 8 is a logic diagram of an embodiment of a data synchronization method provided in the present invention for obtaining a synchronization plan when processing a third synchronization task;
FIG. 9 is a logic diagram of writing data to a second data source while processing a third sync task in an embodiment of a data synchronization method provided by the present invention;
FIG. 10 is a schematic diagram of an embodiment of a data synchronization apparatus according to the present invention;
fig. 11 is a schematic diagram illustrating an embodiment of a terminal according to the present invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The data synchronization method provided by the invention can be executed by the terminal. The terminal includes but is not limited to various computers, servers, mobile phones, tablet computers, wearable devices and the like.
Example one
As shown in fig. 1, in an embodiment of the data synchronization method, the method includes the steps of:
s100, acquiring a first synchronization task, and segmenting the first synchronization task to obtain at least one segmentation task.
Specifically, the data synchronization method provided by this embodiment is used for data synchronization between a first data source and a second data source, where each piece of data in the first data source and the second data source has a unique self-increment ID, and a field name is generally set to be an ID of the data, the first synchronization task includes multiple data IDs, and the first synchronization task is used to migrate part of data in the first data source to the second data source, that is, the data ID included in the first synchronization task exists in the first data source and does not exist in the second data source. Since the number of data to be migrated in the first synchronization task is large, in order to increase the migration speed, as shown in fig. 2, in this embodiment, a fragmentation mechanism is used to increase the processing speed, that is, a plurality of threads are used to respectively execute each fragmentation task. Specifically, the first synchronization task is fragmented to obtain at least one fragmentation task, and each fragmentation task includes a plurality of continuous data IDs. The number of the fragmentation tasks can be set according to actual needs, and the number of the fragmentation tasks directly influences the concurrent processing speed of data migration.
The slicing may be performed on the first synchronization task to obtain each slicing task, and the data IDs included in the first synchronization task may be divided in equal proportion, that is, the number of the data IDs included in each slicing task is equal, or the slicing tasks may be divided unevenly.
In this embodiment, multiple threads respectively process each of the fragmentation tasks, specifically, each fragmentation task corresponds to one migration plan, and each time the fragmentation task is processed, a migration plan corresponding to the current fragmentation task is obtained. That is, the data synchronization method provided in this embodiment further includes the steps of:
s200, generating a migration plan according to each fragmentation task, and when a target fragmentation task in the at least one fragmentation task is executed, respectively reading corresponding target data from a first data source according to a data ID in the migration plan corresponding to the target fragmentation task.
Specifically, the generating a migration plan according to each slicing task includes:
detecting a distributed lock when the number of the generated migration plans is less than the number of the fragmentation tasks;
and generating a migration plan according to the fragmentation tasks for obtaining the distributed locks, and repeatedly executing the step of detecting the distributed locks until the generated migration plan is equal to the number of the fragmentation tasks.
As shown in fig. 3, in order to avoid that multiple fragmentation tasks execute the generation of the migration plan at the same time, in this embodiment, a distributed lock is introduced, only the fragmentation task that obtains the distributed lock is allowed to execute the operation of generating the migration plan, and the competition of the distributed lock may be completed by using setnx command of Redis.
The main structure of the migration plan is: the task name + the fragmentation index + the start ID + the end ID, because the data ID in each fragmentation task is continuous, the start ID in the migration plan is the ID of the first piece of data currently processed by the fragmentation task corresponding to the migration plan, and the start ID is automatically updated in the data migration process, and the end ID in the migration plan is the ID of the last piece of data in the corresponding fragmentation task, and the end ID is kept unchanged in the data migration process, when the end ID is empty, the fifth end condition of the fragmentation task is represented, and the fifth end condition can be executed all the time, and when new data is added subsequently, the data migration is continued through the fragmentation task.
The step of respectively reading corresponding target data from a first data source according to the data ID in the migration plan corresponding to the target fragmentation task comprises the following steps:
reading a preset number of pieces of target data from the first data source according to the starting ID of the migration plan corresponding to the target fragmentation task;
wherein an initial value of the start ID of the migration plan corresponding to the target fragmentation task is a first data ID of a plurality of consecutive data IDs included in the target fragmentation task.
In this embodiment, because the amount of data to be migrated is large, a preset number of pieces of data are migrated each time, specifically, a preset number of pieces of target data are read from the first data source according to a start ID of a migration plan corresponding to the target fragmentation task, where the preset number may be set according to an actual situation, for example, 1w, 2w, and the like, and because data IDs processed by the target fragmentation task are continuous, a preset number of pieces of data IDs may be sequentially obtained according to the start ID, and then data corresponding to the read data ID is searched for in the first data source, so as to obtain a preset number of pieces of the target data. After the preset number of pieces of target data are read, writing the preset number of pieces of target data into the second data source in batch to complete synchronization, which will be specifically described in step S300.
S300, synchronizing each read target data to a second data source.
After each target data is read in step S200, each target data is synchronized to the second data source in a batch writing manner, specifically, each read target data is synchronized to the second data source, including the steps of:
s310, writing the read target data into the second data source after first preset processing is carried out on the target data;
s320, performing second preset processing on the target data written into the second data source, and updating the starting ID in the target fragmentation task by using the ID corresponding to the last piece of data in each target data;
s330, repeatedly executing the step of respectively reading corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task until the data corresponding to the last data ID in the target fragmentation task is written into the second data source.
For each target data read in step S200, a first preset processing method is called first to perform first preset processing on each target data, specifically, the first preset processing method is a reserved processing method, that is, the first preset processing may be performed or not performed on each target data, in some possible implementation manners, when data migration is performed, operations such as data verification may need to be performed on the data, and at this time, the first preset processing method is rewritten in the implementation class to perform data verification on the target data.
The read target data are subjected to first preset processing and then written into the second data source in batch, if the batch writing is successful, a second preset processing method is called, the written data are subjected to second preset processing, the second preset processing method is also a reserved processing method, namely, the written data can be subjected to second preset processing or not, and when the multi-table cascade migration is carried out, the second preset processing method can be rewritten in the implementation class to further process the target data written into the second data source, so that the multi-table cascade migration is realized. After second preset processing is performed on the target data written into the second data source, updating a migration plan corresponding to the target fragmentation task, specifically, updating the start ID in the migration plan corresponding to the target fragmentation task to a next data ID of a last piece of data in each target data written into the second data source this time. And then, repeatedly executing the step S200, reading a batch of the target data again and writing the batch of the target data into the second data source until the data corresponding to the end ID in the migration plan corresponding to the target fragmentation task is written into the second data source, and finishing the execution of the target fragmentation task.
As shown in fig. 4, if batch writing fails when a batch of read target data is written into the second data source, it is determined by data ID one by one whether the target data already exists in the second data source, the existing target data is filtered, matching writing operation is performed again with the filtered data, if successful, step S320 is performed, otherwise, a failure alarm is sent, and the target fragmentation task is ended.
Taking the example of migrating the user table t _ user (100w pieces of data) of the first data source a (sqlserver) to the user table user of the second data source b (mysql) in a full amount by the data synchronization method provided in this embodiment:
setting the number of the fragmentation tasks to be 10, when the first fragmentation task (with a fragmentation index of 0) starts to execute, firstly, detecting whether a migration plan exists currently through t _ user and the own fragmentation index 0, if not, creating a complete migration plan after competition, that is, creating a migration plan corresponding to each fragmentation task, where all the migration plans may be shown in table 1.
| Task name | Piece index | Start ID | Ending ID |
| t_user | 0 | 1 | 100000 |
| t_user | 1 | 100001 | 200000 |
| t_user | 2 | 200001 | 300000 |
| t_user | 3 | 300001 | 400000 |
| t_user | 4 | 400001 | 500000 |
| t_user | 5 | 500001 | 600000 |
| t_user | 6 | 600001 | 700000 |
| t_user | 7 | 700001 | 800000 |
| t_user | 8 | 800001 | 900000 |
| t_user | 9 | 900001 |
TABLE 1
After each fragmentation task has a migration plan, the migration plan of the fragmentation task is queried again, the start ID is 1, the end ID is 100000, 1000 pieces of data are queried from the t _ user table by using "ID > -1 and ID <100000order by ID" as a query condition, then the data are written into the data source B in batches, and after the writing is successful, the start ID in the migration plan is updated by using the ID of the last piece of data (the value of the ID is 1000), as shown in table 2.
| Task name | Piece index | Start ID | Ending ID |
| t_user | 0 | 1000 | 100000 |
Table 2 when all the historical data is migrated, the migration plan contents are shown in table 3.
TABLE 3
Finally, the user table in the second data source B also has the 100w pieces of data. Only the last fragment is reserved at this time, and the subsequent newly added data is continuously migrated.
After the same data ID exists in the first data source and the second data source, when data in the first data source is updated, the data in the second data source with the same data ID needs to be updated in the same way, so that the data in the first data source and the data in the second data source with the same data ID are consistent. In the data synchronization method provided in this embodiment, two ways are provided for data synchronization corresponding to data IDs existing in both the first data source and the second data source: a timestamp-based implementation and a trigger-based implementation. Specifically, for the implementation of the timestamp, the data synchronization method provided in this embodiment, after synchronizing each read target data into the second data source, further includes the steps of:
s410, acquiring a second synchronous task, wherein the second synchronous task comprises a synchronous timestamp;
s420, searching each first target data ID in the first data source according to the synchronization timestamp, wherein the update timestamp of the data corresponding to the first target data ID is not earlier than the synchronization timestamp;
s430, respectively acquiring a first updating time and a second updating time corresponding to each first target data ID, wherein the first updating time is the updating time of the data corresponding to the first target data ID in the first data source, and the second updating time is the updating time of the data corresponding to the first target data ID in the second data source;
s440, when the first updating time corresponding to the first target data ID is later than the second updating time, reading the data corresponding to the first target data ID from the first data source, performing first preset processing, and updating the data corresponding to the first target data ID in the second data source according to the data subjected to the first preset processing.
In the synchronization method based on the timestamp, the synchronization task includes the synchronization timestamp, and the second synchronization task is executed no earlier than the synchronization timestamp at the current time, as shown in fig. 5, when the second synchronization task is executed, first, data is queried from the first data source, specifically, a first target data ID whose update timestamp is no earlier than the synchronization timestamp is searched in the first data source according to the synchronization timestamp, and a time corresponding to the update timestamp of the data is a time when the data is modified last time. After obtaining each first target data ID, obtaining an update time of data corresponding to the first target data ID in the first data source (that is, obtaining an update timestamp of data corresponding to the first target data ID in the first data source), and obtaining an update time of data corresponding to the first target data ID in the second data source (that is, obtaining an update timestamp of data corresponding to the first target data ID in the second data source), respectively. If the first update time corresponding to the first target data ID is later than the second update time, updating the data corresponding to the first target data ID in the second data source according to the data corresponding to the first target data ID read from the first data source, and in a possible implementation manner, after reading the data corresponding to the first target data ID from the first data source, performing first preset processing on the data and then updating the data corresponding to the second data source according to the processed data. Specifically, as shown in fig. 7, the data in the second data source may be updated in an optimistic lock manner. Similarly, after the data in the second data source is updated according to the data read from the first data source, a second preset processing method may be further invoked for post-processing, which may be specifically referred to the foregoing description of the first preset processing and the second preset processing.
After the data corresponding to the first target data ID in the second data source is updated according to the first preset processed data, the synchronization timestamp of the second synchronization task needs to be updated. Specifically, after the data corresponding to the first target data ID in the second data source is updated according to the data subjected to the first preset processing, the method includes the steps of:
when the update timestamp of the latest data in the first data source is the same as the synchronization timestamp, updating the synchronization timestamp, wherein the time corresponding to the updated synchronization timestamp is later than the time corresponding to the synchronization timestamp before updating;
and when the update time stamp of the data with the latest update time in the first data source is later than the synchronization time stamp, updating the synchronization time stamp into the update time stamp of the data with the latest update time in the first data source.
Data whose update time stamp is not earlier than the synchronization time stamp includes data whose update time stamp is the same as the synchronization time stamp and data whose update time stamp is later than the synchronization time stamp, when the update time stamp of the latest update time data in the first data source is later than the synchronization time stamp, the synchronization time stamp is updated to the update time stamp of the latest update time data in the first data source, and when the update time stamp of the latest update time data in the first data source is equal to the synchronization time stamp, if the synchronization time stamp is updated to the update time stamp of the latest update time data in the first data source, the synchronization time stamp in the second synchronization task remains unchanged, which may cause data to be synchronized without new modification to be executed next time because the synchronization time stamp is not updated, the data of the current timestamp is queried again for secondary synchronization, so as shown in fig. 6, when the update timestamp of the data with the latest update time in the first data source is the same as the synchronization timestamp, the synchronization timestamp may be updated to a time later than but close to the synchronization timestamp before updating, specifically, a step value may be set, the step value may be the minimum time unit of the system, for example, 1 microsecond, and the current synchronization timestamp is added to the step value to obtain the updated synchronization timestamp.
Taking the example of synchronizing the updated data in the user table t _ user of the first data source a (sqlserver) to the user table user of the second data source b (mysql) by the data synchronization method provided in this embodiment:
the current data in data source a and data source B are shown in tables 4 and 5.
t _ user table data:
| ID | name (I) | Rank of | Updating timestamps |
| ... | ... | ... | ... |
| 111 | Zhang San | Signing writer | 2021-01-01 12:05:23.352 |
| ... | ... | ... | ... |
Table 4user table data:
| ID | name (I) | Rank of | Updating timestamps |
| ... | ... | ... | ... |
| 111 | Zhang San | Writer for doing homework | 2021-01-01 09:10:00.123 |
| ... | ... | ... | ... |
Table 5 the schedule of the synchronization tasks is shown in table 6.
| Task name | Synchronizing timestamps |
| t_user | 2021-01-01 00:00:00.000 |
TABLE 6
The synchronization timestamp in the synchronization task is first obtained. Whether data with the same time stamp exist is checked, namely data query is carried out from a t _ user table by taking 'last _ mod _ date as' 2021-01-0100: 00:00.000 'order by last _ mod _ date' as a query condition, data in the user table is sequentially queried through id, update time stamps of the data and the user table are compared, and if the update time stamp of the t _ user table is later, the data corresponding to the user table is directly covered by the update time stamp. The timestamp is then updated for +1 millisecond.
The schedule of the synchronization task at this time is shown in table 7.
| Task name | Synchronizing timestamps |
| t_user | 2021-01-01 00:00:00.001 |
TABLE 7
And then processing the data after the previous time stamp, namely performing data query from the t _ user table by taking "last _ mod _ date > '2021-01-0100: 00: 00.000' order by last _ mod _ date" as a query condition, wherein the subsequent processing procedures are consistent in the process. Finally, the update time of the last piece of data (assumed to be 2021-01-0112: 05:23.352) is used as the next synchronization timestamp of the synchronization schedule, and the schedule of the synchronization task is shown in Table 8.
| Task name | Synchronizing timestamps |
| t_user | 2021-01-01 12:05:23.352 |
After the update of table 8 is completed, the user table data is updated as shown in table 9.
TABLE 9
Specifically, for the implementation of the trigger, the data synchronization method provided in this embodiment, after synchronizing each read target data into the second data source, further includes the steps of:
s510, acquiring a third synchronous task, wherein the third synchronous task comprises a synchronous schedule;
s520, when the data in the first data source is changed, recording the data ID of the changed data into the synchronous schedule and marking the state as successful processing;
s530, reading a second target data ID from the synchronization schedule, wherein the second target data ID is a data ID in the synchronization schedule, the state of which is successful in processing, if the update time stamp of the data corresponding to the second target data ID in the first data source is later than that of the data corresponding to the second data source, reading the data corresponding to the second target data ID from the first data source, performing first preset processing, and updating the data corresponding to the second target data ID in the second data source according to the data subjected to the first preset processing;
s540, modifying the state of the second target data ID in the synchronization schedule into processed state;
and S550, clearing the data ID with the processed state in the synchronization schedule.
In a synchronization mode based on a trigger, a third synchronization task is created, the third synchronization task includes a synchronization schedule, a trigger afer update trigger is set, when data in the first data source is changed, the trigger afer update trigger is automatically triggered and written into the synchronization schedule, and a main structure of the synchronization schedule is as follows: task name (typically source table name) + unique key (data ID) + update timestamp + processing status (default as "processing successful"). As shown in fig. 8, when the third sync task is executed, the sync schedule corresponding to the third sync task is read first, and the sync schedule may be processed once every preset time, specifically, each time the sync schedule is processed, as shown in fig. 9, a second target data ID whose processing status is successful is read from the sync schedule, and an update timestamp of data corresponding to the second target data ID in the first data source is compared with an update timestamp of data corresponding to the second target data ID in the second data source, and if the update timestamp of data corresponding to the second target data ID in the first data source is later than the update timestamp of data corresponding to the second data source, the data corresponding to the second target data ID is read from the first data source and first preset processing is performed, and updating the data corresponding to the second target data ID in the second data source according to the data subjected to the first preset processing, namely, performing first preset processing (optional) on the data read from the first data source according to the second target data ID and then covering the data corresponding to the second target data ID in the second data source. After the data is written into the second data source, second preset processing can be selectively performed. And after the data corresponding to the second target data ID in the second data source is updated, modifying the state of the second target data ID in the synchronization schedule into processed state. And after processing the synchronous schedule every time, clearing the data ID with the processed state in the synchronous schedule.
Taking the example of synchronizing the updated data in the user table t _ user of the first data source a (sqlserver) to the user table user of the second data source b (mysql) by the data synchronization method provided in this embodiment:
the data of the current t _ user table and the user table are shown in table 10 and table 11.
TABLE 11
Firstly, an After Update trigger is created on a t _ user table, and when t _ user data is changed as follows, the level of zhang san (ID 111) is updated from a signed writer to a platinum signed writer:
the t _ user table data is shown in table 12.
TABLE 12
Changes are automatically recorded into the synchronization plan as shown in table 13.
The synchronization task firstly acquires all the synchronization plans with the processing states of 'waiting for processing', sequentially uses the unique key to process the data in the user table of the data source B, compares the time stamps of the data and the user table, and if the time stamp of the synchronization plan is later, inquires the latest data from the t _ user table and uses the latest data to cover the data corresponding to the user table.
The post-synchronization user table data is shown in table 14.
Watch 13
Table 14 and updates the processing status of the synchronization plan to "success", as shown in table 15.
Table 15 finally clears all synchronization plans whose processing status is "successful", as shown in table 16.
| Task name | Unique key | Synchronizing timestamps | Processing state |
TABLE 16
The synchronization mode based on the time stamp can realize timely synchronization of data, is suitable for synchronization of data sources with small size, and is more frequent in data updating frequency and more suitable for the synchronization mode based on the trigger for data sources with large size.
In summary, this embodiment provides a data synchronization method, where for a synchronization task that needs to synchronize multiple pieces of data from a first data source to a second data source, the synchronization task is fragmented, a migration plan is generated for each fragmentation task, when the migration plan is executed, corresponding data is read from the first data source and synchronized to the second data source according to a data ID in the fragmentation task, in the whole process, only the steps of searching, reading, and writing need to be performed according to the data ID, a general language can be used for implementation, and data synchronization between heterogeneous data sources can be implemented without depending on the structure of the first data source or the second data source.
It should be understood that, although the steps in the flowcharts shown in the figures of the present specification are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, databases, or other media used in embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
Example two
Based on the foregoing embodiments, the present invention further provides a data synchronization apparatus, as shown in fig. 10, where the data synchronization apparatus includes:
the task obtaining module is configured to obtain a first synchronization task, fragment the first synchronization task, and obtain at least one fragment task, where the first synchronization task includes multiple data IDs, and each fragment task includes multiple consecutive data IDs, which is specifically the same as that in embodiment one;
a data reading module, configured to generate a migration plan according to each fragmentation task, and when a migration plan corresponding to a target fragmentation task in the at least one fragmentation task is executed, respectively read corresponding target data from a first data source according to a data ID in the target fragmentation task, which is specifically described in embodiment one;
and a synchronization module, configured to synchronize each read target data to a second data source, as described in the first embodiment.
EXAMPLE III
Based on the above embodiments, the present invention further provides a terminal, as shown in fig. 11, where the terminal includes a processor 10 and a memory 20. Fig. 11 shows only some of the components of the terminal, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
The memory 20 may in some embodiments be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the terminal. Further, the memory 20 may also include both an internal storage unit and an external storage device of the terminal. The memory 20 is used for storing application software installed in the terminal and various data. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a data synchronization program 30, and the data synchronization program 30 can be executed by the processor 10 to implement the data synchronization method of the present application.
The processor 10 may be a Central Processing Unit (CPU), microprocessor or other chip in some embodiments, and is used for running program codes stored in the memory 20 or Processing data, such as executing the data synchronization method.
In one embodiment, when the processor 10 executes the data synchronization program 30 in the memory 20, the following steps are implemented:
acquiring a first synchronization task, and fragmenting the first synchronization task to obtain at least one fragmentation task, wherein the first synchronization task comprises a plurality of data IDs, and each fragmentation task comprises a plurality of continuous data IDs;
generating a migration plan according to each fragmentation task, and when a target fragmentation task in the at least one fragmentation task is executed, respectively reading corresponding target data from a first data source according to a data ID in the migration plan corresponding to the target fragmentation task;
and synchronizing each read target data to a second data source.
Wherein the generating a migration plan according to each of the slicing tasks includes:
detecting a distributed lock when the number of the generated migration plans is less than the number of the fragmentation tasks;
and generating a migration plan according to the fragmentation tasks for obtaining the distributed locks, and repeatedly executing the step of detecting the distributed locks until the generated migration plan is equal to the number of the fragmentation tasks.
Wherein, the reading the corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task includes:
reading a preset number of pieces of target data from the first data source according to the starting ID of the migration plan corresponding to the target fragmentation task;
wherein an initial value of the start ID of the migration plan corresponding to the target fragmentation task is a first data ID of a plurality of consecutive data IDs included in the target fragmentation task.
Wherein, the step of synchronizing each read target data to a second data source includes:
performing first preset processing on each read target data and writing the target data into the second data source;
performing second preset processing on the target data written into the second data source, and updating the starting ID in the target fragmentation task by using the ID corresponding to the last piece of data in each target data;
and repeatedly executing the step of reading corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task until the data corresponding to the end ID in the migration plan corresponding to the target fragmentation task is written into the second data source.
After synchronizing each read target data into the second data source, the method further includes:
acquiring a second synchronous task, wherein the second synchronous task comprises a synchronous timestamp;
searching each first target data ID in the first data source according to the synchronization timestamp, wherein the update timestamp of the data corresponding to the first target data ID is not earlier than the synchronization timestamp;
respectively acquiring a first updating time and a second updating time corresponding to each first target data ID, wherein the first updating time is the updating time of data corresponding to the first target data ID in the first data source, and the second updating time is the updating time of data corresponding to the first target data ID in the second data source;
when the first updating time corresponding to the first target data ID is later than the second updating time, reading the data corresponding to the first target data ID from the first data source, performing first preset processing, and updating the data corresponding to the first target data ID in the second data source according to the data subjected to the first preset processing.
After updating the data corresponding to the first target data ID in the second data source according to the data subjected to the first preset processing, the method includes:
when the update timestamp of the latest data in the first data source is the same as the synchronization timestamp, updating the synchronization timestamp, wherein the time corresponding to the updated synchronization timestamp is later than the time corresponding to the synchronization timestamp before updating;
and when the update time stamp of the data with the latest update time in the first data source is later than the synchronization time stamp, updating the synchronization time stamp into the update time stamp of the data with the latest update time in the first data source.
After synchronizing each read target data into the second data source, the method further includes:
acquiring a third synchronous task, wherein the third synchronous task comprises a synchronous schedule;
when the data in the first data source is changed, recording the data ID of the changed data into the synchronous schedule and marking the state as successful processing;
reading a second target data ID from the synchronization schedule, wherein the second target data ID is a data ID in the synchronization schedule, the state of which is successful in processing, if the update time stamp of the data corresponding to the second target data ID in the first data source is later than that of the data corresponding to the second data source, reading the data corresponding to the second target data ID from the first data source, performing first preset processing, and updating the data corresponding to the second target data ID in the second data source according to the data subjected to the first preset processing;
modifying the state of the second target data ID in the synchronization schedule to be processed;
and clearing the data ID with the processed state in the synchronization schedule.
Example four
The present invention also provides a computer readable storage medium in which one or more programs are stored, the one or more programs being executable by one or more processors to implement the steps of the data synchronization method as described above.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for synchronizing data, the method comprising:
acquiring a first synchronization task, and fragmenting the first synchronization task to obtain at least one fragmentation task, wherein the first synchronization task comprises a plurality of data IDs, and each fragmentation task comprises a plurality of continuous data IDs;
generating a migration plan according to each fragmentation task, and when a target fragmentation task in the at least one fragmentation task is executed, respectively reading corresponding target data from a first data source according to a data ID in the migration plan corresponding to the target fragmentation task;
and synchronizing each read target data to a second data source.
2. The data synchronization method according to claim 1, wherein the generating a migration plan according to each of the slicing tasks comprises:
detecting a distributed lock when the number of the generated migration plans is less than the number of the fragmentation tasks;
and generating a migration plan according to the fragmentation tasks for obtaining the distributed locks, and repeatedly executing the step of detecting the distributed locks until the generated migration plan is equal to the number of the fragmentation tasks.
3. The data synchronization method according to claim 1, wherein the reading the corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task respectively includes:
reading a preset number of pieces of target data from the first data source according to the starting ID of the migration plan corresponding to the target fragmentation task;
wherein an initial value of the start ID of the migration plan corresponding to the target fragmentation task is a first data ID of a plurality of consecutive data IDs included in the target fragmentation task.
4. The data synchronization method according to claim 3, wherein the synchronizing the read target data into the second data source comprises:
performing first preset processing on each read target data and writing the target data into the second data source;
performing second preset processing on the target data written into the second data source, and updating the starting ID in the target fragmentation task by using the ID corresponding to the last piece of data in each target data;
and repeatedly executing the step of reading corresponding target data from the first data source according to the data ID in the migration plan corresponding to the target fragmentation task until the data corresponding to the end ID in the migration plan corresponding to the target fragmentation task is written into the second data source.
5. The data synchronization method according to claim 1, further comprising, after synchronizing each of the read target data into the second data source:
acquiring a second synchronous task, wherein the second synchronous task comprises a synchronous timestamp;
searching each first target data ID in the first data source according to the synchronization timestamp, wherein the update timestamp of the data corresponding to the first target data ID is not earlier than the synchronization timestamp;
respectively acquiring a first updating time and a second updating time corresponding to each first target data ID, wherein the first updating time is the updating time of data corresponding to the first target data ID in the first data source, and the second updating time is the updating time of data corresponding to the first target data ID in the second data source;
when the first updating time corresponding to the first target data ID is later than the second updating time, reading the data corresponding to the first target data ID from the first data source, performing first preset processing, and updating the data corresponding to the first target data ID in the second data source according to the data subjected to the first preset processing.
6. The data synchronization method according to claim 5, wherein after the updating of the data corresponding to the first target data ID in the second data source according to the data after the first preset processing, the method includes:
when the update timestamp of the latest data in the first data source is the same as the synchronization timestamp, updating the synchronization timestamp, wherein the time corresponding to the updated synchronization timestamp is later than the time corresponding to the synchronization timestamp before updating;
and when the update time stamp of the data with the latest update time in the first data source is later than the synchronization time stamp, updating the synchronization time stamp into the update time stamp of the data with the latest update time in the first data source.
7. The data synchronization method according to claim 1, wherein after the synchronizing the read target data into the second data source, further comprises:
acquiring a third synchronous task, wherein the third synchronous task comprises a synchronous schedule;
when the data in the first data source is changed, recording the data ID of the changed data into the synchronous schedule and marking the state as successful processing;
reading a second target data ID from the synchronization schedule, wherein the second target data ID is a data ID in the synchronization schedule, the state of which is successful in processing, if the update time stamp of the data corresponding to the second target data ID in the first data source is later than that of the data corresponding to the second data source, reading the data corresponding to the second target data ID from the first data source, performing first preset processing, and updating the data corresponding to the second target data ID in the second data source according to the data subjected to the first preset processing;
modifying the state of the second target data ID in the synchronization schedule to be processed;
and clearing the data ID with the processed state in the synchronization schedule.
8. A data synchronization apparatus, comprising:
the task obtaining module is used for obtaining a first synchronization task and fragmenting the first synchronization task to obtain at least one fragmentation task, wherein the first synchronization task comprises a plurality of data IDs, and each fragmentation task comprises a plurality of continuous data IDs;
the data reading module is used for generating a migration plan according to each fragmentation task, and when the migration plan corresponding to a target fragmentation task in the at least one fragmentation task is executed, reading corresponding target data from a first data source according to a data ID in the target fragmentation task;
and the synchronization module is used for synchronizing each read target data to a second data source.
9. A terminal, characterized in that the terminal comprises: a processor, a computer readable storage medium communicatively connected to the processor, the computer readable storage medium adapted to store a plurality of instructions, the processor adapted to invoke the instructions in the computer readable storage medium to perform the steps of implementing the data synchronization method of any of the preceding claims 1-7.
10. A computer-readable storage medium, storing one or more programs, the one or more programs being executable by one or more processors to perform the steps of the data synchronization method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110632999.1A CN113392088B (en) | 2021-06-07 | 2021-06-07 | Data synchronization method, device, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110632999.1A CN113392088B (en) | 2021-06-07 | 2021-06-07 | Data synchronization method, device, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113392088A true CN113392088A (en) | 2021-09-14 |
| CN113392088B CN113392088B (en) | 2024-03-26 |
Family
ID=77618472
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110632999.1A Active CN113392088B (en) | 2021-06-07 | 2021-06-07 | Data synchronization method, device, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392088B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019148722A1 (en) * | 2018-02-01 | 2019-08-08 | 平安科技（深圳）有限公司 | Electronic device, data migrating and calling method and storage medium |
| CN110532247A (en) * | 2019-08-28 | 2019-12-03 | 北京皮尔布莱尼软件有限公司 | Data migration method and data mover system |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN111475483A (en) * | 2019-01-24 | 2020-07-31 | 阿里巴巴集团控股有限公司 | Database migration method and device and computing equipment |
| CN111708763A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Data migration method and device of fragment cluster and fragment cluster system |
| CN111858079A (en) * | 2020-07-27 | 2020-10-30 | 北京达佳互联信息技术有限公司 | Distributed lock migration method and device, electronic equipment and storage medium |
-
2021
- 2021-06-07 CN CN202110632999.1A patent/CN113392088B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019148722A1 (en) * | 2018-02-01 | 2019-08-08 | 平安科技（深圳）有限公司 | Electronic device, data migrating and calling method and storage medium |
| CN111475483A (en) * | 2019-01-24 | 2020-07-31 | 阿里巴巴集团控股有限公司 | Database migration method and device and computing equipment |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN110532247A (en) * | 2019-08-28 | 2019-12-03 | 北京皮尔布莱尼软件有限公司 | Data migration method and data mover system |
| CN111708763A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Data migration method and device of fragment cluster and fragment cluster system |
| CN111858079A (en) * | 2020-07-27 | 2020-10-30 | 北京达佳互联信息技术有限公司 | Distributed lock migration method and device, electronic equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| WEIZHI XU;: "Fast and scalable lock methods for video coding on many-core architecture", 《JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION》, no. 10, 15 October 2014 (2014-10-15) * |
| 薛小燕;任宏德;: "基于全局目录的集中型数据库分布式加锁仿真", 计算机仿真, no. 04, 15 April 2020 (2020-04-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113392088B (en) | 2024-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11429641B2 (en) | Copying data changes to a target database | |
| CN108536752B (en) | Data synchronization method, device and equipment | |
| US9779128B2 (en) | System and method for massively parallel processing database | |
| CN107590207B (en) | Data synchronization method and device and electronic equipment | |
| CN112286941B (en) | Big data synchronization method and device based on Binlog + HBase + Hive | |
| US9384202B1 (en) | Gateway module to access different types of databases | |
| CN107391634B (en) | Data migration method and device | |
| CN111291023A (en) | Data migration method, system, device and medium | |
| CN110795499A (en) | Cluster data synchronization method, device and equipment based on big data and storage medium | |
| CN103106138A (en) | Method and device for synchronization of test case and test script | |
| CN105808653A (en) | User label system-based data processing method and device | |
| CN111680017A (en) | Data synchronization method and device | |
| CN110955672B (en) | Multi-version support method and system for optimistic concurrency control | |
| CN115185955A (en) | Data lake data processing method and system | |
| CN114385760A (en) | Method and device for real-time synchronization of incremental data, computer equipment and storage medium | |
| CN109471901B (en) | Data synchronization method and device | |
| CN114490856A (en) | Database WAL (Web independent language) disk-dropping method and system based on IOURING technology | |
| CN113392088A (en) | Data synchronization method, device, terminal and storage medium | |
| CN111522827A (en) | Data updating method and device and electronic equipment | |
| CN112749147A (en) | Method and device for merging and migrating data tables in database | |
| CN112612647B (en) | Log parallel replay method, device, equipment and storage medium | |
| CN113704267A (en) | Data query method, system, equipment and storage medium based on elastic search | |
| CN112948485B (en) | Question-answer data synchronization method, device, system, server and storage medium | |
| CN114756562A (en) | Metadata synchronization method and device, electronic equipment and storage medium | |
| CN117573641A (en) | Job-based large-table data smooth migration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Cui Chenglong Inventor before: Dang Binhui |
|
| CB03 | Change of inventor or designer information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |