CN117389989A - Data synchronization method, apparatus, computer device, storage medium, and program product - Google Patents
Data synchronization method, apparatus, computer device, storage medium, and program product Download PDFInfo
- Publication number
- CN117389989A CN117389989A CN202311339171.2A CN202311339171A CN117389989A CN 117389989 A CN117389989 A CN 117389989A CN 202311339171 A CN202311339171 A CN 202311339171A CN 117389989 A CN117389989 A CN 117389989A
- Authority
- CN
- China
- Prior art keywords
- data
- current operation
- log
- statement
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 94
- 238000004590 computer program Methods 0.000 claims abstract description 43
- 230000008569 process Effects 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 25
- 230000005012 migration Effects 0.000 description 17
- 238000013508 migration Methods 0.000 description 17
- 230000001360 synchronised effect Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to the field of data processing, and in particular, to a data synchronization method, apparatus, computer device, storage medium, and computer program product. The method comprises the following steps: acquiring a first data operation log of an original database in a data full synchronization process; acquiring a second data operation log of the target database after online; acquiring a current operation statement from a first data operation log, and determining current operation data corresponding to the current operation statement; inquiring whether a target operation statement for operating the current operation data exists in the second operation log or not; when the second operation log does not have a target operation statement for operating the current operation data, synchronizing the current operation data to a target database; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log. By adopting the method, the accuracy can be improved.
Description
Technical Field
The present application relates to the field of data processing technology, and in particular, to a data synchronization method, apparatus, computer device, storage medium, and computer program product.
Background
Database migration is a problem frequently encountered in project development, such as migration of some old systems, performance of the database encountering bottlenecks, the database being used being too expensive, and the like, and the database migration is required.
In the traditional technology, in the process of data migration from an old system to a new system, the operation of the system is stopped, and after the migration is completed, the use of the system is realized, so that the development of normal business is influenced. Or the data migration is realized without shutdown, when the increment occurs, the incremental data is recorded in the migration data, after the first migration of the total data is completed, all the total data needs to be scanned to determine the incremental data, the scanning time is long, and after the scanning is completed, the data in the new system is covered by the migration data.
Therefore, the traditional data migration method is easy to cause the influence on normal service development or easy to cause the coverage of data updated by a new system.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data synchronization method, apparatus, computer device, computer readable storage medium, and computer program product that can avoid data coverage of a new system.
In a first aspect, the present application provides a data synchronization method, the method including:
acquiring a first data operation log of an original database in a data full synchronization process;
acquiring a second data operation log of the target database after online;
acquiring a current operation statement from the first data operation log, and determining current operation data corresponding to the current operation statement;
inquiring whether the second operation log exists or not, and operating the current operation data to obtain a target operation statement;
when the second operation log does not contain a target operation statement for operating the current operation data, synchronizing the current operation data to a target database;
and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, the method further comprises:
when a target operation statement for operating the current operation data exists in the second operation log, processing the current operation data in an original database based on the target operation statement, deleting the target operation statement in the second operation log, and adding the target operation statement into the first operation log;
and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, after the operation statement does not exist in the first data operation log, the method includes:
and stopping updating the second data operation log of the target database after the target database is online.
In one embodiment, after the processing the current operation data in the original database based on the target operation statement, the method further includes:
acquiring the update times corresponding to the current operation data, wherein the update times are used for representing the update times of the current operation data based on a target operation statement;
Outputting the current operation data when the update times are greater than or equal to a preset value;
receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
In one embodiment, before the obtaining the current operation statement from the first data operation log, the method further includes:
arranging the operation sentences in the first data operation log according to a time sequence;
the obtaining the current operation statement from the first data operation log includes:
and acquiring a current operation statement from the first data operation log based on the time sequence.
In one embodiment, the acquiring the first data operation log of the original database in the data full synchronization process includes:
receiving a database full synchronization instruction, and synchronizing data in the original database into the target database based on the full synchronization instruction;
and in the process of synchronizing the data in the original database to the target database, acquiring an operation statement aiming at the data in the original database, and storing the operation statement into a first data operation log.
In a second aspect, the present application further provides a data synchronization device, the device including:
the first data operation log acquisition module is used for acquiring a first data operation log of the original database in the data full synchronization process;
the second data operation log acquisition module is used for acquiring a second data operation log of the target database after being online;
the current operation data determining module is used for acquiring a current operation statement from the first data operation log and determining current operation data corresponding to the current operation statement;
the query module is used for querying whether the second operation log exists or not, and performing operation on the current operation data;
the increment synchronization module is used for synchronizing the current operation data to a target database when no target operation statement for operating the current operation data exists in the second operation log;
and the circulation module is used for acquiring a next operation statement from the first data operation log as a current operation statement, and continuously executing the step of determining the current operation data corresponding to the current operation statement until no operation statement exists in the first data operation log.
In a third aspect, the present application also provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the method of any one of the embodiments described above when the computer program is executed by the processor.
In a fourth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of any of the embodiments described above.
In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any of the embodiments described above.
According to the data synchronization method, the device, the computer equipment, the storage medium and the computer program product, the current operation statement is obtained from the first data operation log, the current operation data corresponding to the current operation statement is determined, whether the second operation log exists or not is queried, if the second operation log does not exist, the fact that the operation on the current operation data does not exist in the target database is indicated, at the moment, the current operation data is synchronized to the target database, data coverage of the target database is not caused, and accuracy of the data is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for a person having ordinary skill in the art.
FIG. 1 is an application environment diagram of a data synchronization method in one embodiment;
FIG. 2 is a flow chart of a data synchronization method in one embodiment;
FIG. 3 is a flow chart of a full data synchronization step in one embodiment;
FIG. 4 is a timing diagram of a data synchronization method in one embodiment;
FIG. 5 is a block diagram of a data synchronization device in one embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The data synchronization method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the original database 104 and the target database 106 are in communication. The data in the original database 104 is first full-amount synchronized into the target database 106, and the original database 104 is used as an external database to receive external operations, and performs data update, and stores a first data operation log of the original database 104 in the data full-amount synchronization process, and after the full-amount synchronization is finished, the target database 106 is online, and is used as an external database to receive external operations, perform data update, and store a second data operation log of the target database 106 after the online. The terminal 102 stores incremental data into the target database 106 based on a first data operation log of the original database 104 in a data full synchronization process, wherein in order to avoid covering the data in the target database 106, in the incremental data synchronization process, the terminal 102 acquires a current operation statement from the first data operation log and determines current operation data corresponding to the current operation statement; inquiring whether a target operation statement for operating the current operation data exists in the second operation log or not; when the second operation log does not contain a target operation statement for operating the current operation data, synchronizing the current operation data to a target database; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log. When a target operation sentence for operating the current operation data exists in the second operation log, processing the current operation data in the original database based on the target operation sentence, deleting the target operation sentence in the second operation log, and adding the target operation sentence into the first operation log; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
According to the data synchronization method, the current operation statement is obtained from the first data operation log, the current operation data corresponding to the current operation statement is determined, whether the second operation log is provided with the target operation statement for operating the current operation data is queried, if the second operation log is not provided with the target operation statement for operating the current operation data, the fact that the target database is not provided with the operation for the current operation data is indicated, at the moment, the current operation data is synchronized to the target database, data coverage of the target database is not caused, and data accuracy is guaranteed.
Wherein, the original database 102 and the target database 104 may be the same database or different databases, and the types of the databases are not particularly limited herein. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like.
In an exemplary embodiment, as shown in fig. 2, a data synchronization method is provided, and an example of application of the method to the terminal in fig. 1 is described, including the following steps S202 to S212. Wherein:
S202: and acquiring a first data operation log of the original database in the data full synchronization process.
The original database corresponds to an old system, and stores the data of the whole amount of the old system, wherein the data of the whole amount is synchronized, namely the operation of synchronizing the data of the whole amount in the original database to the target database. The first data operation log is a record of data operation of the original database in the data full synchronization process, wherein the original database provides service to the outside in the data full synchronization process to generate adding, deleting and modifying operation of data, and the first data operation log is used for storing the adding, deleting and modifying operation, wherein database operation sentences corresponding to the adding, deleting and modifying operation, such as sql sentences, are generally stored.
S204: and acquiring a second data operation log of the target database after the target database is online, and acquiring a current operation statement from the first data operation log.
The target database corresponds to a new system, and stores data which is synchronized in full from the original database and data generated by data operation after the target database is online. The second data operation log is a record of the adding, deleting and modifying operation on the data after the target database is online, i.e. after the target database provides services to the outside, wherein database operation sentences corresponding to the adding, deleting and modifying operation, such as sql sentences, are generally stored.
S206: and determining the current operation data corresponding to the current operation statement.
The current operation statement is in the first data operation log.
In one embodiment, before obtaining the current operation statement from the first data operation log, the method further includes: arranging the operation sentences in the first data operation log according to a time sequence; obtaining a current operation statement from a first data operation log, including: the current operation statement is obtained from the first data operation log based on the time sequence.
The operation sentences in the first data operation log are arranged according to a certain sequence, so that the operation sentences in the first data operation log are ordered according to a time sequence to avoid data confusion, the current operation sentences are acquired based on the time sequence, and the current operation data corresponding to the current operation sentences are determined. The current operation data corresponds to the current operation statement, i.e., the data operated on by the current operation statement.
S208: and inquiring whether the second operation log exists or not, and operating the current operation data by using the target operation statement.
S210: and synchronizing the current operation data to the target database when the target operation statement for operating the current operation data does not exist in the second operation log.
After the target database is online and before the changed data is not synchronized to the target database in the process of synchronizing the data volume, that is, before the current operation data is not synchronized to the target database, the target database may already have an operation on the current operation data, and when the current operation data changed in the process of synchronizing the data volume is synchronized to the target database at this time, the operation on the current operation data in the target database is covered.
If the second operation log does not have the target operation statement for operating the current operation data, that is, the current operation data in the target database or the current operation data synchronized in the full-scale synchronization process, the current operation data can be directly synchronized to the target database, optionally, the target database can be processed based on the current operation statement, so as to realize the operation of synchronizing the current operation data to the target database.
S212: and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
After the current operation statement processing is completed, the terminal continues to acquire the next operation statement from the first data operation log as the current operation statement until no operation statement exists in the first data operation log, and then the data synchronization processing is completed.
In one optional embodiment, when a target operation sentence for operating the current operation data exists in the second operation log, processing the current operation data in the original database based on the target operation sentence, deleting the target operation sentence in the second operation log, and adding the target operation sentence into the first operation log; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
When there is a target operation sentence for operating the current operation data in the second operation log, it is indicated that the current operation data is processed in the target database, for example, an add-delete operation, which is necessarily an operation after the target database is online and is necessarily an operation after full synchronization, so that the target operation sentence is acquired from the second operation log, and the current operation data in the original database is processed based on the target operation sentence, so as to obtain the latest current operation data, and in order to synchronize the latest current operation data in the target database, the target operation sentence is added in the first operation log, and the target operation sentence in the second operation log is deleted, so that the target operation sentence can be acquired when the first operation log is traversed, and the processing as above is performed, so that the synchronization of the data is realized.
In one alternative embodiment, the target operation sentences in the second operation log may include at least one, and when the target operation sentences in the second operation log include two or more target operation sentences, the two or more target operation sentences are sorted according to time sequence, and the original database is operated based on the sorted two or more target operation sentences.
According to the data synchronization method, the current operation statement is obtained from the first data operation log, the current operation data corresponding to the current operation statement is determined, whether the second operation log is provided with the target operation statement for operating the current operation data is queried, if the second operation log is not provided with the target operation statement for operating the current operation data, the fact that the target database is not provided with the operation for the current operation data is indicated, at the moment, the current operation data is synchronized to the target database, data coverage of the target database is not caused, and data accuracy is guaranteed.
In one alternative embodiment, until after the first data operation log has no operation statement, the method includes: and stopping updating the second data operation log of the target database after the target database is online.
When no operation statement exists in the first data operation log, the data synchronization from the original database to the target database is completed, and then the updating of the second data operation log of the target database after being online can be stopped, wherein the stopping updating only aims at the updating of the target operation statement, but other contents in the log are updated as usual, so that the storage amount of the second data operation log is reduced.
In the above embodiment, when no operation statement exists in the first data operation log, updating of the target operation statement in the second data operation log is stopped to reduce the storage amount of the second data operation log.
In one embodiment, after processing the current operation data in the original database based on the target operation statement, the method further includes: acquiring the update times corresponding to the current operation data, wherein the update times are used for representing the update times of the current operation data based on the target operation statement; when the update times are greater than or equal to a preset value, outputting current operation data; and receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
The update times are used to indicate the times that the current operation data is updated based on the target operation statement, that is, the times that the current operation data is frequently updated at present are detected in the second operation log, that is, the times that the current operation data is frequently updated are indicated, therefore, when the update times are greater than or equal to a preset value, the current operation data can be output so that a user can directly update the current operation data, for example, the user can inquire about the operation statement aiming at the current operation data point, then the latest value of the current operation data is determined based on the operation statement, and the current operation data in the target database is directly reset.
After the user performs the processing, deleting the operation sentences aiming at the current operation data in the first operation log and the second operation log so as to avoid repeated operation.
In the above embodiment, in order to avoid repeated operations, manual processing is introduced, so that the processing efficiency is improved.
In one embodiment, before obtaining the first data operation log of the original database in the data full synchronization process, the method includes: receiving a database full synchronization instruction, and synchronizing data in an original database into a target database based on the full synchronization instruction; in the process of synchronizing data in the original database to the target database, operation sentences aiming at the data in the original database are obtained, and the operation sentences are stored into a first data operation log.
Specifically, referring to fig. 3, fig. 3 is a flowchart of a database full synchronization step in an embodiment, in which a database full synchronization instruction is received, and data in an original database is synchronized to a target database, where operations such as cleaning, converting, integrating, etc. are performed on the data based on a data rule during synchronization, and the processed data is synchronized to the target database.
Specifically, referring to fig. 4, fig. 3 is a timing chart of a database synchronization method in an embodiment, in this embodiment, a first operation log table for recording an adding, deleting and modifying operation of an original database is newly added to an original database, a second log table for recording an adding, deleting and modifying operation of a target database is newly added to a target database, when the original database starts full-volume data migration, recording an adding, deleting and modifying sql operation log of the original database at the same time, after the full-volume data migration is completed, stopping the original database service from being provided by the target database, at the moment, reading the sql operation log of the original database in time sequence, performing incremental migration, and completing the migration after the incremental migration is completed.
Specifically, referring to fig. 4, a first sql operation log table is newly added in an original database, then the original database starts full-volume data migration processing, meanwhile, an original database sql adding/deleting/modifying operation log is recorded through the first operation log table, migration processing refers to a process of cleaning, converting and integrating data, after the data are processed, the data are transmitted to a target database, the target database is loaded with full-volume data, after the target database is loaded with full-volume data, the original database is stopped, a second sql operation log table is newly added in the target database, services are provided by the target database, wherein the second operation log table records the target database sql adding/deleting/modifying operation log, and note that the log here is only the log of adding/deleting/modifying operations of an external system on the target database, and the operations on the target database based on the first operation log table of the original database are not included.
Acquiring a current operation statement from a first data operation log, determining current operation data corresponding to the current operation statement, inquiring whether a second operation log exists or not, and synchronizing the current operation data to a target database when the second operation log does not exist the target operation statement for operating the current operation data; when a target operation sentence for operating the current operation data exists in the second operation log, processing the current operation data in the original database based on the target operation sentence, deleting the target operation sentence in the second operation log, and adding the target operation sentence into the first operation log; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log, so as to complete data synchronization.
According to the data synchronization method, the current operation statement is obtained from the first data operation log, the current operation data corresponding to the current operation statement is determined, whether the second operation log is provided with the target operation statement for operating the current operation data is queried, if the second operation log is not provided with the target operation statement for operating the current operation data, the fact that the target database is not provided with the operation for the current operation data is indicated, at the moment, the current operation data is synchronized to the target database, data coverage of the target database is not caused, and data accuracy is guaranteed.
It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data synchronization device for realizing the above related data synchronization method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data synchronization device provided below may refer to the limitation of the data synchronization method hereinabove, and will not be repeated herein.
In an exemplary embodiment, as shown in fig. 5, there is provided a data synchronization apparatus, including: a first data operation log obtaining module 501, a second data operation log obtaining module 502, a current operation data determining module 503, a query module 504, an increment synchronization module 505, and a loop module 506, wherein:
a first data operation log obtaining module 501, configured to obtain a first data operation log of an original database in a data full synchronization process;
a second data operation log obtaining module 502, configured to obtain a second data operation log of the target database after being online;
a current operation data determining module 503, configured to obtain a current operation statement from the first data operation log, and determine current operation data corresponding to the current operation statement;
a query module 504, configured to query whether a target operation statement exists in the second operation log, where the target operation statement is operated on the current operation data;
an increment synchronization module 505, configured to synchronize the current operation data to the target database when there is no target operation statement for operating the current operation data in the second operation log;
and a circulation module 506, configured to obtain a next operation sentence from the first data operation log as a current operation sentence, and continue to execute the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, the synchronization module is further configured to, when there is a target operation statement in the second operation log that operates on the current operation data, process the current operation data in the original database based on the target operation statement, delete the target operation statement in the second operation log, and add the target operation statement to the first operation log;
the loop module 506 is further configured to obtain a next operation sentence from the first data operation log as a current operation sentence, and continue to execute the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, the data synchronization device further includes: and the stopping module is used for stopping updating the second data operation log of the target database after the online.
In one embodiment, the data synchronization device further includes:
the update times acquisition module is used for acquiring update times corresponding to the current operation data, wherein the update times are used for representing the times of updating the current operation data based on the target operation statement;
the setting module is used for outputting current operation data when the update times are greater than or equal to a preset value; and receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
In one embodiment, the data synchronization device further includes:
the ordering module is used for ordering the operation sentences in the first data operation log according to the time sequence;
the current operation data determining module 503 is further configured to obtain a current operation statement from the first data operation log based on the time sequence.
In one embodiment, the data synchronization device further includes:
the full-volume data synchronization module is used for receiving a database full-volume synchronization instruction and synchronizing data in an original database into a target database based on the full-volume synchronization instruction; in the process of synchronizing data in the original database to the target database, operation sentences aiming at the data in the original database are obtained, and the operation sentences are stored into a first data operation log.
The modules in the data synchronization device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In an exemplary embodiment, a computer device, which may be a terminal, is provided, and an internal structure diagram thereof may be as shown in fig. 6. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data synchronization method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in fig. 6 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one exemplary embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of: acquiring a first data operation log of an original database in a data full synchronization process; acquiring a second data operation log of the target database after online; acquiring a current operation statement from a first data operation log, and determining current operation data corresponding to the current operation statement; inquiring whether a target operation statement for operating the current operation data exists in the second operation log or not; when the second operation log does not contain a target operation statement for operating the current operation data, synchronizing the current operation data to a target database; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, the processor when executing the computer program further performs the steps of: when a target operation sentence for operating the current operation data exists in the second operation log, processing the current operation data in the original database based on the target operation sentence, deleting the target operation sentence in the second operation log, and adding the target operation sentence into the first operation log; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, after the processor executes the computer program until no operation statements exist in the first data operation log, the method includes: and stopping updating the second data operation log of the target database after the target database is online.
In one embodiment, after the processor processes the current operation data in the original database based on the target operation statement implemented when executing the computer program, the method further comprises: acquiring the update times corresponding to the current operation data, wherein the update times are used for representing the update times of the current operation data based on the target operation statement; when the update times are greater than or equal to a preset value, outputting current operation data; and receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
In one embodiment, before the processor, when executing the computer program, is implemented to obtain the current operation statement from the first data operation log, the method further includes: arranging the operation sentences in the first data operation log according to a time sequence; the method for obtaining the current operation sentence from the first data operation log when the processor executes the computer program comprises the following steps: the current operation statement is obtained from the first data operation log based on the time sequence.
In one embodiment, the obtaining the first data operation log of the original database in the data volume synchronization process implemented when the processor executes the computer program includes: receiving a database full synchronization instruction, and synchronizing data in an original database into a target database based on the full synchronization instruction; in the process of synchronizing data in the original database to the target database, operation sentences aiming at the data in the original database are obtained, and the operation sentences are stored into a first data operation log.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a first data operation log of an original database in a data full synchronization process; acquiring a second data operation log of the target database after online; acquiring a current operation statement from a first data operation log, and determining current operation data corresponding to the current operation statement; inquiring whether a target operation statement for operating the current operation data exists in the second operation log or not; when the second operation log does not contain a target operation statement for operating the current operation data, synchronizing the current operation data to a target database; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, the computer program when executed by the processor further performs the steps of: when a target operation sentence for operating the current operation data exists in the second operation log, processing the current operation data in the original database based on the target operation sentence, deleting the target operation sentence in the second operation log, and adding the target operation sentence into the first operation log; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, after the computer program is implemented when executed by the processor until no operation statements exist in the first data operation log, the method comprises: and stopping updating the second data operation log of the target database after the target database is online.
In one embodiment, after the processing of the current operation data in the original database based on the target operation statement, which is implemented when the computer program is executed by the processor, the method further comprises: acquiring the update times corresponding to the current operation data, wherein the update times are used for representing the update times of the current operation data based on the target operation statement; when the update times are greater than or equal to a preset value, outputting current operation data; and receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
In one embodiment, before the obtaining the current operation statement from the first data operation log, the computer program when executed by the processor further includes: arranging the operation sentences in the first data operation log according to a time sequence; the method for obtaining the current operation sentence from the first data operation log when the computer program is executed by the processor comprises the following steps: the current operation statement is obtained from the first data operation log based on the time sequence.
In one embodiment, the acquisition of the first data oplog of the original database in the data throughput synchronization process, which is implemented when the computer program is executed by the processor, comprises: receiving a database full synchronization instruction, and synchronizing data in an original database into a target database based on the full synchronization instruction; in the process of synchronizing data in the original database to the target database, operation sentences aiming at the data in the original database are obtained, and the operation sentences are stored into a first data operation log.
In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of: acquiring a first data operation log of an original database in a data full synchronization process; acquiring a second data operation log of the target database after online; acquiring a current operation statement from a first data operation log, and determining current operation data corresponding to the current operation statement; inquiring whether a target operation statement for operating the current operation data exists in the second operation log or not; when the second operation log does not contain a target operation statement for operating the current operation data, synchronizing the current operation data to a target database; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, the computer program when executed by the processor further performs the steps of: when a target operation sentence for operating the current operation data exists in the second operation log, processing the current operation data in the original database based on the target operation sentence, deleting the target operation sentence in the second operation log, and adding the target operation sentence into the first operation log; and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
In one embodiment, after the computer program is implemented when executed by the processor until no operation statements exist in the first data operation log, the method comprises: and stopping updating the second data operation log of the target database after the target database is online.
In one embodiment, after the processing of the current operation data in the original database based on the target operation statement, which is implemented when the computer program is executed by the processor, the method further comprises: acquiring the update times corresponding to the current operation data, wherein the update times are used for representing the update times of the current operation data based on the target operation statement; when the update times are greater than or equal to a preset value, outputting current operation data; and receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
In one embodiment, before the obtaining the current operation statement from the first data operation log, the computer program when executed by the processor further includes: arranging the operation sentences in the first data operation log according to a time sequence; the method for obtaining the current operation sentence from the first data operation log when the computer program is executed by the processor comprises the following steps: the current operation statement is obtained from the first data operation log based on the time sequence.
In one embodiment, the acquisition of the first data oplog of the original database in the data throughput synchronization process, which is implemented when the computer program is executed by the processor, comprises: receiving a database full synchronization instruction, and synchronizing data in an original database into a target database based on the full synchronization instruction; in the process of synchronizing data in the original database to the target database, operation sentences aiming at the data in the original database are obtained, and the operation sentences are stored into a first data operation log.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method of data synchronization, the method comprising:
acquiring a first data operation log of an original database in a data full synchronization process;
acquiring a second data operation log of the target database after online;
acquiring a current operation statement from the first data operation log, and determining current operation data corresponding to the current operation statement;
Inquiring whether the second operation log exists or not, and operating the current operation data to obtain a target operation statement;
when the second operation log does not contain a target operation statement for operating the current operation data, synchronizing the current operation data to a target database;
and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
2. The method according to claim 1, wherein the method further comprises:
when a target operation statement for operating the current operation data exists in the second operation log, processing the current operation data in an original database based on the target operation statement, deleting the target operation statement in the second operation log, and adding the target operation statement into the first operation log;
and acquiring a next operation sentence from the first data operation log as a current operation sentence, and continuously executing the step of determining the current operation data corresponding to the current operation sentence until no operation sentence exists in the first data operation log.
3. The method according to claim 1 or 2, wherein after said until no operation statement is present in said first data operation log, comprising:
and stopping updating the second data operation log of the target database after the target database is online.
4. The method of claim 2, wherein after processing the current operation data in the original database based on the target operation statement, further comprising:
acquiring the update times corresponding to the current operation data, wherein the update times are used for representing the update times of the current operation data based on a target operation statement;
outputting the current operation data when the update times are greater than or equal to a preset value;
receiving a setting instruction aiming at the current operation data through the target database, and updating the current operation data based on the setting instruction.
5. The method of claim 1, wherein prior to obtaining the current operation statement from the first data operation log, further comprising:
arranging the operation sentences in the first data operation log according to a time sequence;
the obtaining the current operation statement from the first data operation log includes:
And acquiring a current operation statement from the first data operation log based on the time sequence.
6. The method of claim 1, wherein the obtaining the first data operation log of the original database in the data volume synchronization process comprises:
receiving a database full synchronization instruction, and synchronizing data in the original database into the target database based on the full synchronization instruction;
and in the process of synchronizing the data in the original database to the target database, acquiring an operation statement aiming at the data in the original database, and storing the operation statement into a first data operation log.
7. A data synchronization device, the device comprising:
the first data operation log acquisition module is used for acquiring a first data operation log of the original database in the data full synchronization process;
the second data operation log acquisition module is used for acquiring a second data operation log of the target database after being online;
the current operation data determining module is used for acquiring a current operation statement from the first data operation log and determining current operation data corresponding to the current operation statement;
The query module is used for querying whether the second operation log exists or not, and performing operation on the current operation data;
the increment synchronization module is used for synchronizing the current operation data to a target database when no target operation statement for operating the current operation data exists in the second operation log;
and the circulation module is used for acquiring a next operation statement from the first data operation log as a current operation statement, and continuously executing the step of determining the current operation data corresponding to the current operation statement until no operation statement exists in the first data operation log.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311339171.2A CN117389989A (en) | 2023-10-16 | 2023-10-16 | Data synchronization method, apparatus, computer device, storage medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311339171.2A CN117389989A (en) | 2023-10-16 | 2023-10-16 | Data synchronization method, apparatus, computer device, storage medium, and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117389989A true CN117389989A (en) | 2024-01-12 |
Family
ID=89471474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311339171.2A Pending CN117389989A (en) | 2023-10-16 | 2023-10-16 | Data synchronization method, apparatus, computer device, storage medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117389989A (en) |
-
2023
- 2023-10-16 CN CN202311339171.2A patent/CN117389989A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2017028514A1 (en) | Method and device for storing and reading data | |
| CN108932286B (en) | Data query method and device | |
| CN117389989A (en) | Data synchronization method, apparatus, computer device, storage medium, and program product | |
| US20120204159A1 (en) | Methods and System for Managing Assets in Programming Code Translation | |
| CN113849524A (en) | Data processing method and device | |
| CN114528045B (en) | Plug-in operation method, device, computer equipment and storage medium | |
| CN114647630A (en) | File synchronization method, information generation method, file synchronization device, information generation device, computer equipment and storage medium | |
| US20210141771A1 (en) | Method of managing files by mobile device and mobile device for implementing the same | |
| CN115952136A (en) | Equipment data storage method, device, computer equipment and medium | |
| CN117435668A (en) | Data synchronization method, device, computer equipment and storage medium | |
| CN117574871A (en) | Report generation method, apparatus, device, medium, and program product | |
| CN116910115A (en) | Group query method, device, computer equipment and storage medium | |
| CN116894044A (en) | Data management method, device, computer equipment and storage medium | |
| CN117435589A (en) | Data transfer method, device, computer equipment and storage medium | |
| CN116932642A (en) | Data synchronization method, device, computer equipment and storage medium | |
| CN116541137A (en) | Transaction processing method, apparatus, computer device, storage medium, and program product | |
| CN117056220A (en) | Batch job testing method and device, computer equipment and storage medium | |
| CN116226187A (en) | Data caching method, device, computer equipment and storage medium | |
| CN117908825A (en) | Chip data sequential processing method, device, computer equipment and storage medium | |
| CN117435198A (en) | Page generation method and device | |
| CN116467332A (en) | White balance adjustment method, white balance adjustment device, computer equipment and storage medium | |
| CN117951223A (en) | Data synchronization method, device, computer equipment and storage medium | |
| CN117632987A (en) | Data synchronization method, device, computer equipment and storage medium thereof | |
| CN114218898A (en) | Online collaborative document editing method and device, computer equipment and storage medium | |
| CN114510492A (en) | Service data processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |