CN107798039B - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN107798039B CN107798039B CN201710357020.8A CN201710357020A CN107798039B CN 107798039 B CN107798039 B CN 107798039B CN 201710357020 A CN201710357020 A CN 201710357020A CN 107798039 B CN107798039 B CN 107798039B
- Authority
- CN
- China
- Prior art keywords
- synchronization
- data
- precondition
- time
- duration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
Abstract
A data synchronization method comprises the following steps: acquiring a precondition which is set in a synchronization tool and needs to be circularly waited, and acquiring the circulating waiting time of the precondition; judging whether the cycle waiting time of the precondition is greater than or equal to the preset cycle execution time according to the pre-stored corresponding relation between the precondition and the cycle execution time; and when the loop waiting duration of the precondition is greater than or equal to the loop execution duration, sending an ending instruction to the synchronization tool. The method can effectively close the cyclic waiting operation of the synchronous tool in time, and is beneficial to improving the convenience of the operation of workers.
Description
Technical Field
The invention belongs to the field of data processing, and particularly relates to a data synchronization method and device.
Background
To ensure the security of data, or to achieve the data sharing requirements of databases in different locations, database synchronization technology is commonly used to transfer data in one database to other databases. For example, the data distributed in the databases of the terminals are synchronized to the database in the backup server, so that the data of the terminals can be backed up, the data can be summarized, and the summarized data can be conveniently queried or otherwise processed.
When data is processed synchronously, a sophisticated data warehouse tool, such as a button tool, is generally used to extract, convert, and load the data from the source end to the destination end by using various components of a graphical interface of the synchronization tool. When using existing data warehouse tools for data synchronization, a precondition for the tool component is typically set. For example, whether reception of certain preposed data is completed is detected in a circulating mode, if the preposed condition is not completed, the synchronization tool is always in a circulating waiting state, a switch needs to be turned off manually by a user, operation is troublesome, and timeliness is high.
Disclosure of Invention
The invention aims to provide a data synchronization method to solve the problems that in the prior art, when a synchronization tool of a graphical interface is used, if preset conditions are not completed, a mahjong tile is in a circular waiting state all the time, manual closing operation of a user is troublesome, and timeliness is poor.
In a first aspect, an embodiment of the present invention provides a data synchronization method, where the method includes:
acquiring a precondition which is set in a synchronization tool and needs to be circularly waited, and acquiring the circulating waiting time of the precondition;
judging whether the cycle waiting time of the precondition is greater than or equal to the preset cycle execution time according to the pre-stored corresponding relation between the precondition and the cycle execution time;
and when the loop waiting duration of the precondition is greater than or equal to the loop execution duration, sending an ending instruction to the synchronization tool.
With reference to the first aspect, in a first possible implementation manner of the first aspect, when the precondition satisfies a preset data synchronization requirement, the method further includes:
recording data changed in a source database in a time period from a last synchronization time point to a current time point;
and sending the recorded changed data in the source database to a target database through the synchronization tool.
With reference to the first aspect or the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, before the step of determining, according to a pre-stored correspondence between the precondition and the loop execution duration, whether the loop waiting duration of the precondition is greater than or equal to a preset loop execution duration, the method further includes:
acquiring the completion deadline of a synchronization task, and the data type and data volume required to be synchronized by the synchronization task;
calculating the estimated time length required for completing the synchronization task according to the synchronization speed of the data type counted in advance;
and determining the cycle waiting time according to the completion time limit of the synchronous task and the synchronous estimated time.
With reference to the first possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the step of recording changed data in the source database includes:
monitoring modification operation in the source database through triggers, wherein the triggers comprise one or more of an insertion trigger, a deletion trigger and a modification trigger;
and when the data in the source database is monitored to be modified, caching the changed data into a synchronous storage area.
With reference to the first possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the step of monitoring data that changes in the source database includes:
the step of recording the changed data in the source database comprises:
acquiring a log of a source database, and determining changed data of the source database according to modification information recorded in the log;
and caching the changed data in the source database to a synchronous storage area.
In a second aspect, an embodiment of the present invention provides a data synchronization apparatus, where the apparatus includes:
the information acquisition unit is used for acquiring a precondition which needs to be circularly waited and is set in the synchronization tool and acquiring the circulating waiting time of the precondition;
the judging unit is used for judging whether the cycle waiting time of the precondition is greater than or equal to the preset cycle execution time according to the pre-stored corresponding relation between the precondition and the cycle execution time;
and the instruction sending unit is used for sending an ending instruction to the synchronization tool when the loop waiting duration of the precondition is greater than or equal to the loop execution duration.
With reference to the second aspect, in a first possible implementation manner of the second aspect, when the precondition satisfies a preset data synchronization requirement, the apparatus further includes:
the data recording unit is used for recording the changed data in the source database in the time period from the last synchronous time point to the current time point;
and the data sending unit is used for sending the recorded data changed in the source database to the target database through the synchronization tool.
With reference to the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the apparatus further includes:
the system comprises a deadline acquisition unit, a synchronization task synchronization unit and a data synchronization unit, wherein the deadline acquisition unit is used for acquiring the completion deadline of a synchronization task and the data type and data amount required to be synchronized by the synchronization task;
the estimated time length obtaining unit is used for calculating the estimated time length required by completing the synchronization task according to the synchronization speed of the data type counted in advance;
and the cyclic waiting time length obtaining unit is used for determining the cyclic waiting time length according to the completion time limit of the synchronous task and the synchronous estimated time length.
In a third aspect, an embodiment of the present invention provides a data synchronization apparatus, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the data synchronization method described in any one of the implementation manners of the first aspect when executing the computer program.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the data synchronization method according to any implementation manner of the first aspect.
In the invention, the precondition which needs to be cyclically waited in the synchronization tool is obtained, the cyclic waiting duration corresponding to the precondition is obtained, the cyclic execution duration corresponding to the precondition is obtained according to the preset corresponding relation between the precondition and the cyclic execution duration, and when the waiting duration is longer than the preset cyclic execution duration, an ending instruction is sent to the synchronization tool, so that the cyclic waiting operation of the synchronization tool can be effectively closed in time, and the convenience of the operation of a worker is improved.
Drawings
Fig. 1 is a flowchart of an implementation of a data synchronization method according to a first embodiment of the present invention;
FIG. 2 is a flowchart of an implementation of determining a cycle execution duration according to a first embodiment of the present invention;
FIG. 3 is a flowchart of an implementation of a data synchronization method according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data synchronization apparatus according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention aims to provide a data synchronization method, which aims to solve the problem that when data is synchronized in the prior art, due to the use of a synchronization tool of a graphical interface component, the precondition of the graphical interface component is set. In the running process of the synchronous tool, data of certain preconditions can be delayed to arrive or be lost, so that the synchronous tool is always in cyclic waiting, the system gives an alarm, a user needs to manually close the switch, the operation is troublesome, and the timeliness is poor. The invention will be further described with reference to the accompanying drawings.
The first embodiment is as follows:
fig. 1 shows an implementation flow of a data synchronization method provided by a first embodiment of the present invention, which is detailed as follows:
in step S101, a precondition that requires loop waiting set in the synchronization tool is acquired, and a loop waiting time period of the precondition is acquired.
Specifically, the synchronization tool according to the embodiment of the present invention may be an Extract-Transform-Load (ETL) tool, and is used to describe a process of extracting (Extract), transforming (Transform), and loading (Load) data from a source end to a destination end. The synchronization tool can be a tool such as a button, a Data Stage, a resyitory explorer, a Bee load, and the like.
The synchronization tool is provided with a precondition which needs to circularly wait, and the precondition comprises data which is synchronized in a graphic component of the synchronization tool. After the data to be synchronized is set, the data can be synchronized according to a predetermined period, including operations of extracting, converting, loading and the like.
The acquisition of the precondition which needs to be cyclically waited can acquire the precondition corresponding to the data which needs to be synchronized when the synchronization tool is set. For example, when the insurance data month ends are synchronized, a precondition that the golden month ends are completed on the same day needs to be prepared, and if the golden month ends are not completed on the same day, the golden month ends are always in a waiting state, and subsequent synchronization operation cannot be performed.
When the precondition requiring the loop waiting is determined, the corresponding relationship between different preconditions and the loop execution time length can be set according to the specific requirement of the synchronous data. The corresponding relation table can be adjusted by the user according to the requirement, so that the user can conveniently adjust the time length of the cyclic waiting.
In this embodiment of the present invention, the step of obtaining the cyclic wait duration of the precondition may include:
acquiring the operation starting time of the graphic assembly corresponding to the precondition according to the synchronization setting requirement in the synchronization tool;
and starting to monitor the precondition at the starting time, and generating the cycle waiting time of the precondition according to the monitoring result.
After the operation starting time of the graphic assembly corresponding to the precondition is determined, the cyclic waiting time length can be obtained conveniently. The operation starting time of the graphic assembly can be determined by the starting time of the set logic flow of the synchronization tool, or the data process corresponding to the logic flow of the synchronization tool can be monitored, and the operation starting time of the graphic assembly is determined according to the monitoring result.
After the starting time is obtained, when the starting time is reached, the precondition is monitored, whether data corresponding to the precondition are input into the synchronization tool or not is detected, if not, the synchronization tool is in a cycle waiting state, and the cycle waiting time is timed through a calculator. And when the data corresponding to the precondition is monitored to be input into the synchronization tool, resetting the cycle waiting time length and recalculating the cycle waiting time length.
In step S102, according to a pre-stored correspondence between the precondition and the cycle execution duration, determining whether the cycle waiting duration of the precondition is greater than or equal to a preset cycle execution duration;
since the corresponding relationship between the precondition and the cycle execution duration is stored in advance, when the precondition set in the synchronization tool is detected, the corresponding cycle execution duration can be inquired in the corresponding relationship table stored in advance. Alternatively, the cycle execution time length may be automatically obtained according to historical data. For example, historical data of the cycle waiting duration corresponding to the precondition is queried, and data of a predetermined multiple of the average cycle waiting duration is selected, for example, 1.2 times of the average cycle waiting duration is selected as the cycle execution duration.
Of course, the cycle execution duration may also be determined according to the method illustrated in fig. 2, and specifically includes:
in step S201, the completion deadline of the synchronization task, and the data type and data amount required to be synchronized by the synchronization task are acquired.
The completion deadline of the synchronization task refers to an expected time for completing data synchronization. The time for completing the data synchronization may be set according to different data synchronization tasks, or for a periodic data synchronization task, for example, for a monthly data synchronization, the expected time for completing the data synchronization may be automatically obtained according to the previous synchronization requirement.
The data types can comprise continuous data and discrete data, and the time required for synchronization is different between the continuous data and the discrete data, so that the data required to be synchronized can be distinguished according to time, and the time required for synchronization can be calculated for different types of data respectively.
Of course, the division of continuous data and discrete data is only one of the division modes, and the synchronization time required for data synchronization can be distinguished according to other division modes.
In step S202, an estimated time period required for completing the synchronization task is calculated according to the synchronization speed of the data type counted in advance.
After the data types are distinguished, the synchronization speeds of different data can be obtained according to historical synchronization data, the synchronization time required by the different types of data can be calculated according to the sizes of the different types of data and the synchronization speeds, and further the synchronization time required by all the data required to be synchronized can be obtained.
In step S203, the cycle waiting time is determined according to the completion time limit of the synchronization task and the synchronization estimation time.
After the time required by all data synchronization is obtained, the time for obtaining the cycle wait can be calculated by combining the completion deadline of the synchronization task. For example, if the completion time limit of the synchronization task is 24, the time required for all data synchronization is 3 hours and 30 minutes, and the current time is 18 points, the cycle execution time period is 30 minutes. Of course, the loop execution time length calculated according to the method described in fig. 2 may be compared with the loop execution time length corresponding to the search precondition, and a shorter time may be selected as the loop execution time length.
In step S103, when the loop waiting duration of the precondition is greater than or equal to the loop execution duration, an end instruction is sent to the synchronization tool.
When the cycle waiting time of the precondition is greater than or equal to the cycle execution time, the current precondition cycle time is represented to exceed the preset cycle execution time, so that the current cycle waiting needs to be finished, a system alarm signal is avoided, a worker needs to manually eliminate the alarm signal, and the convenience of data synchronization can be improved.
In an optional embodiment of the present invention, the step of sending an end instruction to the synchronization tool may include:
and sending an ending instruction to the synchronous tool, and closing a switch of the cyclic waiting corresponding to the precondition, or sending an ending instruction to the synchronous tool to end the operation of the synchronous tool.
And sending an ending instruction to the synchronous tool, and closing a switch corresponding to the precondition when the current cycle waiting time length exceeds the preset characteristic execution time length so as to end the waiting of the precondition. Other synchronized data may continue to be executed and processed.
When the precondition is data of important basic synchronization, an end instruction can be sent to the synchronization tool, the operation of the synchronization tool is ended, and a prompt message for ending synchronization is sent to a user.
It is worth noting that the method of the invention is based on the monitoring judgment program made on the basis of the synchronization tool composed of the existing graphic assembly, and the data is analyzed and processed, so that the operation of the synchronization tool is more reliable, and the operation efficiency of synchronization is improved.
The invention obtains the precondition which needs to be circularly waited in the synchronization tool, and obtains the circular waiting duration corresponding to the precondition, the circular execution duration corresponding to the precondition is obtained through the corresponding relation between the preset precondition and the circular execution duration, and when the waiting duration is longer than the preset circular execution duration, an ending instruction is sent to the synchronization tool, thereby the circular waiting operation of the synchronization tool can be effectively closed in time, and the convenience of the operation of working personnel is improved.
Example two:
fig. 3 shows an implementation flow of the data synchronization method provided by the second embodiment of the present invention, which is detailed as follows:
in step S301, a precondition that requires loop waiting set in the synchronization tool is acquired, and a loop waiting time period of the precondition is acquired.
In step S302, it is determined whether the loop waiting duration of the precondition is greater than or equal to the preset loop execution duration according to the pre-stored correspondence between the precondition and the loop execution duration.
In step S303, when the loop waiting duration of the precondition is greater than or equal to the loop execution duration, an end instruction is sent to the synchronization tool.
Steps S301 to S303 are substantially the same as steps S101 to S103 in the first embodiment, and are not repeated herein.
In step S304, data that changes in the source database during the time period from the last synchronization time point to the current time point is recorded.
In step S305, the recorded data that has changed in the source database is sent to the target database through the synchronization tool.
Specifically, in the embodiment of the present invention, the following method may be specifically adopted to monitor the changed data in the source database:
1. and monitoring modification operation in the source database through a trigger, and caching changed data into a synchronous storage area when the data in the source database is monitored to be modified, wherein the trigger comprises one or more of an insertion trigger, a deletion trigger and a modification trigger.
Specifically, by setting a trigger in the source database, when there is data modification in the source database, including any one of insertion of data, deletion of data, or rewriting of data, a recording function in the source database is triggered, and the modified part of the data is recorded. Includes recording the modified data content, where the modified data content is located, and storing the recorded content in the synchronous storage area. When data synchronization is carried out, only the data in the synchronous storage area needs to be sent to the target database.
The data modification in the source database may also include multiple modifications to the data in the same location, and in order to improve the efficiency of synchronizing the data stored in the storage area, the content of the modified data, the storage location of the modified data, and the time of modification may be recorded, and for the data in the same location that has been modified multiple times, the data corresponding to the last time of modification is selected as the data in the storage location, so as to avoid multiple invalid synchronization operations on the modified data, and improve the synchronization efficiency.
2. Acquiring a log of a source database, determining changed data of the source database according to modification information recorded in the log, and caching the changed data of the source database to a synchronous storage area.
The difference from the method 1 is that the synchronization method includes acquiring log data in a source database, analyzing the log data, searching for a log that modifies the source database, searching for content at a position in the source database according to the position recorded in the log, and synchronizing the searched content with a target database.
Of course, the method is not limited to the above two methods, and may also include a method based on full-table comparison to find modified data. And the modified data is synchronized to the target database, so that the data synchronization efficiency can be improved. Especially for the synchronization of huge source databases of the databases, the synchronization times of repeated data can be greatly reduced, and the utilization efficiency of synchronous resources is improved.
Example three:
fig. 4 shows a schematic structural diagram of a data synchronization apparatus according to a third embodiment of the present invention, which is detailed as follows:
the data synchronization device of the embodiment of the invention comprises:
an information obtaining unit 401, configured to obtain a precondition that needs to be circularly waited and is set in a synchronization tool, and obtain a cyclic waiting duration of the precondition;
a determining unit 402, configured to determine whether a cycle waiting duration of the precondition is greater than or equal to a preset cycle execution duration according to a pre-stored correspondence between the precondition and the cycle execution duration;
an instruction sending unit 403, configured to send an end instruction to the synchronization tool when the loop waiting duration of the precondition is greater than or equal to the loop execution duration.
Preferably, the information acquiring unit includes:
the starting time acquiring subunit is used for acquiring the running starting time of the graphic component corresponding to the precondition according to the synchronization setting requirement in the synchronization tool;
and the monitoring subunit is used for starting to monitor the precondition at the starting time and generating the cyclic waiting duration of the precondition according to the monitoring result.
Preferably, the apparatus further comprises:
the data monitoring unit is used for monitoring the changed data in the source database;
and the sending unit is used for sending the changed data to a target database through the synchronization tool.
Preferably, the data monitoring unit is specifically configured to:
and monitoring modification operation in the source database through a trigger, and caching changed data into a synchronous storage area when the data in the source database is monitored to be modified, wherein the trigger comprises one or more of an insertion trigger, a deletion trigger and a modification trigger.
The data synchronization device according to the embodiment of the present invention corresponds to the data synchronization methods according to the first and second embodiments, and will not be repeated herein.
Fig. 5 is a schematic diagram illustrating a data synchronization apparatus according to an embodiment of the present invention. As shown in fig. 5, the data synchronization apparatus 5 of this embodiment includes: a processor 50, a memory 51 and a computer program 52, such as a program of a data synchronization device, stored in said memory 51 and being executable on said processor 50. The processor 50, when executing the computer program 52, implements the steps in the above-described embodiments of the data synchronization method, such as the steps 101 to 103 shown in fig. 1. Alternatively, the processor 50, when executing the computer program 52, implements the functions of each module/unit in the above-mentioned device embodiments, for example, the functions of the modules 401 to 403 shown in fig. 5.
Illustratively, the computer program 52 may be partitioned into one or more modules/units that are stored in the memory 51 and executed by the processor 50 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 52 in the data synchronization device 5. For example, the computer program 52 may be divided into an information acquisition unit, a judgment unit, and an instruction transmission unit, and each unit functions specifically as follows:
the information acquisition unit is used for acquiring a precondition which needs to be circularly waited and is set in the synchronization tool and acquiring the circulating waiting time of the precondition;
the judging unit is used for judging whether the cycle waiting time of the precondition is greater than or equal to the preset cycle execution time according to the pre-stored corresponding relation between the precondition and the cycle execution time;
and the instruction sending unit is used for sending an ending instruction to the synchronization tool when the loop waiting duration of the precondition is greater than or equal to the loop execution duration.
The data synchronization device 5 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The data synchronization device may include, but is not limited to, a processor 50, a memory 51. Those skilled in the art will appreciate that fig. 5 is merely an example of the data synchronization device 5 and does not constitute a limitation of the data synchronization device 5 and may include more or less components than those shown, or combine certain components, or different components, for example, the data synchronization device may also include an input output device, a network access device, a bus, etc.
The Processor 50 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 51 may be an internal storage unit of the data synchronization device 5, such as a hard disk or a memory of the data synchronization device 5. The memory 51 may also be an external storage device of the data synchronization device 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the data synchronization device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the data synchronization device 5. The memory 51 is used for storing the computer program and other programs and data required by the data synchronization device. The memory 51 may also be used to temporarily store data that has been output or is to be output.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (8)
1. A method for synchronizing data, the method comprising:
acquiring a precondition which is set in a synchronization tool and needs to circularly wait, and acquiring the circulating waiting time of the precondition, wherein the precondition comprises the completion of synchronization operation of data needing to be synchronized;
acquiring a completion deadline of a synchronization task, and a data type and a data amount required to be synchronized by the synchronization task, wherein the completion deadline is expected time of data synchronization;
calculating the estimated time length required for completing the synchronization task according to the synchronization speed of the data type counted in advance;
according to the cycle execution duration obtained by synchronously estimating the duration and the completion duration of the synchronous task, selecting a shorter time as the cycle execution duration;
judging whether the cycle waiting time of the precondition is greater than or equal to the preset cycle execution time according to the pre-stored corresponding relation between the precondition and the cycle execution time;
and when the loop waiting duration of the precondition is greater than or equal to the loop execution duration, sending an ending instruction to the synchronization tool.
2. The data synchronization method according to claim 1, wherein when the precondition satisfies a preset data synchronization requirement, the method further comprises:
recording data changed in a source database in a time period from a last synchronization time point to a current time point;
and sending the recorded changed data in the source database to a target database through the synchronization tool.
3. The data synchronization method according to claim 2, wherein the step of recording the changed data in the source database comprises:
monitoring modification operation in the source database through triggers, wherein the triggers comprise one or more of an insertion trigger, a deletion trigger and a modification trigger;
and when the data in the source database is monitored to be modified, caching the changed data into a synchronous storage area.
4. The data synchronization method according to claim 2, wherein the step of recording the changed data in the source database comprises:
acquiring a log of a source database, and determining changed data of the source database according to modification information recorded in the log;
and caching the changed data in the source database to a synchronous storage area.
5. A data synchronization apparatus, the apparatus comprising:
the system comprises an information acquisition unit, a synchronization tool and a synchronization unit, wherein the information acquisition unit is used for acquiring a precondition which needs to be circularly waited and is set in the synchronization tool and acquiring the circulating waiting time of the precondition, and the precondition comprises that the data which needs to be synchronized completes the synchronization operation;
the system comprises a deadline acquisition unit, a synchronization task synchronization unit and a data synchronization unit, wherein the deadline acquisition unit is used for acquiring the completion deadline of a synchronization task and the data type and data amount required to be synchronized by the synchronization task;
the estimated time length obtaining unit is used for calculating the estimated time length required by completing the synchronization task according to the synchronization speed of the data type counted in advance;
a cycle waiting duration obtaining unit, configured to compare a cycle execution duration obtained according to the synchronization estimated duration with a completion duration of the synchronization task, and select a shorter time as the cycle execution duration;
the judging unit is used for judging whether the cycle waiting time of the precondition is greater than or equal to the preset cycle execution time according to the pre-stored corresponding relation between the precondition and the cycle execution time;
and the instruction sending unit is used for sending an ending instruction to the synchronization tool when the loop waiting duration of the precondition is greater than or equal to the loop execution duration.
6. The apparatus of claim 5, wherein when the precondition satisfies a preset data synchronization requirement, the apparatus further comprises:
the data recording unit is used for recording the changed data in the source database in the time period from the last synchronous time point to the current time point;
and the data sending unit is used for sending the recorded data changed in the source database to the target database through the synchronization tool.
7. A data synchronization device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 4 when executing the computer program.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710357020.8A CN107798039B (en) | 2017-05-19 | 2017-05-19 | Data synchronization method and device |
| PCT/CN2018/083047 WO2018210080A1 (en) | 2017-05-19 | 2018-04-13 | Data synchronisation method, apparatus, device, and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710357020.8A CN107798039B (en) | 2017-05-19 | 2017-05-19 | Data synchronization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107798039A CN107798039A (en) | 2018-03-13 |
| CN107798039B true CN107798039B (en) | 2020-06-05 |
Family
ID=61531119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710357020.8A Active CN107798039B (en) | 2017-05-19 | 2017-05-19 | Data synchronization method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107798039B (en) |
| WO (1) | WO2018210080A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107798039B (en) * | 2017-05-19 | 2020-06-05 | 平安科技（深圳）有限公司 | Data synchronization method and device |
| CN108629002A (en) * | 2018-05-03 | 2018-10-09 | 山东汇贸电子口岸有限公司 | A kind of big data comparison method and device based on kettle |
| CN109656999B (en) * | 2018-10-11 | 2024-03-15 | 平安科技（深圳）有限公司 | Method, device, storage medium and apparatus for synchronizing large data volume data |
| CN110290166B (en) * | 2019-05-05 | 2023-04-18 | 平安科技（深圳）有限公司 | Cross-cluster data interaction method, system and device and readable storage medium |
| CN111309742A (en) * | 2020-01-20 | 2020-06-19 | 深圳壹账通智能科技有限公司 | Data synchronization method and device, computer equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6112502A (en) * | 1998-02-10 | 2000-09-05 | Diebold, Incorporated | Restocking method for medical item dispensing system |
| US6788997B1 (en) * | 1998-06-01 | 2004-09-07 | Medselect, Inc. | Medical cabinet with adjustable drawers |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9112808B2 (en) * | 2007-02-06 | 2015-08-18 | At&T Intellectual Property Ii, L.P. | Devices, systems, and methods for providing data |
| CN101119408B (en) * | 2007-08-16 | 2010-11-10 | 中兴通讯股份有限公司 | Interface, method and communication system for implementing interface data consistency |
| CN102096685B (en) * | 2009-12-11 | 2013-04-17 | 阿里巴巴集团控股有限公司 | Method and device for synchronizing distributive data into data warehouse |
| US9043283B2 (en) * | 2011-11-01 | 2015-05-26 | International Business Machines Corporation | Opportunistic database duplex operations |
| CN103186624B (en) * | 2011-12-31 | 2017-10-20 | 亿阳信通股份有限公司 | A kind of method of data synchronization and data synchronization unit |
| US8751451B2 (en) * | 2012-04-12 | 2014-06-10 | Nokia Corporation | Method and apparatus for facilitating switching between devices |
| CN103823797A (en) * | 2012-11-16 | 2014-05-28 | 镇江诺尼基智能技术有限公司 | FTP (file transfer protocol) based real-time industry database data synchronization system |
| CN107798039B (en) * | 2017-05-19 | 2020-06-05 | 平安科技（深圳）有限公司 | Data synchronization method and device |
-
2017
- 2017-05-19 CN CN201710357020.8A patent/CN107798039B/en active Active
-
2018
- 2018-04-13 WO PCT/CN2018/083047 patent/WO2018210080A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6112502A (en) * | 1998-02-10 | 2000-09-05 | Diebold, Incorporated | Restocking method for medical item dispensing system |
| US6788997B1 (en) * | 1998-06-01 | 2004-09-07 | Medselect, Inc. | Medical cabinet with adjustable drawers |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018210080A1 (en) | 2018-11-22 |
| CN107798039A (en) | 2018-03-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107798039B (en) | Data synchronization method and device | |
| US8290937B2 (en) | Estimating and monitoring query processing time | |
| WO2019019361A1 (en) | Method and apparatus for processing data of database, computer device, and storage medium | |
| CN101158954B (en) | Method for recognizing repeat data in computer storage | |
| US10467192B2 (en) | Method and apparatus for updating data table in keyvalue database | |
| WO2019047441A1 (en) | Communication optimization method and system | |
| CN107589990B (en) | Data communication method and system based on thread pool | |
| CN104679596A (en) | Message processing method and system for improving concurrence performance of server-side | |
| CN115150471B (en) | Data processing method, apparatus, device, storage medium, and program product | |
| WO2017092603A1 (en) | Data transmission method and device | |
| WO2023160608A1 (en) | Robot control method and apparatus, and storage medium and robot cluster | |
| CN107368563B (en) | Database data deleting method and device, electronic equipment and storage medium | |
| CN105610899A (en) | Text file parallel uploading method and device | |
| CN105786917B (en) | Method and device for concurrent warehousing of time series data | |
| US10581997B2 (en) | Techniques for storing or accessing a key-value item | |
| CN111159131A (en) | Performance optimization method, device, equipment and computer readable storage medium | |
| CN111078418B (en) | Operation synchronization method, device, electronic equipment and computer readable storage medium | |
| WO2018000622A1 (en) | Wms system and erp system synchronization method and system | |
| CN109885384B (en) | Task parallelism optimization method and device, computer equipment and storage medium | |
| CN111159009A (en) | Pressure testing method and device for log service system | |
| CN110572234A (en) | Method for realizing clock synchronization based on serial port, intelligent terminal and storage medium | |
| CN112948124B (en) | Acceleration task processing method, device, equipment and readable storage medium | |
| CN110865877B (en) | Task request response method and device | |
| EP3396553B1 (en) | Method and device for processing data after restart of node | |
| CN110245148B (en) | Data storage method, device, system and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |