CN114064803A - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN114064803A CN114064803A CN202111330901.3A CN202111330901A CN114064803A CN 114064803 A CN114064803 A CN 114064803A CN 202111330901 A CN202111330901 A CN 202111330901A CN 114064803 A CN114064803 A CN 114064803A
- Authority
- CN
- China
- Prior art keywords
- data
- data synchronization
- synchronization task
- task
- degradation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention discloses a data synchronization method and device, and relates to the technical field of computers. One embodiment of the method comprises: executing a data synchronization task; if the execution of the data synchronization task fails, matching a degradation rule according to the reason of the execution failure of the data synchronization task, and judging whether to perform degradation processing on the data synchronization task according to the degradation rule; if so, performing degradation processing on the data synchronization task; and if not, recording the reason of the execution failure of the data synchronization task. The implementation method can solve the technical problems of poor system expansibility or manpower and material resource consumption.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data synchronization method and apparatus.
Background
When the task of synchronizing data is executed, the data cannot be synchronized successfully due to data content or format problems, but whether the processing can be discarded or degraded is uncertain. Most of the currently adopted methods are to judge whether the code can be discarded or degraded according to error return information or manual work.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
1) when the data volume is too large, a large amount of manual intervention is needed, so that manpower and material resources are consumed;
2) when new wrong return information exists each time, the code needs to be changed and the system is online, so that the system expansibility is poor;
3) there may be a possibility of long invalid retries or loss of sync data due to an inability to automatically identify whether it can be discarded or degraded.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data synchronization method and apparatus, so as to solve the technical problem that the system has poor scalability or consumes manpower and material resources.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data synchronization method including:
executing a data synchronization task;
if the execution of the data synchronization task fails, matching a degradation rule according to the reason of the execution failure of the data synchronization task, and judging whether to perform degradation processing on the data synchronization task according to the degradation rule;
if so, performing degradation processing on the data synchronization task;
if not, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
Optionally, performing the data synchronization task includes:
if the data synchronization task is a data adding task, adding data to be synchronized corresponding to the data synchronization task into a database, or splitting at least one piece of data from original data in the database according to the data to be synchronized corresponding to the data synchronization task and adding the at least one piece of data into the database;
and if the data synchronization task is a data modification task, modifying the original data in the database according to the data to be synchronized corresponding to the data synchronization task.
Optionally, matching a degradation rule according to the reason for the execution failure of the data synchronization task, and determining whether to perform degradation processing on the data synchronization task according to the degradation rule, includes:
matching a degradation rule according to the execution failure reason of the data synchronization task; wherein the downgrading rules include an execution failure reason and a failure number threshold;
adding one to the execution failure times of the data synchronization task;
and judging whether the execution failure times of the data synchronization task is greater than or equal to the failure time threshold value.
Optionally, the downgrading rule further includes a downgrading processing manner, where the downgrading processing manner includes downgrading synchronization or downgrading discarding;
performing degradation processing on the data synchronization task, including:
and performing degradation synchronization or degradation discarding on the data to be synchronized corresponding to the data synchronization task according to a degradation processing mode in the degradation rule.
Optionally, performing destage synchronization on the data to be synchronized corresponding to the data synchronization task, including:
if the data synchronization task is a data adding task and the original data to be split is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task into the database;
and if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database.
Optionally, the data to be synchronized corresponding to the data synchronization task includes an order identifier and at least one order detail;
if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database, including:
if the data synchronization task is a data modification task and the order identification to be modified is not matched in the database, retrying the task to be synchronized until the order identification to be modified is matched in the database;
and if the data synchronization task is a data modification task and the order details to be modified are not matched in the database, directly adding the order details to be modified into the database.
Optionally, after recording the reason for the execution failure of the data synchronization task and retrying the data synchronization task, the method further includes:
and analyzing the reason of the execution failure of each data synchronization task, and configuring a degradation rule according to the analysis result.
In addition, according to another aspect of the embodiments of the present invention, there is provided a data synchronization apparatus including:
a synchronization module for performing a data synchronization task;
the degradation module is used for matching a degradation rule according to the execution failure reason of the data synchronization task and judging whether to perform degradation processing on the data synchronization task according to the degradation rule if the data synchronization task fails to be executed; if so, performing degradation processing on the data synchronization task; if not, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
Optionally, the synchronization module is further configured to:
if the data synchronization task is a data adding task, adding data to be synchronized corresponding to the data synchronization task into a database, or splitting at least one piece of data from original data in the database according to the data to be synchronized corresponding to the data synchronization task and adding the at least one piece of data into the database;
and if the data synchronization task is a data modification task, modifying the original data in the database according to the data to be synchronized corresponding to the data synchronization task.
Optionally, the degradation module is further configured to:
matching a degradation rule according to the execution failure reason of the data synchronization task; wherein the downgrading rules include an execution failure reason and a failure number threshold;
adding one to the execution failure times of the data synchronization task;
and judging whether the execution failure times of the data synchronization task is greater than or equal to the failure time threshold value.
Optionally, the downgrading rule further includes a downgrading processing manner, where the downgrading processing manner includes downgrading synchronization or downgrading discarding;
the degradation module is further to:
and performing degradation synchronization or degradation discarding on the data to be synchronized corresponding to the data synchronization task according to a degradation processing mode in the degradation rule.
Optionally, the degradation module is further configured to:
if the data synchronization task is a data adding task and the original data to be split is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task into the database;
and if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database.
Optionally, the data to be synchronized corresponding to the data synchronization task includes an order identifier and at least one order detail;
the degradation module is further to:
if the data synchronization task is a data modification task and the order identification to be modified is not matched in the database, retrying the task to be synchronized until the order identification to be modified is matched in the database;
and if the data synchronization task is a data modification task and the order details to be modified are not matched in the database, directly adding the order details to be modified into the database.
Optionally, the degradation module is further configured to:
recording the reason for the execution failure of the data synchronization task, retrying the data synchronization task, analyzing the reason for the execution failure of each data synchronization task, and configuring a degradation rule according to the analysis result.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the embodiments described above.
According to another aspect of the embodiments of the present invention, there is also provided a computer readable medium, on which a computer program is stored, which when executed by a processor implements the method of any of the above embodiments.
One embodiment of the above invention has the following advantages or benefits: the technical means that the degradation rule is matched according to the execution failure reason of the data synchronization task and whether the data synchronization task is subjected to degradation processing is judged according to the degradation rule is adopted, if yes, the data synchronization task is subjected to degradation processing, and if not, the execution failure reason is recorded and retried, so that the technical problems that the system expansibility is poor or manpower and material resources are consumed in the prior art are solved. The embodiment of the invention can not only reduce the manual intervention to the maximum extent, but also only carry out the rule configuration according to the execution failure reason of the statistical data synchronization task; rules can be increased or decreased on line without online, and the system has good expansibility; and moreover, the execution failure reasons can be counted in real time, the rule can be rapidly configured, and the data is prevented from being retried or lost for a long time in an invalid manner.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a data synchronization method according to an embodiment of the present invention;
FIG. 2 is a schematic view of a main flow of a data synchronization method according to a referential embodiment of the present invention;
FIG. 3 is a schematic view of a main flow of a data synchronization method according to another referential embodiment of the present invention;
FIG. 4 is a schematic view of a main flow of a data synchronization method according to still another referential embodiment of the present invention;
FIG. 5 is a schematic diagram of the main modules of a data synchronization apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data synchronization method according to an embodiment of the present invention. As an embodiment of the present invention, as shown in fig. 1, the data synchronization method may include:
step 101, a data synchronization task is performed.
Firstly, a data synchronization task issued by an upstream system is received, and then the data synchronization task is executed. The data to be synchronized corresponding to the data synchronization task may be order data or other business data, and the order data includes an order identifier and at least one order detail.
Optionally, step 101 may comprise: if the data synchronization task is a data adding task, adding data to be synchronized corresponding to the data synchronization task into a database, or splitting at least one piece of data from original data in the database according to the data to be synchronized corresponding to the data synchronization task and adding the at least one piece of data into the database; and if the data synchronization task is a data modification task, modifying the original data in the database according to the data to be synchronized corresponding to the data synchronization task. After receiving a data synchronization task, firstly judging whether the data synchronization task is a data adding task or a data modifying task, if the data synchronization task is the data adding task, continuously judging whether to directly add data to be synchronized into a database or split at least one piece of data from original data and then add the data into the database (for example, splitting order data and splitting at least one sub order from a parent order), and then executing corresponding processing logic; and if the task is a data modification task, modifying the original data in the database according to the data to be synchronized.
It should be noted that, in the embodiment of the present invention, for data that needs to be deleted, the data is not directly deleted, but the data is set to be invalid by the data modification task, and the invalid data is deleted by the subsequent business logic.
If the data synchronization task is to directly add new data, directly adding the data to be synchronized into the database; if the new addition is successful, ending; if the addition fails, step 102 is performed.
If the data synchronization task is a data splitting task, whether an identifier (such as an order identifier or a user identifier) of data to be synchronized exists in a database needs to be inquired first; if the data exists, splitting the data from the original data and adding the split data into the database; if the new addition is successful, ending; if the new addition fails or the original data does not exist, step 102 is performed.
If the data synchronization task is a data modification task, whether an identifier (such as an order identifier or a user identifier) of data to be synchronized exists in a database needs to be inquired first; if yes, modifying the data in the database, and if the modification is successful, ending; if the modification fails or the data does not exist, step 102 is performed.
102, if the execution of the data synchronization task fails, matching a degradation rule according to the reason of the execution failure of the data synchronization task, and judging whether to perform degradation processing on the data synchronization task according to the degradation rule; if yes, go to step 103; if not, go to step 104.
The administrator may pre-configure the downgrading rule, for example, by recording, counting, and analyzing the historical execution failure reason (for example, a call interface failure, a data check failure, a data write failure, or an absence of a parent order), abstract and configure the downgrading rule, so that when the execution failure reason occurs next time, if the downgrading rule is matched, whether to perform downgrading processing is determined according to the downgrading rule. Optionally, the downgrading rule may include a reason for execution failure, and if the data synchronization task fails to be executed, the downgrading rule is matched from the configuration table according to the reason for execution failure of the data synchronization task, and then whether to downgrade the data synchronization task is determined according to the downgrading rule. Alternatively, demotion rules may be matched by regular expressions.
Optionally, matching a degradation rule according to the reason for the execution failure of the data synchronization task, and determining whether to perform degradation processing on the data synchronization task according to the degradation rule, includes: matching a degradation rule according to the execution failure reason of the data synchronization task; adding one to the execution failure times of the data synchronization task; and judging whether the execution failure times of the data synchronization task is greater than or equal to a failure time threshold value. Wherein the downgrading rules may include an execution failure reason and a number of failures threshold. If the execution of the data synchronization task fails, the reason for the execution failure of the data synchronization task matches a degradation rule, then the number of times of the execution failure of the data synchronization task is increased by one through a counter, and then whether the number of times of the execution failure of the data synchronization task is larger than or equal to a failure number threshold value in the degradation rule is judged, if so, the condition of the degradation processing is met, step 103 is executed, if not, the condition of the degradation processing is not met, and step 104 is executed. The limitation of increasing the failure time threshold value in the degradation rule can avoid the accidental reason of executing failure to trigger the degradation process.
It should be noted that, when the execution failure times of the data synchronization task are accumulated, the execution failure reasons are accumulated the same time, for example, the same data synchronization task fails five times, but the execution failure reasons of the first two times and the execution failure reasons of the last three times are different, and the execution failure reasons need to be accumulated respectively for different execution failure reasons.
Optionally, the downgrading rule may further include a time period threshold, and when the number of times of execution failures of the data synchronization task is accumulated, the number of times exceeding the time period threshold is removed, and only the number of times of execution failures in the time period closest to the current time is accumulated, so that the result of the number accumulation of the past reasons of execution failures is prevented from affecting the execution of the current data synchronization task.
Optionally, the downgrading rule further includes a downgrading processing manner, where the downgrading processing manner includes downgrading synchronization or downgrading discard, and the downgrading processing manner determines a downgrading processing manner of the data synchronization task.
And 103, performing degradation processing on the data synchronization task.
And if the data synchronization task can be degraded, calling a degradation interface to perform degradation processing on the data synchronization task.
Optionally, performing degradation processing on the data synchronization task includes: and performing degradation synchronization or degradation discarding on the data to be synchronized corresponding to the data synchronization task according to a degradation processing mode in the degradation rule. In the embodiment of the present invention, the degradation processing mode may include degradation synchronization or degradation discarding, and after the degradation rule is matched, according to the degradation processing mode in the degradation rule, the data to be synchronized corresponding to the data synchronization task is subjected to degradation synchronization or degradation discarding.
Optionally, performing destage synchronization on the data to be synchronized corresponding to the data synchronization task, including: if the data synchronization task is a data adding task and the original data to be split is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task into the database; and if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database. If the original data to be split or the original data to be modified are not matched in the database, the data to be synchronized corresponding to the data synchronization task is directly added to the database, so that data loss can be avoided, long-time invalid retry is avoided, and time consumed by data synchronization is saved.
Optionally, if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database, including: if the data synchronization task is a data modification task and the order identification to be modified is not matched in the database, retrying the task to be synchronized until the order identification to be modified is matched in the database; and if the data synchronization task is a data modification task and the order details to be modified are not matched in the database, directly adding the order details to be modified into the database. Optionally, the data to be synchronized corresponding to the data synchronization task may be order data, and the order data may include an order identifier and at least one order detail. Taking the order data as an example, if the order identifier cannot be queried in the database, the retry is continuously performed until the data modification is completed, possibly because the data adding task arrives later than the data modifying task. If the order details can not be inquired in the database, the order details to be modified are directly added into the database (if the order details to be modified are more than the order details to be modified, the added order details are added, other order details are modified), and retry is not needed. Therefore, data loss can be avoided, and efficient completion of data synchronization tasks can be guaranteed.
In the embodiment of the invention, if the degradation processing mode is degradation discarding, the data synchronized this time is discarded, and the execution is finished; and if the downgrading processing mode is downgrading synchronization, downgrading the synchronous data, if the downgrading processing mode is downgrading synchronization, ending the execution, and if the downgrading processing mode is successful, performing retry processing until the downgrading processing is successful or the retry number is greater than or equal to the retry number threshold.
Optionally, whether to retry the data to be synchronized is determined according to the service type of the data to be synchronized, and if the data to be synchronized fails in the downgrading process, if the data to be synchronized is a service document, the data to be synchronized is subjected to downgrading retry, and if the data to be synchronized is a non-service document, the data to be synchronized is directly discarded.
And 104, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
If the degradation is not possible, recording the reason of the failure, and then calling the interface to perform retry processing until the retry number is greater than or equal to the retry number threshold.
Optionally, after step 104, the method further includes: and analyzing the reason of the execution failure of each data synchronization task, and configuring a degradation rule according to the analysis result. The administrator analyzes according to the statistical execution failure reasons, configures the degradation rules according to the execution failure reasons, and writes the degradation rules into a configuration table, and the configuration table can be stored in a database.
According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problems of poor system expansibility or manpower and material resource consumption in the prior art by matching the degradation rule according to the execution failure reason of the data synchronization task and determining whether to perform degradation processing on the data synchronization task according to the degradation rule, if yes, performing degradation processing, and if not, recording the execution failure reason and performing retry. The embodiment of the invention can not only reduce the manual intervention to the maximum extent, but also only carry out the rule configuration according to the execution failure reason of the statistical data synchronization task; rules can be increased or decreased on line without online, and the system has good expansibility; and moreover, the execution failure reasons can be counted in real time, the rule can be rapidly configured, and the data is prevented from being retried or lost for a long time in an invalid manner.
Fig. 2 is a schematic diagram of a main flow of a data synchronization method according to a referential embodiment of the present invention. As still another embodiment of the present invention, as shown in fig. 2, the data synchronization method may include:
step 201, judging whether a data synchronization task is a data newly-added task or a data modification task; if the data is a new task, executing step 202; if so, go to step 206.
Step 202, judging whether the data synchronization task is to directly add new data or split the data; if the data is directly added, go to step 203; if the data is split, step 204 is performed.
Step 203, adding the data to be synchronized corresponding to the data synchronization task to the database.
Step 204, judging whether the original data to be split is in the database; if yes, go to step 205; if not, go to step 208.
Step 205, splitting at least one piece of data from the original data in the database and adding the at least one piece of data to the database.
Step 206, judging whether the original data to be modified is in the database; if yes, go to step 207; if not, go to step 208.
And step 207, modifying the original data in the database according to the data to be synchronized corresponding to the data synchronization task.
Step 208, judging whether the data synchronization task is successfully executed; if yes, the process is terminated, otherwise, step 209 is executed.
Step 209, matching a degradation rule according to the reason for the execution failure of the data synchronization task, and judging whether to perform degradation processing on the data synchronization task according to the degradation rule; if yes, go to step 210; if not, go to step 211.
And 210, performing degradation processing on the data synchronization task.
And step 211, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
In addition, in one embodiment of the present invention, the detailed implementation of the data synchronization method is described in detail above, and therefore the repeated content is not described herein.
Fig. 3 is a schematic diagram of a main flow of a data synchronization method according to another referential embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 3, the data synchronization method may include:
Step 302, judging whether the data synchronization task is executed successfully; if yes, the process is ended, otherwise, step 303 is executed.
And 303, matching a downgrading rule according to the reason for the execution failure of the data synchronization task.
The degradation rule comprises an execution failure reason, a failure frequency threshold value and a degradation processing mode, wherein the degradation processing mode comprises degradation synchronization or degradation discarding.
Step 304, adding one to the execution failure times of the data synchronization task.
Step 305, judging whether the execution failure times of the data synchronization task is greater than or equal to the failure time threshold value; if yes, go to step 306; if not, go to step 307.
And step 306, performing degradation synchronization or degradation discarding on the data to be synchronized corresponding to the data synchronization task according to a degradation processing mode in the degradation rule.
Step 307, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
In addition, in another embodiment of the present invention, the detailed implementation of the data synchronization method is described in detail above, and therefore the repeated content is not described herein.
Fig. 4 is a schematic diagram of a main flow of a data synchronization method according to still another referential embodiment of the present invention. As still another embodiment of the present invention, as shown in fig. 4, the data synchronization method may include:
Step 402, judging whether the data synchronization task is executed successfully; if yes, the process is terminated, otherwise, step 403 is executed.
And step 403, matching a downgrading rule according to the reason for the execution failure of the data synchronization task.
Step 404, performing degradation discarding on the data synchronization task.
Step 405, performing downgrade synchronization on the data synchronization task.
Step 406, whether the destaging of the synchronous data is successful; if yes, ending; if not, go to step 407.
Step 407, downgrade retries.
In addition, in another embodiment of the present invention, the data synchronization method is described in detail above, and therefore the repeated content will not be described.
Fig. 5 is a schematic diagram of main blocks of a data synchronization apparatus according to an embodiment of the present invention. As shown in fig. 5, the data synchronization apparatus 500 includes a synchronization module 501 and a demotion module 505; the synchronization module 501 is configured to perform a data synchronization task; the degradation module 502 is configured to, if the execution of the data synchronization task fails, match a degradation rule according to a reason for the execution failure of the data synchronization task, and determine whether to perform degradation processing on the data synchronization task according to the degradation rule; if so, performing degradation processing on the data synchronization task; and if not, recording the reason of the execution failure of the data synchronization task.
Optionally, the synchronization module 501 is further configured to:
if the data synchronization task is a data adding task, adding data to be synchronized corresponding to the data synchronization task into a database, or splitting at least one piece of data from original data in the database according to the data to be synchronized corresponding to the data synchronization task and adding the at least one piece of data into the database;
and if the data synchronization task is a data modification task, modifying the original data in the database according to the data to be synchronized corresponding to the data synchronization task.
Optionally, the downgrading module 502 is further configured to:
matching a degradation rule according to the execution failure reason of the data synchronization task; wherein the downgrading rules include an execution failure reason and a number threshold;
adding one to the execution failure times of the data synchronization task;
and judging whether the execution failure times of the data synchronization task is greater than or equal to the time threshold value.
Optionally, the downgrading rule further includes a downgrading processing manner, where the downgrading processing manner includes downgrading synchronization or downgrading discarding;
the degradation module 502 is further configured to:
and performing degradation synchronization or degradation discarding on the data to be synchronized corresponding to the data synchronization task according to a degradation processing mode in the degradation rule.
Optionally, the downgrading module 502 is further configured to:
if the data synchronization task is a data adding task and the original data to be split is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task into the database;
and if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database.
Optionally, the data to be synchronized corresponding to the data synchronization task includes an order identifier and at least one order detail;
the degradation module 502 is further configured to:
if the data synchronization task is a data modification task and the order identification to be modified is not matched in the database, retrying the task to be synchronized until the order identification to be modified is matched in the database;
and if the data synchronization task is a data modification task and the order details to be modified are not matched in the database, directly adding the order details to be modified into the database.
Optionally, the downgrading module 502 is further configured to:
and after recording the execution failure reasons of the data synchronization tasks, analyzing the execution failure reasons of each data synchronization task, and configuring a degradation rule according to the analysis result.
According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problems of poor system expansibility or manpower and material resource consumption in the prior art by matching the degradation rule according to the execution failure reason of the data synchronization task and determining whether to perform degradation processing on the data synchronization task according to the degradation rule, if yes, performing degradation processing, and if not, recording the execution failure reason and performing retry. The embodiment of the invention can not only reduce the manual intervention to the maximum extent, but also only carry out the rule configuration according to the execution failure reason of the statistical data synchronization task; rules can be increased or decreased on line without online, and the system has good expansibility; and moreover, the execution failure reasons can be counted in real time, the rule can be rapidly configured, and the data is prevented from being retried or lost for a long time in an invalid manner.
It should be noted that, in the implementation of the data synchronization apparatus of the present invention, the data synchronization method has been described in detail above, and therefore, the repeated content herein is not described again.
Fig. 6 illustrates an exemplary system architecture 600 to which the data synchronization method or data synchronization apparatus of embodiments of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. The terminal devices 601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 601, 602, 603. The background management server can analyze and process the received data such as the article information query request and feed back the processing result to the terminal equipment.
It should be noted that the data synchronization method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data synchronization apparatus is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a synchronization module and a demotion module, where the names of the modules do not in some way constitute a limitation on the modules themselves.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, implement the method of: executing a data synchronization task; if the execution of the data synchronization task fails, matching a degradation rule according to the reason of the execution failure of the data synchronization task, and judging whether to perform degradation processing on the data synchronization task according to the degradation rule; if so, performing degradation processing on the data synchronization task; and if not, recording the reason of the execution failure of the data synchronization task.
According to the technical scheme of the embodiment of the invention, because the technical means of matching the degradation rule according to the reason of the execution failure of the data synchronization task and judging whether to perform degradation processing on the data synchronization task according to the degradation rule is adopted, if so, the degradation processing is performed, and if not, the reason of the execution failure is recorded and retried, the technical problems of poor system expansibility or consumption of manpower and material resources in the prior art are solved. The embodiment of the invention can not only reduce the manual intervention to the maximum extent, but also only carry out the rule configuration according to the execution failure reason of the statistical data synchronization task; rules can be increased or decreased on line without online, and the system has good expansibility; and moreover, the execution failure reasons can be counted in real time, the rule can be rapidly configured, and the data is prevented from being retried or lost for a long time in an invalid manner.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data synchronization, comprising:
executing a data synchronization task;
if the execution of the data synchronization task fails, matching a degradation rule according to the reason of the execution failure of the data synchronization task, and judging whether to perform degradation processing on the data synchronization task according to the degradation rule;
if so, performing degradation processing on the data synchronization task;
if not, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
2. The method of claim 1, wherein performing a data synchronization task comprises:
if the data synchronization task is a data adding task, adding data to be synchronized corresponding to the data synchronization task into a database, or splitting at least one piece of data from original data in the database according to the data to be synchronized corresponding to the data synchronization task and adding the at least one piece of data into the database;
and if the data synchronization task is a data modification task, modifying the original data in the database according to the data to be synchronized corresponding to the data synchronization task.
3. The method of claim 1, wherein matching a downgrading rule according to a reason for execution failure of the data synchronization task, and determining whether to downgrade the data synchronization task according to the downgrading rule comprises:
matching a degradation rule according to the execution failure reason of the data synchronization task; wherein the downgrading rules include an execution failure reason and a failure number threshold;
adding one to the execution failure times of the data synchronization task;
and judging whether the execution failure times of the data synchronization task is greater than or equal to the failure time threshold value.
4. The method of claim 1, wherein the destage rules further comprise destage handling means comprising destage synchronization or destage discard;
performing degradation processing on the data synchronization task, including:
and performing degradation synchronization or degradation discarding on the data to be synchronized corresponding to the data synchronization task according to a degradation processing mode in the degradation rule.
5. The method according to claim 4, wherein performing destage synchronization on the data to be synchronized corresponding to the data synchronization task comprises:
if the data synchronization task is a data adding task and the original data to be split is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task into the database;
and if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database.
6. The method according to claim 4, wherein the data to be synchronized corresponding to the data synchronization task comprises an order identification and at least one order detail;
if the data synchronization task is a data modification task and the original data to be modified is not matched in the database, directly adding the data to be synchronized corresponding to the data synchronization task to the database, including:
if the data synchronization task is a data modification task and the order identification to be modified is not matched in the database, retrying the task to be synchronized until the order identification to be modified is matched in the database;
and if the data synchronization task is a data modification task and the order details to be modified are not matched in the database, directly adding the order details to be modified into the database.
7. The method of claim 1, wherein after recording the reason for the failed execution of the data synchronization task and retrying the data synchronization task, further comprising:
and analyzing the reason of the execution failure of each data synchronization task, and configuring a degradation rule according to the analysis result.
8. A data synchronization apparatus, comprising:
a synchronization module for performing a data synchronization task;
the degradation module is used for matching a degradation rule according to the execution failure reason of the data synchronization task and judging whether to perform degradation processing on the data synchronization task according to the degradation rule if the data synchronization task fails to be executed; if so, performing degradation processing on the data synchronization task; if not, recording the reason of the execution failure of the data synchronization task and retrying the data synchronization task.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
the one or more programs, when executed by the one or more processors, implement the method of any of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111330901.3A CN114064803A (en) | 2021-11-11 | 2021-11-11 | Data synchronization method and device |
| PCT/CN2022/129179 WO2023083066A1 (en) | 2021-11-11 | 2022-11-02 | Data synchronization method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111330901.3A CN114064803A (en) | 2021-11-11 | 2021-11-11 | Data synchronization method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114064803A true CN114064803A (en) | 2022-02-18 |
Family
ID=80274973
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111330901.3A Pending CN114064803A (en) | 2021-11-11 | 2021-11-11 | Data synchronization method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114064803A (en) |
| WO (1) | WO2023083066A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023083066A1 (en) * | 2021-11-11 | 2023-05-19 | 北京沃东天骏信息技术有限公司 | Data synchronization method and apparatus |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10372624B2 (en) * | 2017-08-18 | 2019-08-06 | International Business Machines Corporation | Destaging pinned retryable data in cache |
| CN111400104B (en) * | 2020-03-24 | 2021-09-28 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN113297357B (en) * | 2021-07-27 | 2021-11-16 | 北京健康之家科技有限公司 | Asynchronous processing method and device for business process data |
| CN114064803A (en) * | 2021-11-11 | 2022-02-18 | 北京沃东天骏信息技术有限公司 | Data synchronization method and device |
| CN114528140A (en) * | 2022-02-15 | 2022-05-24 | 京东科技控股股份有限公司 | Method and device for service degradation |
-
2021
- 2021-11-11 CN CN202111330901.3A patent/CN114064803A/en active Pending
-
2022
- 2022-11-02 WO PCT/CN2022/129179 patent/WO2023083066A1/en unknown
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023083066A1 (en) * | 2021-11-11 | 2023-05-19 | 北京沃东天骏信息技术有限公司 | Data synchronization method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023083066A1 (en) | 2023-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111190888A (en) | Method and device for managing graph database cluster | |
| CN110019263B (en) | Information storage method and device | |
| CN114091704A (en) | Alarm suppression method and device | |
| CN111427899A (en) | Method, device, equipment and computer readable medium for storing file | |
| CN110705271A (en) | System and method for providing natural language processing service | |
| CN111831536A (en) | Automatic testing method and device | |
| CN114064803A (en) | Data synchronization method and device | |
| CN113783913A (en) | Message pushing management method and device | |
| CN112948138A (en) | Method and device for processing message | |
| CN112148705A (en) | Data migration method and device | |
| CN112073395B (en) | File distribution method and device | |
| CN109087097B (en) | Method and device for updating same identifier of chain code | |
| CN110688355A (en) | Method and device for changing container state | |
| CN112688982B (en) | User request processing method and device | |
| CN113726885A (en) | Method and device for adjusting flow quota | |
| CN114049065A (en) | Data processing method, device and system | |
| CN113743630A (en) | Method and device for executing intelligent call reservation | |
| CN112711572A (en) | Online capacity expansion method and device suitable for sub-warehouse and sub-meter | |
| CN112749204A (en) | Method and device for reading data | |
| CN112214500A (en) | Data comparison method and device, electronic equipment and storage medium | |
| CN112732728A (en) | Data synchronization method and system | |
| CN113766437B (en) | Short message sending method and device | |
| CN111950232A (en) | Method and device for automatically switching number segments | |
| CN111290927A (en) | Data monitoring method and device | |
| CN109656519B (en) | Method and device for automatically accessing service data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |