CN111400104A - Data synchronization method and device, electronic equipment and storage medium - Google Patents
Data synchronization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111400104A CN111400104A CN202010211913.3A CN202010211913A CN111400104A CN 111400104 A CN111400104 A CN 111400104A CN 202010211913 A CN202010211913 A CN 202010211913A CN 111400104 A CN111400104 A CN 111400104A
- Authority
- CN
- China
- Prior art keywords
- execution
- data
- abnormal
- type
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1443—Transmit or communication errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Retry When Errors Occur (AREA)
Abstract
The specification provides a data synchronization method and device, an electronic device and a storage medium, wherein the method comprises the following steps: under the condition that the execution of a synchronous task fails, acquiring an abnormal reason causing the execution failure of the synchronous task, wherein the synchronous task is used for implementing data synchronization operation from a source data table to a target data table; determining a retry execution condition according to the execution parameters of the synchronous task under the condition that the type of the abnormal reason is determined to be a recoverable type; and in the case that the retry execution condition is met, attempting to execute the incomplete part of the synchronization task until the execution is successful so as to continue to implement the data synchronization operation of the source data table to the destination data table. By optimizing the data synchronization task, the fault tolerance of data synchronization is improved, and the data which is not successfully synchronized is marked without additionally arranging a storage space, so that the resource consumption is reduced, and the efficiency of data synchronization is improved.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a data synchronization method and apparatus, an electronic device, and a storage medium.
Background
In the current data synchronization process, especially the synchronization of the full data, if the data synchronization task fails, the exception is directly thrown out, so that the synchronization task fails and terminates. Because the synchronization task of the full data is difficult to mark the point of failure, it is impossible to determine which data are successfully synchronized and which data are not successfully synchronized, and if the synchronization task is executed again, the previously synchronized data need to be re-synchronized, which easily causes resource waste and high cost.
Disclosure of Invention
In order to overcome the problems in the related art, the present specification provides a data synchronization method and apparatus, an electronic device, and a storage medium.
According to a first aspect of embodiments of the present specification, there is provided a data synchronization method including:
under the condition that the execution of a synchronous task fails, acquiring an abnormal reason causing the execution failure of the synchronous task, wherein the synchronous task is used for implementing data synchronization operation from a source data table to a target data table;
determining a retry execution condition according to the execution parameters of the synchronous task under the condition that the type of the abnormal reason is determined to be a recoverable type;
and in the case that the retry execution condition is met, attempting to execute the incomplete part of the synchronization task until the execution is successful so as to continue to implement the data synchronization operation of the source data table to the destination data table.
Optionally, before determining the type of the cause of the abnormality, the method further includes:
judging whether the execution time length of the synchronous task is smaller than a time length threshold value or not, or whether the synchronized data volume is smaller than a data volume threshold value or not;
if the synchronous task is judged to be the synchronous task, the synchronous task is executed again and/or alarm information is generated;
if not, executing the step of determining the type of the abnormal reason.
Optionally, the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
determining a retry execution condition according to the execution parameters of the synchronization task, including:
determining the trial execution time period according to a time length coefficient and the execution parameters, wherein the time length coefficient is determined by historical statistical recovery time length of the abnormal reason;
and/or determining the maximum number of attempts according to a number coefficient and the execution parameter, wherein the number coefficient is determined by the historical statistical recovery duration of the abnormal reason.
Optionally, the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
the execution parameters comprise execution duration and/or synchronized data volume;
the calculation formula of the time length of the attempt execution time period is as follows:
t ═ a × T, or T ═ b × S;
the maximum number of attempts is calculated by the following formula:
i ═ c × t, or I ═ d × S;
wherein T denotes a time length of the trial execution period, T denotes the execution time length, S denotes the synchronized data amount, a denotes a first time length coefficient, b denotes a second time length coefficient, I denotes the maximum number of trials, c denotes a first number coefficient, and d denotes a second number coefficient.
Optionally, if the execution time period or the execution times are greater than the maximum number of attempts and still unsuccessful, re-executing the synchronization task and/or generating alarm information.
Optionally, the data synchronization method further includes:
under the condition of receiving a type setting request, configuring an abnormal type list according to the type setting request, wherein the abnormal type list comprises abnormal reason information;
determining the type of the abnormal reason as a recoverable type, wherein the determining comprises the following steps:
and if the abnormal cause information matched with the abnormal cause exists in the abnormal type list, determining that the type of the abnormal cause is a recoverable type.
Optionally, the data synchronization method further includes:
and if the type of the abnormal reason is determined to be a recoverable type, if an abnormal type list does not contain abnormal reason information corresponding to the abnormal reason, adding the abnormal reason information corresponding to the abnormal reason to the abnormal type list, wherein the abnormal type list is used for recording the abnormal reason information of the recoverable type.
According to a second aspect of embodiments herein, there is provided a data synchronization apparatus comprising:
the system comprises an acquisition module, a synchronization module and a processing module, wherein the acquisition module is used for acquiring an abnormal reason causing the execution failure of a synchronization task under the condition of the execution failure of the synchronization task, and the synchronization task is used for implementing data synchronization operation from a source data table to a target data table;
the determining module is used for determining a retry execution condition according to the execution parameters of the synchronous task under the condition that the type of the abnormal reason is determined to be a recoverable type;
and the execution module is used for trying to execute the uncompleted part of the synchronization task under the condition that the retry execution condition is met until the execution is successful so as to continue to implement the data synchronization operation of the source data table to the target data table.
Optionally, the data synchronization apparatus further includes:
the judging module is used for judging whether the execution time length of the synchronous task is smaller than a time length threshold value or not, or whether the synchronized data volume is smaller than a data volume threshold value or not;
the judging module is further used for re-executing the synchronous task and/or generating alarm information under the condition that the judgment is yes; and under the condition that the judgment is negative, calling the determining module.
Optionally, the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
when determining a retry execution condition according to the execution parameters of the synchronization task, the determining module is configured to:
determining the trial execution time period according to a time length coefficient and the execution parameters, wherein the time length coefficient is determined by historical statistical recovery time length of the abnormal reason;
and/or determining the maximum number of attempts according to a number coefficient and the execution parameter, wherein the number coefficient is determined by the historical statistical recovery duration of the abnormal reason.
Optionally, if the time period of the execution attempts is longer than the maximum number of attempts and the number of times of the execution attempts is not yet successful, the execution module is further configured to re-execute the synchronization task and/or generate alarm information.
Optionally, the data synchronization apparatus further includes:
the device comprises a configuration module, a processing module and a processing module, wherein the configuration module is used for configuring an abnormal type list according to a type setting request under the condition of receiving the type setting request, and the abnormal type list comprises abnormal reason information;
in determining the type of the cause of the abnormality, the determination module is to:
and if the abnormal cause information matched with the abnormal cause exists in the abnormal type list, determining that the type of the abnormal cause is a recoverable type.
According to a third aspect of the embodiments of the present specification, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the data synchronization method according to any one of the above items when executing the computer program.
According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data synchronization method of any one of the above.
The technical scheme provided by the embodiment of the specification can have the following beneficial effects:
in the embodiment of the present specification, when a data synchronization failure captures an abnormal cause, the synchronization task is not directly stopped, but it is determined whether the abnormal cause is a recoverable abnormality. If yes, determining a retry execution condition according to the execution parameters of the synchronous task, and periodically trying to continue executing the incomplete part of the synchronous task under the condition that the retry execution condition is met until the execution is successful, synchronizing the data which is not successfully synchronized to the destination data table, and not needing to be resynchronized for the data which is successfully synchronized. Therefore, by optimizing the data synchronization task, the fault tolerance of the data synchronization is improved, and the data which is not successfully synchronized is marked without additionally arranging a storage space, so that the resource consumption is reduced, and the efficiency of the data synchronization is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the specification.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present specification and together with the description, serve to explain the principles of the specification.
FIG. 1 is a flow chart illustrating a method of data synchronization according to an exemplary embodiment of the present description.
FIG. 2 is a flow chart illustrating another method of data synchronization shown in accordance with an exemplary embodiment of the present description.
FIG. 3 is a flow chart illustrating another method of data synchronization shown in accordance with an exemplary embodiment of the present description.
Fig. 4 is a block diagram of a data synchronization apparatus, shown in accordance with an exemplary embodiment of the present specification.
Fig. 5 is a schematic structural diagram of an electronic device shown in the present specification according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the specification, as detailed in the appended claims.
The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present specification. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
In the related technology, in the data synchronization process, especially the synchronization of the full data, if the data cannot be successfully written into the data table due to the problems of the network or the database environment and the like in the synchronization task, the exception is directly thrown out, the execution failure of the synchronization task is judged, and the synchronization task is failed to terminate. For the synchronization of the full data, because the synchronization task of the full data is difficult to mark the point of failure, that is, which data are successfully synchronized and which data are not successfully synchronized are not marked, after the network or the database is recovered to be normal, if the synchronization task is executed again, the previous data successfully synchronized are synchronized again, so as to ensure the correctness of the execution of the synchronization task. The synchronization task of the full amount of data is executed once, the data amount is usually large, and under the condition that a large amount of data is synchronized, when the task is started next time, the previously synchronized data is resynchronized again, so that the cost is high, and the efficiency is low.
At present, if the execution of the task fails in the process of the full data synchronization, the data which is not synchronized and/or the corresponding identifier may also be stored by allocating a storage area to mark a point at which the task fails, but this method needs to additionally allocate a resource space to store the data which is not synchronized and/or the corresponding identifier, which is relatively expensive.
Based on the above problems, the present specification provides a data synchronization method, wherein in a database synchronization process, when a recoverable abnormal cause occurs, a retry execution condition of a synchronization task is determined according to an executed condition of the synchronization task, and an incomplete part of the synchronization task is repeatedly tried to be executed according to the retry execution condition until the execution is successful, so that resynchronization is not required for data that has been successfully synchronized, the efficiency of data synchronization is improved, and resource waste is not caused.
The following provides a detailed description of examples of the present specification.
As shown in fig. 1, fig. 1 is a flow chart illustrating a data synchronization method according to an exemplary embodiment, including the following steps:
In this embodiment, but not limited to, the ET L (data warehouse technology) synchronization tool may be used to perform a synchronization task to perform a data synchronization operation from the source data table to the destination data table, that is, to obtain the full amount of data from the source data table and write the obtained full amount of data into the destination data table.
If the synchronous task fails to be executed, the database feeds back error codes and/or specific error information, and the data synchronization tool can determine the reason causing the synchronous task to be executed failure according to the error codes and/or the specific error information fed back by the database, and cannot directly judge that the synchronous task fails to be executed.
And 102, under the condition that the type of the abnormal reason is determined to be a recoverable type, determining a retry execution condition according to the execution parameters of the synchronous task.
Wherein, the execution parameters include: an execution duration and/or an amount of synchronized data. The retry execution condition includes an attempt execution period and/or a maximum number of attempts.
It can be understood that the retry execution condition is determined by the execution parameter of the synchronization task, and since the execution parameter of different synchronization tasks is generally different, the retry execution condition determined by different synchronization tasks is generally different; in the process of meeting the abnormal reason twice, the same synchronous task has different execution parameters, so that the same synchronous task has different retry execution conditions.
In step 102, if it is determined that the abnormal reason is a recoverable type, which indicates that the abnormal reason is recovered within a short time period, the synchronous task enters a waiting state and step 103 is executed; if the abnormal reason is determined to be of an unrecoverable type, the abnormal reason is indicated to not be recovered within a short time period, and even if the incomplete part of the synchronization task is tried for multiple times, the synchronization task is not successfully executed, in this case, step 103 is not executed, the synchronization task is determined to be failed, the synchronization task is stopped, and meanwhile, an alarm message can be generated so as to inform operation and maintenance personnel to timely overhaul. The alarm information may be, but is not limited to, sent to the operation and maintenance personnel by a short message and/or an email.
In one embodiment, a user may set, according to actual needs, which abnormal causes are recoverable and which abnormal causes are not recoverable, for example, set abnormal causes such as a database connection abnormality (recoverable by restarting or repairing a database), a network oscillation (recoverable automatically later), and the like as a recoverable type, and set other abnormal causes such as a hardware failure of the database as an unrecoverable type.
In another embodiment, a user interface for setting the exception type can be further provided, so that a user can manually add the exception reason required by the user. The user can select and set which abnormal reasons causing the synchronous task to fail to execute through the user interface according to experience so as to be recoverable. The user interface can list some abnormal reasons of the database, generate a type setting request according to the selection of a user, and then configure an abnormal type list according to the type setting request, wherein the abnormal type list records the related information of the abnormal reasons selected by the user. The user can also modify the exception type list according to actual requirements.
In another embodiment, the type of the abnormal cause may be determined as a recoverable type if the duration is short and the automatic recovery is possible by performing statistical analysis on historical information of the abnormal cause of the database, for example, by performing statistical analysis on information such as occurrence time, end time, duration, whether to perform automatic recovery, and the like. The results of the statistical analysis may be, but are not limited to, saved in the form of an anomaly type list, where the anomaly type list records information about the anomaly cause selected by the user.
In another embodiment, the two manners of configuring the abnormal type list may be used in combination, and a user may add some abnormal reasons in the abnormal type list by himself/herself, and at the same time, the abnormal type list may be updated through a statistical analysis result of historical information of the abnormal reasons.
In the data synchronization process, if an abnormal reason causing the synchronization task failure is judged to be a recoverable reason and the abnormal reason information corresponding to the abnormal reason is not contained in the abnormal type list, the abnormal reason information is added to the abnormal type list.
In step 102, when determining the type of the abnormal cause, the data synchronization tool may call the abnormal type list, match the abnormal cause obtained in step 101 with information in the abnormal type list, determine that the type of the abnormal cause is a recoverable type if recoverable type information matching the type of the abnormal cause exists in the abnormal type list, and may execute step 103. The recoverable type information may include, but is not limited to, a name of a cause of the exception, an error code, other error information, and the like.
And 103, under the condition that the retry execution condition is met, attempting to continue executing the uncompleted part of the synchronization task until the execution is successful so as to continue implementing the data synchronization operation of the source data table to the destination data table.
In the related art, the data synchronization tool is provided with a buffer area for storing a small amount of data acquired from the source data table, and when data is written, one piece of data is sequentially acquired from the buffer area and written into a corresponding position of the destination data table. If the write data operation is suspended and the buffer area is full, the read data operation is suspended. As long as the currently executed synchronization task is not determined to be failed to execute, the data in the cache region is cached continuously, and the data in the cache region is not emptied until the synchronization task is determined to be failed to execute, so that the point of the task failure to execute can be marked by the cache region, and a storage region does not need to be additionally arranged.
In step 103, if the retry execution condition is satisfied, the incomplete part of the synchronization task is executed in a loop until the execution is successful. For example, if the abnormal cause is network oscillation, assuming that the abnormal cause can be recovered normally after 10 minutes, the execution of the 10:00 synchronization task fails, and in step 103, the incomplete part of the synchronization task is attempted to be executed at a frequency of once every 8 minutes; 10:00 making a first attempt, the attempt failing because the network state is not recovered; 08 makes the second attempt, because the network state is not recovered, the attempt fails; and 10:16, performing a third attempt, wherein the network state is recovered before the third attempt (recovery is performed at 10: 10), the third attempt is successful, the data synchronization tool continues to sequentially acquire data from the cache region and write the data into the corresponding position of the destination data table, and the corresponding data synchronization tool sequentially acquires the data from the source data table and stores the data into the cache region. The data which has been successfully synchronized does not need to be synchronized again.
In step 103, if the retry execution condition is not satisfied, for example, the retry number exceeds the maximum retry number and is not executed successfully, or the retry number exceeds the retry execution time period and is not executed successfully, which indicates that the reason for the abnormality is not eliminated, in this case, the synchronization task is determined to be an execution failure, at this time, the data in the cache area is cleared, and meanwhile, an alarm message is generated, so as to notify the operation and maintenance personnel to overhaul in time, and the remaining data will not be synchronized. And after the abnormal reason is eliminated, re-executing the synchronization task, and re-synchronizing all data, including the data which is successfully synchronized before.
In this embodiment, when the data synchronization failure captures an abnormal cause, the synchronization task is not directly stopped, but it is determined whether the abnormal cause is a recoverable abnormality. If yes, a retry execution condition is determined according to the execution parameters of the synchronous task in a targeted mode, in the case that the retry execution condition is met, the incomplete part of the synchronous task is tried to be executed continuously periodically until the execution is successful, the data which is not successfully synchronized is synchronized to the target data table, and the data which is successfully synchronized does not need to be resynchronized. Therefore, by optimizing the data synchronization task, the fault tolerance of the data synchronization is improved, and the data which is not successfully synchronized is marked without additionally arranging a storage space, so that the resource consumption is reduced, and the efficiency of the data synchronization is improved.
In another embodiment, before executing step 102, it may be determined whether the execution duration of the synchronization task is less than a duration threshold, or whether the synchronized data amount is less than a data amount threshold.
If the judgment result is yes, the data synchronization task is only executed for a small part, and the data volume is small even if the synchronization is carried out again on the synchronized data, at this time, the task failure can be judged and the synchronization task can be stopped without entering the requirement of circularly trying to execute the unfinished part of the synchronization task, and meanwhile, the alarm information can be generated. After the abnormal reason is eliminated, the synchronization task is executed again, and the data which is successfully synchronized needs to be synchronized again.
If no, it indicates that the synchronization task has successfully synchronized the data with a certain amount of data, and if it needs to consume a large amount of resources to synchronize the data again, step 102 is executed.
Fig. 2 is a flowchart illustrating another data synchronization method according to an exemplary embodiment, and this embodiment describes, on the basis of the foregoing embodiments, a process of determining a retry execution condition according to an execution duration and performing a data synchronization operation, including the following steps:
The specific implementation manner of step 201 is similar to that of step 101, and is not described here again.
And 202, under the condition that the type of the abnormal reason is determined to be a recoverable type, determining a retry execution condition according to the execution duration of the synchronous task.
Wherein the retry execution condition includes an attempt execution time period and/or a maximum number of attempts. The trial execution time period is determined by a duration coefficient and an execution duration, the duration coefficient may be determined empirically, but not limited to, and the duration coefficient may also be determined by historical statistical recovery durations of causes of the anomaly. The maximum number of attempts is determined by a number coefficient and an execution time length, the number coefficient can be determined by experience, but not limited to, and the number coefficient can also be determined by historical statistics recovery time length of the abnormality cause.
The calculation formula of the time length of the attempt execution period may be expressed as follows, but is not limited thereto:
T＝a*t；
the calculation formula of the maximum number of attempts may be expressed as follows, but is not limited thereto:
I＝c*t；
where T denotes a time length of the trial execution period, T denotes an execution time length, a denotes a time length coefficient, I denotes a maximum number of trials, and c denotes a number coefficient.
For example, if the duration factor is 1/8, the number of times factor is 1/32; then if the synchronization task has been executed for 8 hours, but not limited to, taking the product of the time length coefficient and the execution time length, that is, 1 hour as the time length of the attempted execution time period; the maximum number of attempts is taken to be 4, which is the product of the number coefficient and the execution duration. The trial is performed 4 times in 1 hour, and the frequency of the trial execution is executed once every 20 minutes.
And step 203, in the trial execution time period, trying to continue executing the uncompleted part of the synchronization task until the execution is successful, so as to continue implementing the data synchronization operation of the source data table to the destination data table.
In step 203, the frequency of the attempted execution is determined according to the attempted execution time period and the maximum execution times, and if the attempted execution time period is 1 hour and the maximum execution times is 5 times, the attempted execution frequency is executed once every 15 minutes. I.e. the unfinished part of the synchronization task is executed in cycles at a frequency of once 15 minutes within 1 hour until the execution is successful.
The starting point of the trial execution time period may be a time when the type of the abnormal cause is determined to be a recoverable type, that is, if the type of the abnormal cause is determined to be a recoverable type, the first trial is started, or the first trial may be performed after a preset time period.
For example, if the abnormal cause is network oscillation, assuming that the abnormal cause can be recovered normally after 10 minutes, 9:59 synchronization task fails to be executed, step 103 tries to execute the incomplete part of the synchronization task at a frequency of once every 15 minutes; 10:00 making a first attempt, the attempt failing because the network state is not recovered; 10:15, performing a second attempt, wherein the network state is recovered before the second attempt (10:09 recovery), the second attempt is successful, the data synchronization tool continues to sequentially acquire data from the buffer area and write the data into the corresponding position of the destination data table, the corresponding data synchronization tool sequentially acquires data from the source data table and stores the data into the buffer area, and the data which is successfully synchronized does not need to be synchronized again.
In step 203, if the execution time period is over, or the retry number exceeds the maximum attempt number, and the execution is not successful, it indicates that the reason for the abnormality is not eliminated, in this case, the synchronization task is determined to have failed to be executed, at this time, the data in the cache area is cleared, and meanwhile, an alarm message is generated, so as to notify the operation and maintenance staff to overhaul in time, and the remaining data will not be synchronized. And after the abnormal reason is eliminated, re-executing the synchronization task, and re-synchronizing all data, including the data which is successfully synchronized before.
In another embodiment, before executing step 202, it may be determined whether the execution duration of the synchronization task is less than a duration threshold, for example, set to 30 minutes.
If the judgment result is yes, the data synchronization task is only executed for a small part, and the data volume is small even if the synchronization is carried out again on the synchronized data, at this time, the task failure can be judged and the synchronization task can be stopped without entering the requirement of circularly trying to execute the unfinished part of the synchronization task, and meanwhile, the alarm information can be generated. After the abnormal reason is eliminated, the synchronization task is executed again, and the data which is successfully synchronized needs to be synchronized again.
If no, it indicates that the synchronization task has successfully synchronized a certain amount of data, and if it needs a large resource consumption to synchronize the part of data again, then step 202 is executed.
Fig. 3 is a flowchart illustrating another data synchronization method according to an exemplary embodiment, where this embodiment describes a process of determining a retry execution condition according to a synchronized data amount and performing a data synchronization operation based on the foregoing embodiments, and includes the following steps:
The specific implementation process of step 301 is similar to step 101, and is not described herein again.
Wherein the retry execution condition includes an attempt execution time period and/or a maximum number of attempts. The trial execution time period is determined by a duration coefficient and the synchronized data amount, the duration coefficient may be determined empirically, but not limited to, and the duration coefficient may also be determined by historical statistical recovery durations of the causes of the anomalies. The maximum number of attempts is determined by a number of coefficients and the amount of synchronized data, the number of coefficients may be determined empirically, but not limited to, the number of times coefficients may also be determined by historical statistical recovery durations of the causes of the anomaly.
The calculation formula of the time length of the attempt execution period may be expressed as follows, but is not limited thereto:
T＝b*S；
the calculation formula of the maximum number of attempts may be expressed as follows, but is not limited thereto:
I＝d*S；
where T denotes a time length of the trial execution period, S denotes a synchronized data amount, b denotes a time length coefficient, I denotes a maximum number of trials, and d denotes a number coefficient.
For example, if the duration factor is 1/10000, the number of times factor is 1/30000; then, if the sync task has synchronized 300000 pieces of data (synchronized data amount), the product of the time length coefficient and the synchronized data amount, that is, 30 minutes, may be but is not limited to be used as the trial execution time period; the product of the number coefficient and the synchronized data volume, i.e. 10 times, may be, but is not limited to, taken as the maximum number of attempts, so that each 3-minute attempt is performed within 30 minutes.
And step 303, in the execution attempt time period, attempting to continue executing the incomplete part of the synchronization task until the execution is successful, so as to continue implementing the data synchronization operation of the source data table to the destination data table.
In step 303, the frequency of the attempted execution is determined according to the attempted execution time period and the maximum execution times, and if the attempted execution time period is 30 minutes and the maximum execution times is 10 times, the attempted execution frequency is executed once every 3 minutes. I.e. within 30 minutes, the unfinished part of the synchronization task is executed cyclically with an attempt execution frequency of 3 minutes until the execution is successful.
The starting point of the trial execution time period may be a time when the type of the abnormal cause is determined to be a recoverable type, that is, if the type of the abnormal cause is determined to be a recoverable type, the first trial is started, or the first trial may be performed after a preset time period.
For example, if the abnormal cause is network oscillation, assuming that the abnormal cause can be recovered normally after 10 minutes, 9:59 synchronization task fails to be executed, step 303 tries to execute the incomplete part of the synchronization task at a frequency of once every 3 minutes; 03, performing a first attempt, wherein the attempt fails because the network state is not recovered; 10:06, carrying out a second attempt, wherein the attempt fails because the network state is not recovered; and 10:09, performing a third attempt, wherein the network state is recovered (10:09 recovery), the third attempt is successful, the data synchronization tool continues to sequentially acquire data from the cache region and write the data into the corresponding position of the destination data table, and the corresponding data synchronization tool sequentially acquires data from the source data table and stores the data into the cache region. The data which has been successfully synchronized does not need to be synchronized again.
In step 303, if the execution time period is over, or the retry number exceeds the maximum attempt number, and the execution is not successful, which indicates that the reason for the abnormality is not eliminated, in this case, the synchronization task is determined to have failed to be executed, at this time, the data in the cache area is cleared, and meanwhile, an alarm message is generated, so as to notify the operation and maintenance staff to overhaul in time, and the remaining data will not be synchronized. And after the abnormal reason is eliminated, re-executing the synchronization task, and re-synchronizing all data, including the data which is successfully synchronized before.
In another embodiment, before performing step 302, it may be determined whether the synchronized data amount is less than a data amount threshold, which is set to 1/10 of the total data amount of the synchronization task, for example.
If the judgment result is yes, the data synchronization task is only executed for a small part, and the data volume is small even if the data synchronization is carried out again for the synchronized data, at this time, the task failure can be judged and the synchronization task can be stopped without entering the requirement of circularly trying to execute the unfinished part of the synchronization task, and meanwhile, the alarm information can be generated. After the abnormal reason is eliminated, the synchronization task is executed again, and the data which is successfully synchronized needs to be synchronized again.
If no, it indicates that the synchronization task has successfully synchronized a certain amount of data, and if it needs a large resource consumption to synchronize the part of data again, then step 202 is executed.
In any of the above embodiments, for the same synchronization task, the retry execution condition (the retry execution time period and the maximum number of attempts) is determined according to the execution condition of the synchronization task, when the synchronization task is started, the retry execution condition is not determined, only when the exception cause causing the task execution failure is recoverable, the retry execution time period and the maximum number of attempts are determined according to the execution condition of the synchronization task, and the execution time length of the synchronization task or the synchronized data amount directly affects the retry execution time period and the maximum number of attempts. The mode of dynamically determining the execution time period and the maximum number of attempts is beneficial to reducing the exception handling cost of full data synchronization.
Corresponding to the embodiment of the data synchronization method, the specification also provides an embodiment of the data synchronization device and a terminal applied by the data synchronization device.
Fig. 4 is a block diagram of a data synchronization apparatus shown in the present specification according to an exemplary embodiment, the data synchronization apparatus including: an acquisition module 41, a determination module 42 and an execution module 43.
An obtaining module 41, configured to, in a case that execution of a synchronization task fails, obtain an abnormal reason causing the execution failure of the synchronization task, where the synchronization task is used to implement data synchronization operation from a source data table to a destination data table;
a determining module 42, configured to determine a retry execution condition according to an execution parameter of the synchronization task when it is determined that the type of the exception cause is a recoverable type;
and the execution module 43 is configured to, in a case that the retry execution condition is satisfied, attempt to execute the incomplete part of the synchronization task until the execution is successful, so as to continue to implement the data synchronization operation of the source data table to the destination data table.
Optionally, the data synchronization apparatus further includes:
the judging module is used for judging whether the execution time length of the synchronous task is smaller than a time length threshold value or not, or whether the synchronized data volume is smaller than a data volume threshold value or not;
the judging module is further used for re-executing the synchronous task and/or generating alarm information under the condition that the judgment is yes; and under the condition that the judgment is negative, calling the determining module.
Optionally, the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
when determining a retry execution condition according to the execution parameters of the synchronization task, the determining module is configured to:
determining the trial execution time period according to a time length coefficient and the execution parameters, wherein the time length coefficient is determined by historical statistical recovery time length of the abnormal reason;
and/or determining the maximum number of attempts according to a number coefficient and the execution parameter, wherein the number coefficient is determined by the historical statistical recovery duration of the abnormal reason.
Optionally, if the time period of the execution attempts is longer than the maximum number of attempts and the number of times of the execution attempts is not yet successful, the execution module is further configured to re-execute the synchronization task and/or generate alarm information.
Optionally, the data synchronization apparatus further includes:
the device comprises a configuration module, a processing module and a processing module, wherein the configuration module is used for configuring an abnormal type list according to a type setting request under the condition of receiving the type setting request, and the abnormal type list comprises abnormal reason information;
in determining the type of the cause of the abnormality, the determination module is to:
and if the abnormal cause information matched with the abnormal cause exists in the abnormal type list, determining that the type of the abnormal cause is a recoverable type.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in the specification. One of ordinary skill in the art can understand and implement it without inventive effort.
Fig. 5 is a schematic diagram of an electronic device according to an exemplary embodiment of the present invention, showing a block diagram of an exemplary electronic device 50 suitable for use in implementing any of the embodiments of the present invention. The electronic device 50 shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 5, the electronic device 50 may be embodied in the form of a general purpose computing device, which may be, for example, a server device. The components of the electronic device 50 may include, but are not limited to: the at least one processor 51, the at least one memory 52, and a bus 53 connecting the various system components (including the memory 52 and the processor 51).
The bus 53 includes a data bus, an address bus, and a control bus.
The memory 52 may include volatile memory, such as Random Access Memory (RAM)521 and/or cache memory 522, and may further include Read Only Memory (ROM) 523.
The processor 51 executes various functional applications and data processing, such as the methods provided by any of the above embodiments, by running a computer program stored in the memory 52.
The electronic device 50 may also communicate with one or more external devices 54 (e.g., keyboard, pointing device, etc.) such communication may be through AN input/output (I/O) interface 55, and the model-generated electronic device 50 may also communicate with one or more networks (e.g., a local area network (L AN), a Wide Area Network (WAN) and/or a public network, such as the Internet) through a network adapter 56. As shown, the network adapter 56 communicates with other modules of the model-generated electronic device 50 through a bus 53.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
The present specification also provides a computer readable storage medium, on which a computer program is stored, wherein the computer program is configured to implement the steps of any one of the methods described above when executed by a processor.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (14)
1. A data synchronization method, characterized in that the data synchronization method comprises:
under the condition that the execution of a synchronous task fails, acquiring an abnormal reason causing the execution failure of the synchronous task, wherein the synchronous task is used for implementing data synchronization operation from a source data table to a target data table;
determining a retry execution condition according to the execution parameters of the synchronous task under the condition that the type of the abnormal reason is determined to be a recoverable type;
and in the case that the retry execution condition is met, attempting to execute the incomplete part of the synchronization task until the execution is successful so as to continue to implement the data synchronization operation of the source data table to the destination data table.
2. The data synchronization method of claim 1, prior to determining the type of the cause of the anomaly, further comprising:
judging whether the execution time length of the synchronous task is smaller than a time length threshold value or not, or whether the synchronized data volume is smaller than a data volume threshold value or not;
if the synchronous task is judged to be the synchronous task, the synchronous task is executed again and/or alarm information is generated;
if not, executing the step of determining the type of the abnormal reason.
3. The data synchronization method according to claim 1, wherein the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
determining a retry execution condition according to the execution parameters of the synchronization task, including:
determining the trial execution time period according to a time length coefficient and the execution parameters, wherein the time length coefficient is determined by historical statistical recovery time length of the abnormal reason;
and/or determining the maximum number of attempts according to a number coefficient and the execution parameter, wherein the number coefficient is determined by the historical statistical recovery duration of the abnormal reason.
4. The data synchronization method according to claim 1, wherein the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
the execution parameters comprise execution duration and/or synchronized data volume;
the calculation formula of the time length of the attempt execution time period is as follows:
t ═ a × T, or T ═ b × S;
the maximum number of attempts is calculated by the following formula:
i ═ c × t, or I ═ d × S;
wherein T denotes a time length of the trial execution period, T denotes the execution time length, S denotes the synchronized data amount, a denotes a first time length coefficient, b denotes a second time length coefficient, I denotes the maximum number of trials, c denotes a first number coefficient, and d denotes a second number coefficient.
5. The data synchronization method according to claim 3 or 4, characterized in that if the execution time period of the attempts or the number of execution of the attempts is greater than the maximum number of attempts and still has not been successful, the synchronization task is re-executed and/or alarm information is generated.
6. The data synchronization method of claim 1, wherein the data synchronization method further comprises:
under the condition of receiving a type setting request, configuring an abnormal type list according to the type setting request, wherein the abnormal type list comprises abnormal reason information;
determining the type of the abnormal reason as a recoverable type, wherein the determining comprises the following steps:
and if the abnormal cause information matched with the abnormal cause exists in the abnormal type list, determining that the type of the abnormal cause is a recoverable type.
7. The data synchronization method of claim 1, wherein the data synchronization method further comprises:
and if the type of the abnormal reason is determined to be a recoverable type, if an abnormal type list does not contain abnormal reason information corresponding to the abnormal reason, adding the abnormal reason information corresponding to the abnormal reason to the abnormal type list, wherein the abnormal type list is used for recording the abnormal reason information of the recoverable type.
8. A data synchronization apparatus, characterized in that the data synchronization apparatus comprises:
the system comprises an acquisition module, a synchronization module and a processing module, wherein the acquisition module is used for acquiring an abnormal reason causing the execution failure of a synchronization task under the condition of the execution failure of the synchronization task, and the synchronization task is used for implementing data synchronization operation from a source data table to a target data table;
the determining module is used for determining a retry execution condition according to the execution parameters of the synchronous task under the condition that the type of the abnormal reason is determined to be a recoverable type;
and the execution module is used for trying to execute the uncompleted part of the synchronization task under the condition that the retry execution condition is met until the execution is successful so as to continue to implement the data synchronization operation of the source data table to the target data table.
9. The data synchronization apparatus of claim 8, wherein the data synchronization apparatus further comprises:
the judging module is used for judging whether the execution time length of the synchronous task is smaller than a time length threshold value or not, or whether the synchronized data volume is smaller than a data volume threshold value or not;
the judging module is further used for re-executing the synchronous task and/or generating alarm information under the condition that the judgment is yes; and under the condition that the judgment is negative, calling the determining module.
10. The data synchronization apparatus according to claim 8, wherein the retry execution condition includes an attempt execution time period and/or a maximum number of attempts;
when determining a retry execution condition according to the execution parameters of the synchronization task, the determining module is configured to:
determining the trial execution time period according to a time length coefficient and the execution parameters, wherein the time length coefficient is determined by historical statistical recovery time length of the abnormal reason;
and/or determining the maximum number of attempts according to a number coefficient and the execution parameter, wherein the number coefficient is determined by the historical statistical recovery duration of the abnormal reason.
11. The data synchronization apparatus of claim 10, wherein the execution module is further configured to re-execute the synchronization task and/or generate alarm information if the execution time period or the number of execution attempts is greater than the maximum number of attempts and is still unsuccessful.
12. The data synchronization apparatus of claim 8, wherein the data synchronization apparatus further comprises:
the device comprises a configuration module, a processing module and a processing module, wherein the configuration module is used for configuring an abnormal type list according to a type setting request under the condition of receiving the type setting request, and the abnormal type list comprises abnormal reason information;
in determining the type of the cause of the abnormality, the determination module is to:
and if the abnormal cause information matched with the abnormal cause exists in the abnormal type list, determining that the type of the abnormal cause is a recoverable type.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data synchronization method of any one of claims 1 to 7 when executing the computer program.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data synchronization method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010211913.3A CN111400104B (en) | 2020-03-24 | 2020-03-24 | Data synchronization method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010211913.3A CN111400104B (en) | 2020-03-24 | 2020-03-24 | Data synchronization method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400104A true CN111400104A (en) | 2020-07-10 |
| CN111400104B CN111400104B (en) | 2021-09-28 |
Family
ID=71436514
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010211913.3A Active CN111400104B (en) | 2020-03-24 | 2020-03-24 | Data synchronization method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400104B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112347189A (en) * | 2020-11-05 | 2021-02-09 | 江苏电力信息技术有限公司 | Cloud computing-based financial data consistency failure discovery and recovery method |
| CN112800134A (en) * | 2021-01-27 | 2021-05-14 | 北京明略软件系统有限公司 | Data synchronization method and system |
| CN113051349A (en) * | 2021-04-02 | 2021-06-29 | 广东美电贝尔科技集团股份有限公司 | Data synchronization method for duty system |
| WO2023083066A1 (en) * | 2021-11-11 | 2023-05-19 | 北京沃东天骏信息技术有限公司 | Data synchronization method and apparatus |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081665A (en) * | 2008-12-10 | 2011-06-01 | 阿里巴巴集团控股有限公司 | Data synchronization method and device |
| US20140250326A1 (en) * | 2013-03-01 | 2014-09-04 | Teradata Corporation | Method and system for load balancing a distributed database providing object-level management and recovery |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
-
2020
- 2020-03-24 CN CN202010211913.3A patent/CN111400104B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081665A (en) * | 2008-12-10 | 2011-06-01 | 阿里巴巴集团控股有限公司 | Data synchronization method and device |
| US20140250326A1 (en) * | 2013-03-01 | 2014-09-04 | Teradata Corporation | Method and system for load balancing a distributed database providing object-level management and recovery |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112347189A (en) * | 2020-11-05 | 2021-02-09 | 江苏电力信息技术有限公司 | Cloud computing-based financial data consistency failure discovery and recovery method |
| CN112800134A (en) * | 2021-01-27 | 2021-05-14 | 北京明略软件系统有限公司 | Data synchronization method and system |
| CN113051349A (en) * | 2021-04-02 | 2021-06-29 | 广东美电贝尔科技集团股份有限公司 | Data synchronization method for duty system |
| WO2023083066A1 (en) * | 2021-11-11 | 2023-05-19 | 北京沃东天骏信息技术有限公司 | Data synchronization method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400104B (en) | 2021-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111400104B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN105677469B (en) | Timed task execution method and device | |
| CN108563502B (en) | Task scheduling method and device | |
| CN108710544B (en) | Process monitoring method of database system and rail transit comprehensive monitoring system | |
| CN112307121B (en) | Oracle-based data synchronization method and device and computer equipment | |
| CN111274052A (en) | Data distribution method, server, and computer-readable storage medium | |
| JP2008210148A (en) | Failure handling system and failure handling method | |
| CN110427422B (en) | Data consistency checking method and equipment during data synchronization abnormity and storage medium | |
| CN105095008B (en) | A kind of distributed task scheduling fault redundance method suitable for group system | |
| CN110895488B (en) | Task scheduling method and device | |
| CN112506702A (en) | Data center disaster tolerance method, device, equipment and storage medium | |
| CN112631761A (en) | Task scheduling monitoring method and device | |
| CN111198662B (en) | Data storage method, device and computer readable storage medium | |
| CN107943617B (en) | Data restoration method and device and server cluster | |
| CN109257396B (en) | Distributed lock scheduling method and device | |
| CN112486707A (en) | Redis-based message asynchronous consumption method and device | |
| CN110798339A (en) | Task disaster tolerance method based on distributed task scheduling framework | |
| CN109165181A (en) | A kind of more boards are from configuration and task from compartment system and method | |
| CN115185787A (en) | Method and device for processing transaction log | |
| US20090106781A1 (en) | Remote call handling methods and systems | |
| CN115687019A (en) | Database cluster fault processing method, intelligent monitoring platform, equipment and medium | |
| CN112000492B (en) | Public number user tag management system and method | |
| CN110113395B (en) | Shared file system maintenance method and device | |
| CN108667636A (en) | Configuration information recovery method, apparatus, memory and computer equipment | |
| CN111177232B (en) | Data processing method, device, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |